[e-lang] How to fix the Data-E unserialization order problem

[Kevin Reid]

How to fix Data-E:

The problem with Data-E is that serialization may create an ordering  
of calls which does not succeed when executed/unserialized: in  
particular, x.y() might be executed before x is resolved.

In order to fix this robustly, we need to, at unserialization time,  
perform the calls in an ordering which respects dependencies;  
recipients must be unserialized before calls to them, and some calls  
may require that some arguments are unserialized before those calls  
occur (e.g. making a ConstSet).

We would like to preserve the property that the serialized form is  
interpretable as a subset of E.


My thought: An E programmer might want, if they can express it  
reasonably cleanly, the same behavior of "assemble these objects in  
whichever order will succeed" that Data-E needs; it *might* enable  
the writing of more declarative E programs.

Whether or not this is actually the case, it suggests a way to fix  
Data-E:

   1. Write a library providing the operation of performing a set of  
calls in an appropriate order.

   2. Redefine Data-E as the subset of E which expresses usage of  
this library, which I'll call the "assembler".

The non-E (binary, AST, ...) forms for Data-E remain essentially as  
they are, except they cease being nested, and are considered to refer  
to the library; the E-expression form becomes E programs which use  
the assembler.

Data-E continues to be a subset of E; but instead of being "only  
LiteralExpr, CallExpr, DefineExpr ..." it is "only these patterns of  
calls to an object providing the assembler protocol".


MarkM and I just finished discussing this idea, and at the moment we  
think it is an adequate solution to the Data-E problem (provided that  
we can design a suitable assembler, of course).


The remaining problem is: What is the protocol for the assembler?

A very simple answer:

def [finish, o1, o2, _] := assemble([
   [o2, "getSomeFacet", []],
   [makeThingy, "run", [o3]],
   [__makeList, "run", [0, 0, 17, 5, 0]]
])
finish()
o1

This would be adequate for Data-E, but it is unpleasant for  
programmer usage; furthermore, it generates a large list of promises  
which is not used other than for variable binding.

Eliminating the binding list and finish():

def o2Resolver := (def o2)
def o3Resolver := (def o3)
assemble([
   [def o1, o2, "getSomeFacet", []],
   [o2Resolver, makeThingy, "run", [o3]],
   [o3Resolver, __makeList, "run", [0, 0, 17, 5, 0]]
])
o1

This is simple, but requires either pre-declaring promises and  
resolvers or the entries to be ordered (when possible) such that uses  
occur after declarations, defeating one of the motivations for the  
assembler.

Stateful assembler, accumulating a record of what it needs to deliver:

def o3
def [o1, o2] := [assemble(o2, "getSomeFacet", []),
                  assemble(makeThingy, "run", [o3])]
bind o3 := assemble(__makeList, "run", [0, 0, 17, 5, 0])
assemble.finish()

As in this contrived example, this allows the user to use any normal  
definition technique (recursive 'def', forward declaration) to  
construct the loops.


Now about the call syntax. Here's a variation on the stateful assembler:

def o3
def [o1, o2] := [assemble(o2).getSomeFacet(),
                  assemble(makeThingy).run(o3)]
bind o3 := assemble(__makeList).run(0, 0, 17, 5, 0)
assemble.finish()

The trick here is that assemble#run/1 takes the recipient promise and  
returns a plumbing object[1], which takes a call to it, no matter  
what the verb, as the message to deliver to the recipient when it is  
resolved. The intended advantage of this is that normal E call syntax  
is used for the verb and arguments.

At this point, I now realize, we actually have something which could  
be generated in the same shape as current Data-E, with added  
'assemble' calls, and succeed where current Data-E fails. I'll post a  
definition of the assembler tomorrow.

-- 
Kevin Reid                            <http://homepage.mac.com/kpreid/>