[e-lang] Scope / EEnv split

[Kevin Reid]

[Sorry, this message was delayed a couple days due to an outgoing mail  
server screwup.]

On Sep 7, 2010, at 9:58, Thomas Leonard wrote:
> On Mon, 2010-05-24 at 09:50 -0400, Kevin Reid wrote:
>> On May 21, 2010, at 9:22, Thomas Leonard wrote:
>>
>>> Here, the Scope type fills both roles ("scope" and "newScope"). Some
>>> safej rules prevent E programs from accessing "internal" details  
>>> (e.g.
>>> locals[]). This means you can't write the interpreter in E (using
>>> Scope), because E can't access these bits even when it needs them.
>>>
>>> If they were separate types ("Map env/Scope scope" rather than  
>>> "Scope
>>> scope/Scope newScope"), the safej restrictions could go away and
>>> normal
>>> capability rules would apply (if you have a Scope, you can call all
>>> its
>>> methods, whether you're E or Java).
>>
>> It's perfectly all right for an interpreter to have a private
>> environment-with-extra-useful-bits type, but it should not be named
>> Scope. And the public environment type cannot be a Map because it has
>> additional metadata, as previously discussed.
>>
>> I agree that it would be useful not to require safej for what-is-
>> currently-known-as-Scope. The proper answer is to as you suggest make
>> it a separate type, but that should not be known as Scope. Perhaps
>> 'EvalEnv' would be a good name.
>
> OK, I've had a go at this. Here are the patches so far:

I'm not going to have time to take a serious look at this for a while  
(I haven't looked at the source), but I'm liking the general direction  
you describe.

This is a review of your comments, not of your changes. This DOES NOT  
constitute approval to commit them.

> 1. Split Scope into Scope and EEnv, by making EEnv a copy of Scope

This is a good way to eliminate the "what the interpreter needs" vs  
"what the exposed API should be" problem; insofar as it creates  
efficiency issues, they would be raised by a compiling implementation  
as well.

I am uncertain about whether the type should be named Env or EEnv.

> EExpr.eval (and EExpr.compile) now take an EEnv instead of a Scope.
> evalToPair still takes a Scope, because it's specific to the  
> interpreter
> (a compiler probably wouldn't offer this?).

No. evalToPair, though poorly named, is the most-general interface and  
eval should be merely a shortcut for evalToPair(...)[0]. evalToPair is  
a highly useful tool for setting up environments and implementing  
REPLs. Whether the implementation interprets or compiles is irrelevant.

Also, we want to move away from eval on E ASTs toward using the  
elang.evm.Evaluator interface, because that allows "eval in other vat"  
to be conveniently expressed. (That interface should be reviewed; the  
first thing I note is its use of ConstMaps instead of Envs, which is  
obsolete in the current world.)

> 2. Remove unnecessary EEnv/Scope methods
>
> e.g. there's no need for Scope to have a bindings/0 method, and no  
> need
> for EEnv to have an EvalContext, etc

*nod*

> 3. Re-enabled LiteralSlotNounExpr and LiteralNounExpr optimisations
>
> These weren't used at all, but it appears that the intention was to  
> use
> them for DeepFrozen Slots only. I don't understand what the plan was
> here, but I switched to using them for all slots in the environment.
> Will this cause problems? This means that EvalContext's outers only
> needs to contain values defined in the E code, not values from the
> environment.

A quick glance at the EoJ source suggests this is reasonable. E-on-CL  
already does this type of optimization. Make sure that there are no  
user-visible differences in the behavior of the object gotten from &foo.

> 4. Removed ScopeLayout from EEnv
>
> Now, an EEnv is just a map from names to Slots, plus a fqnPrefix.

It also must track which bindings were made in the innermost scope,  
and discard that information upon nestOuter().

Remember, Envs must contain exactly the information such that it is  
possible to write a correct but inefficient E interpreter with the  
signature (for expressions):
  eval(expr :EExpr, env :Env) :Tuple[any, Env]

> Note this causes a slight change:
>
> def x := 4
> e`x := 5`.eval(safeScope.withSlot("&x", &x))
>
> Before, this gave the run-time error "Final variables may not be  
> changed"
> Now, it gives the compile-time error "Can't assign to final  
> variable: x"

It's OK for this to change. Insofar as there is a problem here, it's  
that the static detection exists at all; it should be a warning so  
that E program semantics are not dependent on whether an entire  
program or a fragment was evaluated.

> 5. Added ScopeLayoutEnv, to avoid creating lots of NounPatterns
>
> This layout just holds a reference to the EEnv and creates the  
> Patterns
> from that on demand, rather than creating all the patterns at the  
> start.

I'm not familiar enough with the EoJ evaluator to comment on this from  
description.

> The overall effect of these changes is that safeScope is now an EEnv,
> whose state is simply:
>
> public class EEnv implements EIteratable {
>   private final ConstMap myOuters;
>   private final String myFqnPrefix;
>
> This means that EEnv's bindings are clearly immutable (unlike Scope,
> which is mutable but tries not to let E code see this, leading to the
> previously-reported security issue).
>
> It should therefore be safe to give safeScope to E code, without  
> relying
> on taming.

Agreed. But note that (a) nestOuter must be supported; (b) safeScope  
should be renamed to safeEnv at the same time as the rest of this  
faint API upheaval.

Also, generally regarding Env, it should have provision for supporting  
the bindings used for guard-based auditing; take a look at the methods  
already present in E-on-CL for an example. (Though, it would be worth  
going over what we need to support and designing a clean interface for  
all implementations to support.)

> This should make the API easier to understand (e.g. it's now clear  
> that
> you can't pass locals to EExpr.eval, because EEnv can't contain any).

I don't understand what you mean by "passing locals". This code must  
succeed (even if the API changes, there must be some equivalent):

  var x := 1
  someExpr.eval(safeEnv.withSlot("x", &x))

> - Clean up EEnv API to avoid adding and stripping "&" all over the
> place?

Yes. String munging in environment interfaces is unsafe, messy, and  
slow. If we decide that it's convenient to have interfaces that take  
values ordinarily and slots optionally, it should be done using a non- 
string type; e.g. instead of the current

  makeScope.fromState([
    "x" => x,
    "&y" => &y,
  ], fqnp)

have

  makeScope.fromState([
    "x" => x,
    slotname("y") => &y,
  ], fqnp)

or perhaps even:

  makeScope.fromState([
    epatt`x` => x,
    epatt`&y` => &y,
  ], fqnp)

> Does this look like a reasonable approach?

See above for details :)

-- 
Kevin Reid                                  <http://switchb.org/kpreid/>