Build Problems Resolved
Mark S. Miller
markm@caplet.com
Tue, 31 Aug 1999 12:48:58 -0700
--=====================_66720899==_.ALT
Content-Type: text/plain; charset="us-ascii"; format=flowed
At 04:57 PM 8/30/99 , Paul Snively wrote:
>Folks,
>
>I have to ask this; please forgive me.
>
>Is it time to rewrite E?
You are forgiven ;)
In short, it is always time to rewrite E, as well as most living imperfect
code. But one must distinguish between two sorts of rewrite.
In this last year, I believe we've achieved a sufficiently modular
architecture that (with one exception noted below) there is no need to
start from scratch. E does not need a big monolithic rewrite. Rather, all
the pieces of E can, and should, be rewritten over time. As each
replacement comes to be seen as superior to the corresponding existing
component, the replacement can happen in-place. No rewrite task need any
longer to take on the E system as a whole.
If this is true, then I have done my first job as open source coordinator:
bringing the system to the point where other individuals can incrementally
improve the system, often by wholesale replacement of individual pieces,
and without much coordination with work happening elsewhere in the
system. However, I hope you understand that I don't have the time to do
all the required replacement myself. It would be wonderful if some of the
desire for a rewrite turned into a desire to do some of the rewriting
;) This would bring me to my second job as open-source coordinator --
integrating in such submitted replacements.
> It seems to me that there are several goals that have already been
> expressed on this list that might best be addressed by a top-to-bottom
> rewrite of E, incorporating support (as best we can at this point) for
> all of the desiderata:
So let's go through this list, and see if it indeed needs a "top-to-bottom"
rewrite, or whether a much less risky bazaar-process-compatible
bottom-to-top rewrite will do.
> 1) Reintroducing persistence--of particular significance, perhaps, now
> that ColdStore is in beta.
This is probably the element on your list with the greatest risk of
requiring surprising non-modular changes elsewhere in E. We are not
currently planning to use ColdStore or any other native solution. As for
what we are planning, Tyler, would you care to explain?
>2) Replacing the non-open-source OROMatcher (which, frankly, I suspect to
>be at the root of at least the Visual Age problem if not also the JIT
>problem) with the GNU Java regex package.
Replacing OROMatcher with another jvm-based regex package should be as
modular as putting OROMatcher in was in the first place. Where do I find
the GNU Java regex package? Does it do Perl5 regular expressions (not a
requirement, but it would be good)?
> 3) Replacing the rather byzantine byaccj process with a grammar etc.
> built around ANTLR. Using the tree support in ANTLR might lead us directly to:
Parsing the string of characters into a parse-tree should be, and I believe
is, orthogonal from how/whether one generates code from those trees. I
would love for someone to replace the byaccj parser with an ANTLR-based
parser, and I believe the result could be slid in without effecting much
else. Separately, I have been planning to generate jvm bytecodes from my
parse-tree for a while now. I'm perfectly receptive to the notion that
ANTLR's tree-grammar stuff might provide a better framework for doing
so. Again, I don't see the change as effecting much elsewhere.
Btw, a more valuable exercise would be to integrate in ANTLR as a
quasi-parser generation tool, such that its use to create an E-grammar
quasi-parser was just one case of that. Were this done, then one should be
able to turn any existing ANTLR grammar into a quasi-parser for that
grammar easily.
>4) Bytecode-compiling E rather than interpreting. Apart from leveraging
>ANTLR for this, we might also want to look at the open-source GNU Java
>bytecode-generation library written by Per Bothner at Cygnus as part of
>his Kawa implementation of Scheme in Java.
Worth a look. Please gimme a pointer.
> 5) While we're compiling E, let's add support for multiple targets: Java
> bytecode + ELib being the first, of course, and some sort of runtime
> (ColdStore etc.?) atop EROS being the second.
The E *language* itself could easily be retargetted to a non-jvm platform
such as Squeak, or perhaps ColdStore. After all, E is an almost minimal
lambda language. However, there is the huge matter of the libraries. E
benefits tremendously by being able to inherit the Java libraries and
provide them to the E programmer almost as if they had been written for
E. Other platforms such as Squeak or Python also have rich libraries it
would be good to inherit, but one must choose which of these provide the
default definition of the standard E libraries. For various reasons, we've
chosen Java's libraries.
Having made this choice, it would be too much work to build compatible
libraries on platforms in which the Java libraries were not available as a
starting point. Notice that this does *not* pin E into compiling to jvm
bytecodes. In a system such as Cygnus's, where jvm code is compiled to the
runtime formats compatible with corresponding C++ code, and where they've
already done the work to reproduce the java libraries, we could compile
into a native form that used these libraries.
This was as good as I thought it could get until Tyler made a crucial
point: A major motivation for E is mobile confined code, especially for
Smart Contracting. A confined Smart Contract does not need, and typically
will not be given access to, many of these Java libraries. The normal
environment for a confined Smart Contract will include important
preexisting libraries, but probably a small number, and probably not
inherited from Javasoft's work. Once we've carefully defined this
environment, we will probably find that we can easily reproduce it on other
platforms.
So E isn't easily portable to non-java platforms, but the Confined-E subset
of E should be.
I'd love to see E running on EROS, but this should simply follow from a
port of the underlying implementation platform (such as a jvm) to
EROS. Once E is on EROS, EROS's capabilities can be made to show up in E
as E capabilities. I fantasize about E being EROS's shell language.
Cheers,
--MarkM
--=====================_66720899==_.ALT
Content-Type: text/html; charset="us-ascii"
At 04:57 PM 8/30/99 , Paul Snively wrote:
Folks,
I have to ask this; please forgive me.
Is it time to rewrite E?
You are forgiven ;)
In short, it is always time to rewrite E, as well as most living
imperfect code. But one must distinguish between two sorts of
rewrite.
In this last year, I believe we've achieved a sufficiently modular
architecture that (with one exception noted below) there is no need to
start from scratch. E does not need a big monolithic rewrite.
Rather, all the pieces of E can, and should, be rewritten over
time. As each replacement comes to be seen as superior to the
corresponding existing component, the replacement can happen
in-place. No rewrite task need any longer to take on the E system
as a whole.
If this is true, then I have done my first job as open source
coordinator: bringing the system to the point where other individuals can
incrementally improve the system, often by wholesale replacement of
individual pieces, and without much coordination with work happening
elsewhere in the system. However, I hope you understand that I
don't have the time to do all the required replacement myself. It
would be wonderful if some of the desire for a rewrite turned into a
desire to do some of the rewriting ;) This would bring me to my
second job as open-source coordinator -- integrating in such submitted
replacements.
It seems to me that there
are several goals that have already been expressed on this list that
might best be addressed by a top-to-bottom rewrite of E, incorporating
support (as best we can at this point) for all of the
desiderata:
So let's go through this list, and see if it indeed needs a
"top-to-bottom" rewrite, or whether a much less risky
bazaar-process-compatible bottom-to-top rewrite will do.
1) Reintroducing
persistence--of particular significance, perhaps, now that ColdStore is
in beta.
This is probably the element on your list with the greatest risk of
requiring surprising non-modular changes elsewhere in E. We are not
currently planning to use ColdStore or any other native solution.
As for what we are planning, Tyler, would you care to explain?
2) Replacing the non-open-source
OROMatcher (which, frankly, I suspect to be at the root of at least the
Visual Age problem if not also the JIT problem) with the GNU Java regex
package.
Replacing OROMatcher with another jvm-based regex package should be as
modular as putting OROMatcher in was in the first place. Where do I
find the GNU Java regex package? Does it do Perl5 regular
expressions (not a requirement, but it would be good)?
3) Replacing the rather
byzantine byaccj process with a grammar etc. built around ANTLR. Using
the tree support in ANTLR might lead us directly
to:
Parsing the string of characters into a parse-tree should be, and I
believe is, orthogonal from how/whether one generates code from those
trees. I would love for someone to replace the byaccj parser with
an ANTLR-based parser, and I believe the result could be slid in without
effecting much else. Separately, I have been planning to generate
jvm bytecodes from my parse-tree for a while now. I'm perfectly
receptive to the notion that ANTLR's tree-grammar stuff might provide a
better framework for doing so. Again, I don't see the change as
effecting much elsewhere.
Btw, a more valuable exercise would be to integrate in ANTLR as a
quasi-parser generation tool, such that its use to create an E-grammar
quasi-parser was just one case of that. Were this done, then one
should be able to turn any existing ANTLR grammar into a quasi-parser for
that grammar easily.
4) Bytecode-compiling E rather
than interpreting. Apart from leveraging ANTLR for this, we might also
want to look at the open-source GNU Java bytecode-generation library
written by Per Bothner at Cygnus as part of his Kawa implementation of
Scheme in Java.
Worth a look. Please gimme a pointer.
5) While we're compiling E,
let's add support for multiple targets: Java bytecode + ELib being the
first, of course, and some sort of runtime (ColdStore etc.?) atop EROS
being the second.
The E *language* itself could easily be retargetted to a non-jvm platform
such as Squeak, or perhaps ColdStore. After all, E is an almost
minimal lambda language. However, there is the huge matter of the
libraries. E benefits tremendously by being able to inherit the
Java libraries and provide them to the E programmer almost as if they had
been written for E. Other platforms such as Squeak or Python also
have rich libraries it would be good to inherit, but one must choose
which of these provide the default definition of the standard E
libraries. For various reasons, we've chosen Java's
libraries.
Having made this choice, it would be too much work to build compatible
libraries on platforms in which the Java libraries were not available as
a starting point. Notice that this does *not* pin E into compiling
to jvm bytecodes. In a system such as Cygnus's, where jvm code is
compiled to the runtime formats compatible with corresponding C++ code,
and where they've already done the work to reproduce the java libraries,
we could compile into a native form that used these libraries.
This was as good as I thought it could get until Tyler made a crucial
point: A major motivation for E is mobile confined code, especially for
Smart Contracting. A confined Smart Contract does not need, and
typically will not be given access to, many of these Java
libraries. The normal environment for a confined Smart Contract
will include important preexisting libraries, but probably a small
number, and probably not inherited from Javasoft's work. Once we've
carefully defined this environment, we will probably find that we can
easily reproduce it on other platforms.
So E isn't easily portable to non-java platforms, but the Confined-E
subset of E should be.
I'd love to see E running on EROS, but this should simply follow from a
port of the underlying implementation platform (such as a jvm) to
EROS. Once E is on EROS, EROS's capabilities can be made to show up
in E as E capabilities. I fantasize about E being EROS's shell
language.
Cheers,
--MarkM
--=====================_66720899==_.ALT--