Backing up one more step...

[Jonathan Shapiro]

   I wasn't proposing to try to maintain consistency automatically between
   different districts - just keep them completely independent.  As I said
   in my first posting, _all_ connections between two districts get severed if
   _either_ of them dies and restarts.

I understood.  That was why I chose the space bank as my example.
Keeping them independent is impossible in the KeyKOS architecture.  At
the top of the world there will be the Prime Space Bank that all
domains, directly or indirectly, derive from.

If you propose to change this, you haven't got a single system image
anymore, because what you've done is said "these subsystems are not
allowed to communicate with each other through the persistence layer."
Actually, such a design makes a lot of sense, and my end conclusion in
EROS was that if you bought in to the persistent single level store
then the best way to implement a distributed system was as
higher-level protocols built on the low-level individual machines.

It's not simple to specify what a "well defined" connection between
districts is, particularly given that there are lots of services you
*want* to have span the boundary.  Factories, for example, shouldn't
need to be replicated.

Recovery in distributed systems is possible largely because it is
accomplished using high-level semantics.  The very low level nature of
KeyKOS components (Pages, Nodes) makes it difficult to get a handle on
the semantics at the right level of abstraction.  Simultaneously, the
rich connectedness of the KeyKOS world does not lend itself to the
sort of isolation you are discussing.

Actually, the problem is not confined to persistent systems.  Consider
that any TCP/IP connection has the same properties.  The connections
are far enough away from the general consistency management, however,
that the loss does not much impact things in practice.  Applications
communicating over such a connection are expected to manage the time
travel issue using transactions or some such. 

Jonathan