Backing up one more step...

[Jonathan Shapiro]

   For example,
   all the data, processes, keys, etc. belonging to a particular user might
   be isolated in one persistence district.

I didn't understand it this way, and there are advantages to what you
propose, but it doesn't change the problem I raised.  You can
construct persistence districts either statically or dynamically --
the same problems arise.

Consider an individual user's process in persistence district A.  It
takes a page fault, and needs to acquire a new page.  Two
possibilities devolve:

   a) Space Bank is within district A.  No problem.

   b) Space Bank is in some other district B.

For "Space Bank" you can substitute any object you like.  The issue of
importance is that a message crossed from one district to another.

To invoke the Space Bank, the domain (or some domain acting on it's
behalf) must send a message that passes from district A to district B.
The instant the message crosses the boundary both districts must be
tied so that they checkpoint together at the next checkpoint.  The
effect is that districts must be dynamically combined, which is NOT at
all straightforward.  It quickly defeats the benefits of districting.

Strictly speaking, the dependency isn't quite that bad, but exploiting
the real dependency makes things worse rather than better.  For all
districts A and B, if a message goes from district A to district B,
there are two possibilities:

	1) A checkpointed more recently than B

	=> When EITHER A or B next checkpoints, the other must
	   checkpoint.

	2) B checkpointed more recently than A
	
	=> If B subsequently checkpoints, A MUST be checkpointed to
	   ensure that the message is not lost on rollback.

It's been a long time, and I may not have recollected the consequences
correctly.

To see the problem, think through what happens if either district
fails after the message is sent.  In particular, you must ensure that
both districts involved in the message agree about whether the message
has been sent in the event of rollback.  If you start thinking this
through, you'll discover that rollbacks cascade in remarkably ugly
ways unless you introduce message replay logic.

Once again, I'ld point you to the Virtual Time paper.  I'll try to
find my copy and refer you to the journal.


Jonathan