Heterogeneity

[Jonathan Shapiro]

I'm stumped about something.  It's related to the ongoing memory
object discussion.  Actually, I've come to the conclusion that calling
it a memory object was a really unfortunate choice of name.  It
implied that the only way to access such objects was by mapping them,
which was never my intention.  I'm going to try to call them data
objects from now on.  We can then distinguish between the semantics of
data objects and the semantics of mappings of data objects.

In any case, here is today's trivia challenge question:

Imagine that we are interested in implementing a single system image
over a heterogeneous cluster of machines.  For the moment, assume that
we will constrain things so as to insist on compatible alignment, byte
sex, and structure packing rules.  That is, two distinct processors
running distinct code bases can sucessfully access a shared region of
memory and manipulate it.

  [ Aside: I believe that I know how to handle byte-sex
    incompatibilities, which makes this somewhat more realistic ]

Now assume that we are sharing a region of memory, and that the
processors do not implement the same page size.  For simplicity,
assume that all page sizes are an integral power of two multiple of
some lesser size, which I shall call the distributed data frame (DDF)
size.

Note that in such a system architecture, it is quite possible to share
data, but not straightforward to share mapping restrictions simply.
If one machine says "make page 3 of the object be read-only," it is
not clear what this should mean on machines with incompatible page
sizes.

Modifying the notion to protection over DDF units doesn't help.
Suppose that the DDF size is 4K, but the host page size is 16k, and
that DDF 3 of an object has been made read-only.  On the 16k page
machine, there exists a page whose content is:

	DDF 1 :: DDF 2 :: DDF 3 :: DDF 4	:: is concatenation.

What should the protection attributes of such a page be?  How should
the page-level protections change in response to DDF-level
protections?

I'ld appreciate some discussion of what should be done here and how.
I understand how to implement various hacks that would do most of the
right thing (e.g. periodically verifying that DDF 3 has not actually
been modified while making the page read-write).  All of them admit of
race conditions that seem unhealthy.  Implementing the policy that the
page protections are the most restrictive of the composed DDF
protections seems like a bad idea, as it creates the opportunity for a
variety of truly obscure bugs.


Interpreting memory references is feasible, but I'll dismiss it
without further discussion.


Unless I have missed something fairly obvious, it seems that in such a
system the minimal unit of protection must be the data object rather
than the DDF.  A process should be able to map an object with reduced
authority, but the MAPPING is not distributable.

It's clearly necessary to define a sub-object locking semantics, but
such locking it doesn't interact very well with memory mapping.  I'm
therefore inclined to think that the memory mapping interface is an
inappropriate place to implement it, and that mappings are orthogonal
to object protections.


I haven't thought it through, but my gut says that this alters the
semantics for object composition as well.  If the permissions of a
composed object are the most restrictive of the component objects,
then all is well.  Any composition involving mixed permissions admits
the possibility that a permission change might not fall at an object
offset that is a modulus of the native page size, and might therefore
not be enforceable on a particular choice of hardware.  Since it's
hard to predict future hardware, this seems to be an issue that
doesn't admit of obvious long-term solutions.

If you buy this, then the binding between KeyKOS's segment of pages
and protections needs to be substantially weakened.  You can still
build a tree of DDF units, but protections must work differently.

A corollary is that faulting data into/out of a data object cannot in
general be done in terms of pages, because there isn't a uniform
notion of pages.

Actually, this leads me to believe that DSM at the system level may be
a bad idea.  It's very useful for applications, but I'm starting to
think that it's an inappropriate substrate on which to implement
primitive services.


Reactions?


Jonathan