Message Passing and Segment Faults

[Jonathan Shapiro]

   You're basically just talking about DSM here.  Are you assuming
   sequential consistency?  If so, note that it's generally been found
   that sequentially-consistent DSM is pretty much useless for most
   practical purposes - it's just plain too slow.  If you want DSM
   to be useful at all, you really have to allow for different
   consistency models and protocols, such as release consistency.
   Have you thought about this at all yet?

I haven't given it a lot of thought in a couple of years.  I'm
planning to leverage the work of other people here at Penn who have
thought about it a lot (Ron Minnich, John Schaffer).  On the other
hand, the problem isn't all that different from cache coherency
management in a large multiprocessor, which I know a fair bit about.
The issues, and even the ratio of access time to coherency management
latency, are similar.

   If the above are your primary reasons for the "core data frame" model,
   which seems to be causing a lot of problems and complexity,
   then I think you may be barking up the wrong tree.  DSM research
   has shown that it doesn't really matter a whole lot whether you have
   4K pages or 16K pages or whatever - any page size bigger than a few
   words is generally "too big" for most uses.  The only real solution,
   outside of hoping for processors that support ludicrously small pages,
   is to use smarter protocols...

You may be right, and I'll freely acknowledge that I'm not up to speed
on this issue.  My impression, however, is that your statement
oversimplifies things a bit.

DSM research has found that page size choices between 4k and 16k don't
matter between small numbers of machines (< 10) over *local* networks
whose link speed is 7 Mbits or higher, when running commodity
protocols (IP) not designed for block transfer, on platforms whose
protocol implementation make several copies of the data between the
application and the network card, only the last of which is to work
around the absence of decent hardware assist (cards that lack even
vaguely intelligent DMA subsystems and controllers).

However, the cards are getting smaller, and most of the kernels it has
been tried on were not designed for DSM from the start.  I suspect
that it matters, though I may be wrong.

It's true that the page size doesn't matter between 2 or 3 machines if
we're talking about ethernet speeds or faster.  Between 20 machines
the ethernet saturates, and the packet size is a factor in determining
the saturation point.

Over an ANS/DNS line or a 14.4 Kbps modem line page size matters quite
a lot.  I'ld move 512 byte chunks if the hardware made it worthwhile.

On smaller machines, even with a fast link, core pressure remains an
issue.  Transferring 16k pages into a PC with 8 mbytes of memory is
going to hurt.  The faster your link is the more it's going to hurt,
and the more the small machine will impede larger machines that call
back the pages due to false sharing.

It may turn out that DSM has been so expensive partly because previous
implementations have *not* adopted a core data frame model, which has
made it difficult for them to do channel-style virtual I/O.

   If you haven't already, you might want to read John Carter's Munin paper:
   ftp://mancos.cs.utah.edu:/papers/munin.ps.Z.

I'll certainly have a look.  Thanks for the pointer.