Hardware for EROS?

[Kragen Sitaker]

Mark Miller writes:
> I agree.  The military seems to be interested in the exotic technology
> because they want instant-on.  I don't care about instant-on; but for
> distributed computing, I'm desperate for low latency stable storage so that
> I can do instant commit.  Then, I could afford to have machine X not release
> message M until the state that had computed message M had been committed to.
> 
> If a disk seek is the cost of a commit, then many realistic distributed
> computing scenarios cannot wait that long before releasing a message.

"low latency stable storage" can take many forms, with varying kinds of
stability.

The Rio guys, who also built Vista, just use DRAM as their "stable
storage".  When the OS crashes, they write the Rio area to disk before
rebooting.  They claim this works extremely well in their tests (in
which they simulated software failures by randomly flipping bits) ---
in fact, after they did some VM twiddling to protect the Rio area from
random kernel writes, they claimed it was more reliable than just
writing stuff to disk.

This obviously has problems if you're on flaky hardware that can crash
without giving the OS this chance to recover.  It also obviously has
problems if someone can press the power button.

The NetApp guys have a couple of megabytes of RAM --- can't remember
what kind, but probably SRAM, so mucho dinero --- on a card with two
batteries to sustain it through power failures.  They commit their
"disk writes" to there.

A third alternative --- probably not reasonable if you want <10ms
latency, but possibly reasonable if you want high throughput --- is to
use the RAM of several geographically-separated machines connected via
the Internet as your "stable storage".  As long as they don't all lose
power simultaneously or close to it, and as long as you don't have any
non-fail-stop failures that corrupt your data, this is probably more
reliable than using a disk.

(Non-fail-stop failures that corrupt your data can happen even if you
are using a disk.)

I don't know if this has been tried; it is likely to have unforeseen problems.

If your network is fast enough, this alternative could be faster than
using a disk.


By the way, I'm interested to hear about your distributed-computing
scenarios that require stable commitment.

-- 
<kragen@pobox.com>       Kragen Sitaker     <http://www.pobox.com/~kragen/>
The Internet stock bubble didn't burst on 1999-11-08.  Hurrah!
<URL:http://www.pobox.com/~kragen/bubble.html>
The power didn't go out on 2000-01-01 either.  :)