[cap-talk] request for comments on capability design

[Sandro Magi]

Peter Amstutz wrote:
> On Thu, May 17, 2007 at 09:29:56AM -0400, Sandro Magi wrote:
> 
> Because prioritizing data transmission this way reaches down to the 
> lowest level of packet scheduling (effectively it requires implementing 
> application-level QoS), it requires awareness by the rest of the system 
> all the way up to the application layer.

I'm pretty sure you can implement a scheduler to prioritize message
sends; Waterken server is FIFO at the moment. I'm not sure what criteria
you use, so there might be a number of ways to achieve this.

Given your description, the problems with using the web-calculus as-is
lie in, a) scaling/concurrency, b) message verbosity, c) durability
guarantees.

In terms of reducing bandwidth, you'd want a highly compressed, binary
web-calculus document format; I'm not sure how tight the current binary
serialization format is as I've never used it. Not sure there's much
that can be done about this.

Regarding server-side scalability: the Waterken server is currently
based around isolated, single-threaded event loops, like E Vats or
Erlang lightweight processes. This means message processing is atomic,
execution is serialized within an event loop, and there is no shared
state, resulting in few concurrency bugs. Scaling is accomplished by
distributing objects among multiple event loops, each of which can run
in their own OS threads.

I've read that each in-game object is an actor; this corresponds to the
case of one object per event loop. That seems a bit much, and I imagine
you can get away with some happy medium. Clearly running this many OS
threads is still undesirable, but you can view an event loop as a
process and write a scheduler to dispatch OS threads from a pool for
event loops that have pending messages. If your system calls are
asynchronous, you run #threads=#cpus, and you achieve optimal CPU
utilization.

The downside is that distributing objects among event loops is a manual
procedure, and cannot currently be inferred automatically. Many
applications have "natural concurrency" boundaries, but getting the
right segregation can still be tricky.

The web-calculus also requires some durability guarantees, where the
reply is not sent to the client until the changes have been written to a
durable medium. Depending on the nature of the application, this
requirement may be less important.

The above assumes that all the processing is done server-side, and the
clients are dumb terminals that simply display the results. Smart
clients that make predictions and then reconcile with server replies
will improve latency dramatically.

That's what I can think of off the top of my head for scaling the
web-calculus.

Another unexplored avenue which other game developers have considered is
Software Transactional Memory (STM); it allows you to design the
application in a more traditional multi-threaded style while retaining
safety. There were some interesting discussions comparing the
scalability of event loops and STM on this list and off [1][2].

> I am familiar with Croquet, and certainly interested in borrowing some 
> of their better ideas.  However, I'm not sure if they've really tried to 
> tackle the question of security beyond collecting requirements as 
> documented on the page you linked to.  Their distributed computing model 
> also seems to make the notion of "ownership" very ambigious, so I'm not 
> sure how you would go about specifying that only a subset of 
> participants in a Croquet space can modify some shared object.

I'm not too familiar with it, I just know Mark Miller (on this list) has
had extensive security discussions with them in the past.

Sandro

[1] http://lambda-the-ultimate.org/node/1277
[2] "A game developer presents needs for future language",
http://www.eros-os.org/pipermail/e-lang/2006-February/thread.html#11067