[EROS-Arch] Proposed new invocation type: POST

[Jonathan S. Shapiro]

As many of you know, I have been struggling to work out how to provide some
form of asynchronous notification mechanism in EROS. Various proposals have
come and gone, none happily. I finally have one that I think I am
comfortable with.

The proposal is to introduce a new invocation type: POST. A POST invocation
transmits a specially restricted message payload, which is delivered (and
received) just as any other message is delivered. Specifically, a POST
message contains an order code and a single register value.

POST behaves exactly like SEND/FORK, with two exceptions:

    1. All POSTs are prompt. If the invoked
       capability is a start capability, and the
       recipient is not available, the POST
       message will be queued by the kernel
       for later delivery, and the invoker will
       continue.
    2. The message payload is restricted as
       described above.

As with SEND, no resume capability is generated by a POST invocation.

DATA STRUCTURES:

As the proposed implementation takes advantage of a difference between the
EROS and KeyKOS internal data structures, so I shall briefly review those
differences here.

In EROS, there is a separate Activity structure (currently mis-named thread)
that represents an executing thread of control. Every Activity is either
free or occupies a domain/process. A given domain/process can be occupied by
at most one activity.

The Activity structure is the least part of a running process that is
stalled on a kernel queue that must remain in memory in order for the
running process to be successfully restarted. It presently contains a
process key to the process root of the running process, and the last known
priority of the process. The in-memory form contains link pointers for
enqueueing and a pointer to the loaded dib/context structure if any.

Also, the EROS Context structure differs from the DIB in that it contains
the capabilities for the process. This is relevant because it is possible
(in principle) to create a Context structure that does not actually have
associated nodes. The current EROS kernel uses this for kernel processes,
and we will be re-using this trick to implement POST.

IMPLEMENTATION:

We will add a type field to the Activity structure. In addition to the
existing "running" activity, we will introduce a "posting" activity. The
"posting" activity will contain a copy of the invoked capability. We will
add two new fields to the Activity structure to contain the order code and
argument word to be delivered.

The POST operation allocates a "post" activity (as with SEND, it can block
due to available activity structure starvation). Just as an attempting to
CALL an unavailable process via a start key internally queues the calling
process's Activity structure on the destination process root, a POST
activity is similarly queued on the target process root if needed. When
these blocked activities are awakened by availability, they will be
scheduled according to the usual scheduling rules. A POST activity carries
the priority/schedule of its sender. Running POST activities sit in the
scheduler run queue just as running process activities do. An undelivered
POST is in fact a special form of running process -- this will make more
sense in a moment.

The LOGICAL implementation will proceed as follows:

1. A per-cpu Context structure will be reserved for use in the
implementation of POST. For the remainder of the discussion I shall assume a
single CPU and call this context the PostContext structure.

2. Whenever a POST activity is scheduled, the PostContext structure is
repopulated as follows:

    All capability registers hold void, except that
        cap register 1 is populated with the invoked
        capability stored in the post activity.
    All registers hold zero, except those that would
        contain the order code andfirst msg word in
        a SEND invocation. These are populated from
        the values stored in the POST activity structure
    The PostContext is made to appear as though
        it has just invoked the SEND operation.

3. The kernel processes the SEND invocation from the PostContext as though
it came from a real process.

4. In the event that this invocation stalls in the kernel, the POST activity
is queued on the stall queue and the PostContext is released for use in
other attempts to POST.


RESULT

The result of this is that we get asynchronous, single-word, guaranteed
delivery messages at the cost of possible thread structure exhaustion.
Thread structure exhaustion is already a concern that needs to be addressed
in both designs, and I believe that I see a coloring strategy by which to
achieve this (to be send later).