[cap-talk] "ACLs don't" paper rejected from Oakland 09

[David Wagner]

Tyler Close  wrote:
>I thought I had written an evaluation of the ACL model, and its
>derivatives; studying whether it actually provides the functionality
>that is commonly claimed. Saying the paper is on broad principles
>seems to imply a paper that declares certain properties are "good" and
>provides scant justification for it. That's not the paper I thought I
>wrote. The studied features are ones the ACL model has declared to be
>good, not me. I then use concrete examples to study whether or not the
>claimed features are actually implemented. In many cases, I succeed in
>showing that not only are the features not implemented, but they
>cannot be implemented given the input data the ACL reference monitor
>has. In the second part of the paper, I go on to show that these
>failings aren't theoretical curiosities, but severe problems in the
>dominant computing platform in the world.

I'm not sure how to answer.

I feel like there is a category difference between your paper and the
typical submission to these conferences.

When we have a submission that argues "It's possible to build a system
to do X and get performance at such-and-such a level", one can read the
performance measurements in the paper and see whether the system meets
the claimed performance.

When we have a submission that presents a theorem, one can read the
proof and see whether the theorem has indeed been proven.

Both of these kinds of submission are relatively easy to evaluate,
compared to the kind of paper you've got.

In contrast, your paper argues for some design principles, and supports
its claims by examples and arguments.  Nothing wrong with that, but it
seems more challenging to evaluate than a "we built it" or "we proved
it" theorem.  How do we know whether you overlooked some consideration?
How do we know whether you're focusing on the right issues?

And, when reviewers look at the paper, I can imagine they may be thinking:
"OK, I could take a "we built it" or a "we proved it" paper, and the
contribution is clear but narrow; or I could take this "principles"
paper, which might possibly turn out to be very important but might
also be a disaster; should I take the sure thing, or gamble on this
"principles" paper?".  And reviewers are often pretty conservative.

To put it another way, reviewers are being asked to accept a paper
that's awfully different from what they're used to accepting, and
being asked to evaluate a paper that's different from what they're
used to evaluating.  They're being asked to accept a paper without much
"raw meat" (i.e., development of technical detail; proofs of theorems;
hardcore measurements of new systems; etc.).  That's asking reviewers
to go against their grain and taking them out of their comfort zone.
That's a tough position to be in.

This is just a personal opinion.  I don't equate "hard for reviewers to
evaluate" with "of poor merit".  Some of my favorite papers I've ever
read have been "principles" papers.  For instance, the paper introducing
and arguing for the end-to-end principle is a classic systems paper that
has had significant influence, and certainly impressed me when I first
read it.  If I remember my anecdotes correctly, that paper was rejected
twice before being eventually accepted, and was controversial at the time.

I suspect I'm not really answering your question.