[cap-talk] "ACLs don't" paper rejected from Oakland 09

[Tyler Close]

Hi David,

Wow, these last two messages of yours have been very enlightening for
me. Thank you for providing this perspective.

I think I get your basic point about paper categories, but I think
"ACLs don't" is in a different category than a "principles paper"; a
category that should also be easier to review.

More below...

On Sat, Jan 31, 2009 at 2:03 AM, David Wagner <daw at cs.berkeley.edu> wrote:
> Tyler Close  wrote:
>>I thought I had written an evaluation of the ACL model, and its
>>derivatives; studying whether it actually provides the functionality
>>that is commonly claimed. Saying the paper is on broad principles
>>seems to imply a paper that declares certain properties are "good" and
>>provides scant justification for it. That's not the paper I thought I
>>wrote. The studied features are ones the ACL model has declared to be
>>good, not me. I then use concrete examples to study whether or not the
>>claimed features are actually implemented. In many cases, I succeed in
>>showing that not only are the features not implemented, but they
>>cannot be implemented given the input data the ACL reference monitor
>>has. In the second part of the paper, I go on to show that these
>>failings aren't theoretical curiosities, but severe problems in the
>>dominant computing platform in the world.
>
> I'm not sure how to answer.
>
> I feel like there is a category difference between your paper and the
> typical submission to these conferences.
>
> When we have a submission that argues "It's possible to build a system
> to do X and get performance at such-and-such a level", one can read the
> performance measurements in the paper and see whether the system meets
> the claimed performance.
>
> When we have a submission that presents a theorem, one can read the
> proof and see whether the theorem has indeed been proven.
>
> Both of these kinds of submission are relatively easy to evaluate,
> compared to the kind of paper you've got.

OK, I get this, and I can see how it's an issue that "ACLs don't"
doesn't fit either of these categories.

> In contrast, your paper argues for some design principles, and supports
> its claims by examples and arguments.  Nothing wrong with that, but it
> seems more challenging to evaluate than a "we built it" or "we proved
> it" theorem.  How do we know whether you overlooked some consideration?
> How do we know whether you're focusing on the right issues?

"ACL's don't" doesn't propose any design principles. The paper studies
the claimed features of the ACL model to determine whether or not they
are actually implemented. The paper does this by providing concrete
examples where the claimed protection is not provided. The reviewer
should examine whether or not the ACL mechanism has been accurately
applied within the described example. If so, and the claimed
protection is not provided, then a successful counter-example has been
provided.

> And, when reviewers look at the paper, I can imagine they may be thinking:
> "OK, I could take a "we built it" or a "we proved it" paper, and the
> contribution is clear but narrow; or I could take this "principles"
> paper, which might possibly turn out to be very important but might
> also be a disaster; should I take the sure thing, or gamble on this
> "principles" paper?".  And reviewers are often pretty conservative.

A paper providing a set of counter-examples to existing claims should
be easier to review than a paper that introduces a set of principles.
In fact, I would expect the counter-example paper to be easier to
review than even the "we built it" or "we proved it" paper. The "I
broke it" paper should be the most conservative choice on the menu.

> To put it another way, reviewers are being asked to accept a paper
> that's awfully different from what they're used to accepting, and
> being asked to evaluate a paper that's different from what they're
> used to evaluating.  They're being asked to accept a paper without much
> "raw meat" (i.e., development of technical detail; proofs of theorems;
> hardcore measurements of new systems; etc.).  That's asking reviewers
> to go against their grain and taking them out of their comfort zone.
> That's a tough position to be in.

I have this mental image of a person up to their teeth in raw,
stinking, rotting meat, and screaming out: "I need more raw meat!".
And I'm thinking: "Ahh... No, what you need is someone with a really
big shovel to dig you out of that pile of rot." The "ACLs don't" paper
brings a really big shovel that clears away a lot of rot.

> This is just a personal opinion.  I don't equate "hard for reviewers to
> evaluate" with "of poor merit".  Some of my favorite papers I've ever
> read have been "principles" papers.  For instance, the paper introducing
> and arguing for the end-to-end principle is a classic systems paper that
> has had significant influence, and certainly impressed me when I first
> read it.  If I remember my anecdotes correctly, that paper was rejected
> twice before being eventually accepted, and was controversial at the time.

The end-to-end paper was indeed a principle paper, and I understand
that it therefore has a much harder case to prove before it can be
published. "ACLs don't" is not in that category; it's not claiming a
"do"; it's claiming a "don't". Lampson's "Protection" is the
principles paper. "ACLs don't" is the corresponding "I broke it"
paper.

> I suspect I'm not really answering your question.

Partly yes and partly no. The difference in categories is an issue
that I didn't appreciate the importance of. I think it's crazy (more
rotting meat please!), but I get it. We disagree about what category
"ACLs don't" is in.

Thanks,
--Tyler