[cap-talk] Scholarship of P-1935

[David Hopwood]

Jed Donnelley wrote:
> Regarding P-1935 and more generally the TCSEC, I wonder
> why that process wasn't more 'scholarly' and include
> wider review.  I wasn't involved, but seeing so many
> names on it I expect the authors believed those people
> constituted a sort of "review".
> 
> Do you know enough about what they went through to
> describe how their process differed from a scholarly
> review?
> 
> I'm just curious.  From my perspective it seems that
> so much damage was done by P-1935's interpretation
> in the Orange Book that I wonder if their adopting
> a more formal review process might have helped?

I am extremely doubtful that any practical review process is sufficient to
prevent bad papers from being published.

At the end of the day, the progress of a field like computer access
control cannot be allowed to rely on the absence of published papers
that contain serious errors. Other fields do not rely on that. The
ultimate responsibility lies with readers of a paper (including other
authors who cite it) to treat it with an appropriate degree of skepticism,
regardless of where it is published or what kind of review it has had.
IMHO the main purpose of academic review is to save readers' time by
giving them *fewer* bad papers to read, and to help maintain the
reputation of journals; not to allow readers to be completely credulous
or to accept argument from authority.

(Note that I'm referring here to papers, and not to standards like TCSEC
itself that are sponsored by governments and/or official standards bodies.
Even though the latter should also be treated with appropriate skepticism
by their potential users, there is a case for actively *preventing*
publication of a bad standard that is much stronger than for a bad paper.)

-- 
David Hopwood