[cap-talk] Scholarship of P-1935

[Jed Donnelley]

At 06:40 PM 2/2/2008, David Hopwood wrote:
>Jed Donnelley wrote:
> > Regarding P-1935 and more generally the TCSEC, I wonder
> > why that process wasn't more 'scholarly' and include
> > wider review.  I wasn't involved, but seeing so many
> > names on it I expect the authors believed those people
> > constituted a sort of "review".
> >
> > Do you know enough about what they went through to
> > describe how their process differed from a scholarly
> > review?
> >
> > I'm just curious.  From my perspective it seems that
> > so much damage was done by P-1935's interpretation
> > in the Orange Book that I wonder if their adopting
> > a more formal review process might have helped?
>
>I am extremely doubtful that any practical review process is sufficient to
>prevent bad papers from being published.
>
>At the end of the day, the progress of a field like computer access
>control cannot be allowed to rely on the absence of published papers
>that contain serious errors. Other fields do not rely on that. The
>ultimate responsibility lies with readers of a paper (including other
>authors who cite it) to treat it with an appropriate degree of skepticism,
>regardless of where it is published or what kind of review it has had.
>IMHO the main purpose of academic review is to save readers' time by
>giving them *fewer* bad papers to read, and to help maintain the
>reputation of journals; not to allow readers to be completely credulous
>or to accept argument from authority.
>
>(Note that I'm referring here to papers, and not to standards like TCSEC
>itself that are sponsored by governments and/or official standards bodies.
>Even though the latter should also be treated with appropriate skepticism
>by their potential users, there is a case for actively *preventing*
>publication of a bad standard that is much stronger than for a bad paper.)

It is exactly the above parenthetical paragraph that
has me feeling that the process for P-1935 and the
Orange book might be worth review.  They did result
in what amounted to a government "standard" (call
it best practice if you wish) and yet certainly
the suggestion I got from Jonathan previously is that
he didn't consider the review up to "scholarly"
standards.

 From what you write above, it seems to me that such
writing directed at something like government policy
should, if anything, be held to higher standards
than those of a "scholarly" paper.  The consequences
of recommended policy are typically higher.

Is the situation today better?  Are there still such
government policies being established with such
writing?  Are the reviews better or worse or ???
Perhaps AlanK might comment here?

I hope I'm not wasting people's time.  I spent so
much time looking into what P-1935 "really" was
saying and investigating how in fact one can
meet the challenges that they were concerned about
with "traditional" capability systems (supporting
the insertion property and emulation of object
references), that I hope we don't have to go
through such a situation again.  Perhaps if
there had been a more thorough review then (as
we are doing now), then we wouldn't have cost
the IT industry/community so much (loss of POLA
for so many years)?  Maybe if we can assure more
thorough review in future we can avoid such
problems?

Is making such reviews more "scholarly" likely
to help?  If so then we might be able to make
such a recommendation (where?).

I'm of course willing to drop this if people
feel it's a pointless effort to "fight city
hall."  I just wanted to see where it led.

--Jed  http://www.webstart.com/jed-signature.html