[cap-talk] evaluating tools

[zooko]

On Dec 16, 2007, at 6:12 PM, David Wagner wrote:

> Well, "dreadful" is rhetoric that appeals to emotion rather than  
> reason.
> I think it's a good idea to quantify these claims.  How much slower  
> is a
> Java-based web server, using metrics that are an appropriate  
> measure of
> end-to-end web server performance?
>
> It's my impression that Java is arguably the dominant platform for web
> services, in the enterprise world.  How do you reconcile that with  
> your
> view of Java?

This may sound flippant, but I write with consideration and relevant  
experience:

I reconcile it by believing that the dominant platform for web  
services in the enterprise world is a technically inferior platform.

One shouldn't think that market dominance implies technical  
superiority.  Neither, of course, should one assume that technical  
superiority is the most important criterion for your decisions.  If  
you want to contribute to the development of the world's enterprise  
web services platform, then by all means you should use Java to do so.

It would be really good if we could quantify claims such as "Tool X  
is technically superior for task Y.".  Presumably we could do that  
using the decades-old practices of controlled trials, such as are  
used in psychology, human factors engineering, and so on.   
Unfortunately such an experiment would be expensive, and few people  
have attempted to run one.  Here is the best example that I have found:

"An empirical comparison of C, C++, Java, Perl, Python, Rexx, and Tcl  
for a search/string-processing program" (2000) by Lutz Prechelt
http://citeseer.ist.psu.edu/311372.html

It is unfortunate that designers of programming languages often  
consider themselves to be working in the fields of mathematics or  
computer science, and they use the practices and tools of those arts,  
when we would be better served if they thought of themselves as  
working in psychology and used the standards of measurement required  
for publishing results in *that* field.

Regards,

Zooko

P.S.  Why is Python so easy to learn and to use?  Perhaps because its  
immediate ancestor ABC went through many iterations of semi- 
controlled, observed user testing during its design.