Beowolf machines don't have adequate IO for applications other than
things that resemble Linpack. The teraflop machine at Sandia would like
to generate output every 5 or 10 minutes, but it takes an hour to get
the computation result out of the machine. Beowolf machines tend to not
have adequate bisection bandwidth more many kinds of codes. Anything that
requires a cornerturn (transpose) is a good example. Also, Beowolf machines
tend to run a single code for many hours: they don't multiplex well,
or sometimes at all. But multiplexing and adapting to a dynamic load
is expected of many large scale machines and that requires some OS support
not found in the single-cpu or small-cpu-count OS or existing middleware.
We cannot afford to build
machines that are good only for Linpack.... it is better
to build a Beowolf cluster if that is the best fit for the job.
It is rather ignorant to think that because something like Beowolf can do some
some types of applications well, that some software work can make it do
all jobs well. That's like saying a Volkswagon could go 300 kph
if you drive carefully.
Linux has proven to be worthwhile as a node controller in an MPP architecture -
that's what a Beowolf is. But that does not make it ready for SMP nodes
that scale to large numbers. It seems wasteful to program a large scale
ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away
most of the capabilities of the hardware. That's why I don't
think it is interesting or particularly useful... unless a massive amount
of work went into rewriting the io and memory management subsystems,
not to mention scheduling, administration, etc.