Re: A proposal for a General Clustering Framework

[Alan Robertson <alanr@unix.sh>]

Bill Todd wrote:
> 
> An aspect that I haven't yet seen covered (though I might have missed it) is
> the efficient processing of large chunks of data - e.g., file/database pages
> (which these days can easily reach 128 KB in size and in the future might
> reasonably get even larger - in many cases, any size where the seek
> time/rotational latency eclipses the transfer time can be 'reasonable') and
> even larger chunks of streaming data.
> 
> Facilities like (Illinois) Fast Messages at least potentially allow (FM's
> ability may depend upon the implementation and/or its whims of the moment,
> but one can get down-and-dirty in, e.g., NT and provide stronger guarantees)
> the receiver of such data to use the start of the message to recognize the
> contents and identify an appropriate receptacle (e.g., pre-allocated target
> buffer[s]) where the payload can be dumped without extraneous copy activity
> (copying being a non-negligible concern with payloads in the 100 KB -
> multi-MB range).
> 
> It may be due to simple ignorance on my part, but it's not clear to me how a
> generic XML processor would allow such optimization - or other FM-like
> facilities such as the ability to process some short messages with
> 'handlers' at interrupt level when appropriate.

The short answer to this is that this protocol is for control messages, not
bulk data transfers.  The needs of the two are somewhat different, in some
ways, radically different.  My belief is that no single mechanism can meet
the needs of the two parts well.

	-- Alan Robertson
	   alanr@unix.sh

Linux-cluster: generic cluster infrastructure for Linux
Archive:       http://mail.nl.linux.org/linux-cluster/