Re: cluster list

["Albert D. Cahalan" <acahalan@cs.uml.edu>]

Greg Lindahl writes:
> On Wed, Feb 28, 2001 at 06:47:30PM -0500, Albert D. Cahalan wrote:

>> Each node consists of a CPU, local RAM, optional flash ROM, L3 cache,
>> and a multi-purpose chip that does DMA between nodes. There isn't any
>> expansion opportunity on the node, or even a PCI chip.
>>
>> So stop assuming clusters have network cards!
>
> Your nodes are much like CPUs on the Cray T3E. Sometimes people think
> of your DMA chip as a network card; I do DMA puts and gets with
> Myrinet network cards.

To make things clear, what behaviors can you get?

1 initiator==sender, data queued on destination
2 initiator==sender, initiator chooses memory location on destination
3 initiator==destination, data queued on destination
4 initiator==destination, choosing where to put the data in advance
5 initiator==3rd-party, data queued on destination
6 initiator==3rd-party, initiator chooses memory location on destination

I get #2 and #4 between CPUs. Maybe #6 works too, if I want to risk
remote control of another node's DMA engine. All of #2, #4, and #6
would work fine with devices, including a variant of #6 where I call
myself the 3rd party.

> Devices on a system like this usually sit on
> special CPUs and you have to DMA them a request or pass a message to
> the remote OS instance in order to talk to the device, instead of
> talking to the device itself.

Not for me. I get a bridge to PCI. I guess you could say it has
an IO MMU, since the initiator (any node in the system) must set up
the bridge to direct PCI DMA to/from the right nodes. PCI interrupts
are sent to nodes as mail; they might not go to the node getting data.
(so node 18, getting the interrupt mail and controlling a SCSI device,
could cause SCSI transfers directly to/from node 101)

Linux-cluster: generic cluster infrastructure for Linux
Archive:       http://mail.nl.linux.org/linux-cluster/