[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Hardware for a cluster
It depends (on budget, performance needs, etc.)
Last year, we put together a small cluster for evaluation. Five nodes (dual
600 Mhz Pentium III's w/ 256M each), dual 100 base T ethernet w/ switches,
rack, serial console connections, etc. It cost about $20,000 US. Best
measured performance was about 140 Mbit/sec on the network (2x 70 Mbit).
That system worked well for us, but it has been disassembled to support
other testing, and provide general network services.
This year we are deploying four clusters for our main application (real
time simulation of the International Space Station) with...
- 12 nodes (dual 1000 Mhz Pentium III's, 512M each, Serverworks HE
motherboards)
- Scalable Coherent Interface (SCI) - measured >20-200 Mbyte/sec in DMA
tests, 4-133 Mbyte/sec in PIO tests
- SCI connections to four existing VME racks using PMC adapters
- Dual gigabit ethernet from the head node to the LAN
- switched 100 Base T on a private LAN to all nodes
- Keyboard, video, mouse switch
- Racks, power distribution, timer cards, etc.
Total cost about $100,000 US per cluster. We are integrating our
application right now with delivery to NASA in a few months.
Dual CPU support costs about $300 - $500 per node (2nd CPU, better
motherboard). That is a lot less than a second machine & allows the two
CPU's to share memory and improve performance. When we did equipment
selection the Pentium IV was not dual CPU capable and much more expensive
than the Pentium III. We also looked at four CPU machines - way too
expensive when we looked [$20,000 and up]. Eight CPU and higher machines
are even worse. [ymmv]
SCI (or Myrinet) is a lot more expensive than Ethernet but the performance
is about 10 times higher. SCI is also optimized for "writes" (the creator
sends the data to the receiver) - that accounts for the range in
performance listed above. SCI also allows general shared memory accesses w/
care in your coding. Your choice here depends on your budget & how much
data you must pass around. You may find it cheaper to repeat calculations &
send less data to get better performance if you use Ethernet.
We were looking at prices about $1500 per node for the connections vs. a
per node price of about $2000 (even with dual CPU's). Going with single
CPU's would add about $20,000 to the system price and add latency - not
something I would recommend. We have had good support from both our cluster
vendor (PSSC Labs) and Dolphin (SCI vendor).
I would try to stay away from adding too much hardware to any node. On a
separate system we had...
- 4 port ethernet
- Matrox G200/MMS (we use 3 of 4 displays)
- 3 Win TV tuners for video input
and could not get that equipment to run reliably. Our major problem is IRQ
sharing and the BIOS spreading each card (network, display) on all four
IRQ's dedicated for the PCI. Bad behaving drivers then caused the problems
- generally a system lock up. We split that hardware to two machines & it
works OK.
--Mark H Johnson
<mailto:Mark_H_Johnson@raytheon.com>
"J . A . Magallon"
<jamagallon@able.es> To: Lista Linux-Cluster <linux-cluster@nl.linux.org>
Sent by: cc:
owner-linux-cluster@nl Subject: Hardware for a cluster
.linux.org
09/18/01 09:14 AM
Hi everyone.
I ask for the expert's opinion.
We are going to build a cluster. Some questions:
- Dual mobos or single ones ? Some people here say that with dual systems
you fill half the space. Is it really an advantage to use dual processor
boards ?
- Is PentiumIV so much better that a PIII ? Will nowadays gcc (3.0.1)
generate
code for PIV ?
- we are planning to spent not too much money on the network just now, we
will
see if Ether-100Mb is enough for our communication requierementes (and wait
for
next year grants...). But I think
that an Starfire with 4 ports will be good for internal network, with
channel
bonding. Or perhaps we can bind NFS to eth0 and rest of comms to eth1, etc.
Any known issues ?
Thanks in advance for your replies.
--
J.A. Magallon # Let the source be with you...
mailto:jamagallon@able.es
Mandrake Linux release 8.1 (Cooker) for i586
Linux werewolf 2.4.9-ac11 #1 SMP Mon Sep 17 00:00:56 CEST 2001 i686
Linux-cluster: generic cluster infrastructure for Linux
Archive: http://mail.nl.linux.org/linux-cluster/
Linux-cluster: generic cluster infrastructure for Linux
Archive: http://mail.nl.linux.org/linux-cluster/