Re: Clustering for Linux 2.3.x?!

["Albert D. Cahalan" <acahalan@cs.uml.edu>]

Erik writes:
> Quoting Albert D. Cahalan (acahalan@cs.uml.edu):
>> Erik writes:
>>
>>> I consider this to be exactly the /wrong/ way to go about it! Why? It'll
>>> make application lazy. And why is that bad? Because redundancy is needed
>>> in apps too, and we'll end up implementing clustering both in kernel and
>>> in the apps.
>>
>> Some people need redundancy in apps. Some people don't.
>
> Could you give an example of a large clustered app where redundancy wouldn't
> be any benefit at all? Failing that (the question is slightly unfair) please
> describe the least redundancy requireing app. Note that more nodes increases
> the rate-of-faliure...

Redundancy is not important when
1. you can checkpoint
2. you need to perform a huge task, not serve continuously

>> Kernel support is also needed for process ID allocation and scheduling.
> 
> For distributed-ipc: yes. For message-passing: no. What will the cost be for
> the common case if such modifications are made??

You do need scheduling changes for message-passing. Without them, you
lose performance as other jobs are scheduled at different times on
different machines.

Your cluster app sends a message from node P to node Q. Node Q is busy
running part of a parallel compile, so it can not respond quickly.
Node P has to wait a long time or schedule something else. When node Q
responds, node P has started something else...

You could get most nodes waiting for one node. That would be bad.
Part of the Beowulf system is a gang scheduler to fix this.

-
Linux-future: thinking about the future of the Linux kernel
Archive:      http://humbolt.nl.linux.org/lists/
Wish list:    http://users.ox.ac.uk/~mert0236/linux-future.html