Re: A proposal for a General Clustering Framework

[Peter Badovinatz <tabmowzo@yahoo.com>]

Sorry to be even later jumping in, been on vacation for a week...

--- Lars Marowsky-Bree <lmb@suse.de> wrote:
> Hi guys,
> 
> I am jumping in to a rather old discussion, but I didn't find time before.
> Customers. Eek. (Should any participant reading this be a customer, you and
> everyone else you know is of course excluded from that statement ;-)
> 
<snip>
> 
> So in fact, it is desireable that the answer is "No" to reduce complexity.
> 
> So, _can_ the answer be "No"? Yes.

From experience on the proprietary HA cluster front, the answer can't be no. 
The customers want us to be able to upgrade the software, and reintegrate the
node into the cluster.  They also want failover to work during the upgrade
process.  A set of disks connected to two nodes has no redundancy in the period
when one of the nodes is being upgraded, they want that node active as quickly
as possible.
> 
> What upgrading a node to a new software release effectively is manually
> partitioning the cluster - into the nodes which already have the new software
> and those which do not.

No, it has to be one cluster.
> 
> If a new node is updated, it will join the "new" cluster. If you figure that
> the new software release doesn't work, you downgrade it and it will go back
> to
> the old cluster.

Many serious HA  customers have 'test' clusters, which are set up the same as
their production clusters, and they test the upgrade here first.  But, yes,
they still want the ability to downgrade the node and back off an update.
> 
> As part of this, you do not have to have the two versions talk to eachother,
> because they are completely independant.

They can't be independent, as they are managing a common set of resources.
> 
> The only requirement you have to satisfy here is that two versions of the
> protocol on the same wire (logically speaking) truely do not interfere and
> that the software ignores any version of the protocols but its own.
> 
> This does have a slight penalty obviously: During the upgrade period, your
> redundancy is reduced. However, I think this is acceptable, as the upgrade is
> a controlled operation and you have experts on site to fix everything which
> might go wrong.

Fix what?  If there is no backup node integrated with the active node, where is
the recovery?  You're correct that there are experts on site, although how
expert they are varies by customer.
> 
> It may be desireable to support the following features in the resource
> management to make this more seamless for the clients:
> 
> - Be able to instruct the resource manager that a node is about to drop out,
>   but that the services which were run on this node should NOT be restarted
> on
>   another node, ie "detaching" a node and all services it is running.
> 
> - When upgrading the software on the local node, be able to tell the local
>   resource manager that even though it is going down, it should NOT take the
>   resources down.
> 
> - Being able to "reattach" to resources.

HACMP on AIX calls this support 'forced down'.  You can tell HACMP to leave the
resources running where they are, shut itself down, the other nodes do not take
the resources over, you upgrade HACMP code, restart it, and it reattaches the
resources.  The exposure here is that IF a resource fails during this upgrade
period, it will NOT be recovered, and you have to take manual action.  This
feature is popular among HACMP customers, at least some notable minority of
them.
> 
> Comments?

We have found that customers with HA clusters have some common features:
- they never want to upgrade, once it's working.
- if they must upgrade, it must be one node at a time, and it must be able to
be reversed if anything appears to be going wrong.
- upgraded nodes reintegrate into the cluster seamlessly.
- windows exposing the cluster to SPOFs must be as small as possible.
- they should never need to shut down a resource to upgrade the HA code.
> 
> -- 
> "I'm extraordinarily patient provided I get my own way in the end."
>         -- Margeret Thatcher

Peter

=====
These have been the opinions of:
Peter R. Badovinatz -- (503)578-5530 (TL 775)
wombat@us.ibm.com/tabmowzo@yahoo.com
and in no way should be construed as official opinion of 
IBM, Corp.

__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

Linux-cluster: generic cluster infrastructure for Linux
Archive:       http://mail.nl.linux.org/linux-cluster/