From owner-linux-cluster@nl.linux.org Mon Jun 4 17:28:06 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 4 Jun 2001 17:27:47 +0200 Received: from perth.fpcc.net ([207.174.142.141]:26928 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Mon, 4 Jun 2001 17:27:45 +0200 Received: from unix.sh (03-152.026.popsite.net [64.24.105.152]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id JAA07842; Mon, 4 Jun 2001 09:27:16 -0600 Message-ID: <3B1BA89D.313D31DD@unix.sh> Date: Mon, 04 Jun 2001 09:26:21 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster Subject: A proposal for a General Clustering Framework Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Hi, I've talked off-list to many of you about the prospects for creating a general clustering framework - usable for many or all HA (and HPC) clusters. I'm currently in the process of writing a paper on these framework ideas. If any of you are interested in providing me feedback on a VERY EARLY version of the paper, you can find a copy of it in StarOffice format here: http://www.linux-ha.org/framework/HAFramework.sdw I'll send in an extended abstract derived from this version to submit to ALS tomorrow. The final paper is due a few months from now. Please send your comments to me directly. Otherwise, there'll be lots of cross-list postings, etc. Of course, if it's possible to send me comments and thoughts before midday tomorrow, that would be wonderful! Otherwise, they'll be incorporated after the initial extended abstract is submitted. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 4 22:03:13 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 4 Jun 2001 22:03:03 +0200 Received: from adglinux1.hns.com ([139.85.108.152]:42371 "EHLO adglinux1.hns.com") by humbolt.nl.linux.org with ESMTP id ; Mon, 4 Jun 2001 22:02:50 +0200 Received: from nbecker by adglinux1.hns.com with local (Exim 3.22 #1) id 1570Xr-0003zD-00; Mon, 04 Jun 2001 16:01:47 -0400 To: Alan Robertson Cc: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1BA89D.313D31DD@unix.sh> From: nbecker@fred.net Date: 04 Jun 2001 16:01:47 -0400 In-Reply-To: <3B1BA89D.313D31DD@unix.sh> Message-ID: Lines: 1 User-Agent: Gnus/5.0808 (Gnus v5.8.8) XEmacs/21.1 (Cuyahoga Valley) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Can you write a PDF? Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 4 23:20:45 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 4 Jun 2001 23:20:37 +0200 Received: from inet-mail3.oracle.com ([148.87.2.203]:16851 "EHLO inet-mail3.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Mon, 4 Jun 2001 23:20:25 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail3.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f54LGsv13082; Mon, 4 Jun 2001 14:16:54 -0700 (PDT) Received: from oracle.com (dbrower-sun.us.oracle.com [130.35.180.64]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f54LKFX25814; Mon, 4 Jun 2001 14:20:15 -0700 (PDT) Message-ID: <3B1BFB8F.110F6ED6@oracle.com> Date: Mon, 04 Jun 2001 14:20:15 -0700 From: David Brower Organization: Oracle Corporation X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1BA89D.313D31DD@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list IMO, IBM and Compaq both Got It Right when they put an generalized pub/sub event system in the middle of their cluster frameworks. I've been pointing this out for a while, but it doesn't seem to have gotten any traction-- neither Tweedie's old proposal or Robertson's here have one AFAICT. FWIW, xml-rpc is a perfectly fine mechanism for such an event system, once you get the subscription mechanism in place. You probably also want more authentication and authorization than is used on things that assume private networks for cluster traffic. -dB -- Butterflies tell me to say: "The statements and opinions expressed here are my own and do not necessarily represent those of Oracle Corporation." Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 4 23:41:17 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 4 Jun 2001 23:41:08 +0200 Received: from gw.xkey.com ([206.86.100.52]:23049 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Mon, 4 Jun 2001 23:41:01 +0200 Received: (from smtp@localhost) by happy.xkey.com id OAA13369 for ; Mon, 4 Jun 2001 14:40:53 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma013356; Mon Jun 4 14:40:46 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f54Lh7T02316 for linux-cluster@nl.linux.org; Mon, 4 Jun 2001 17:43:07 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Mon, 4 Jun 2001 17:43:07 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010604174307.A2207@wumpus> Mail-Followup-To: linux-cluster References: <3B1BA89D.313D31DD@unix.sh> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1BA89D.313D31DD@unix.sh>; from alanr@unix.sh on Mon, Jun 04, 2001 at 09:26:21AM -0600 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Mon, Jun 04, 2001 at 09:26:21AM -0600, Alan Robertson wrote: > I'm currently in the process of writing a paper on these framework ideas. > If any of you are interested in providing me feedback on a VERY EARLY > version of the paper, you can find a copy of it in StarOffice format here: > > http://www.linux-ha.org/framework/HAFramework.sdw And for those who don't have Star Office: http://www.pbm.com/~lindahl/HAFramework.pdf -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 4 23:50:19 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 4 Jun 2001 23:50:11 +0200 Received: from smtp7.us.dell.com ([143.166.224.233]:24078 "EHLO smtp7.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Mon, 4 Jun 2001 23:49:55 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp7.us.dell.com (8.11.2/8.11.0) with ESMTP id f54LnnD11149; Mon, 4 Jun 2001 16:49:49 -0500 Received: from xeon.michaels-house.net ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id QAA29118; Mon, 4 Jun 2001 16:49:39 -0500 (CDT) Date: Mon, 4 Jun 2001 16:49:38 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: David Brower cc: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1BFB8F.110F6ED6@oracle.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list xml-rpc and/or soap is a great idea. I've just been using it in one of my projects to connect some C<-->Perl<-->PHP (and possibly VB on windows in the future) and it is awesome. -- Michael Brown On Mon, 4 Jun 2001, David Brower wrote: > IMO, IBM and Compaq both Got It Right when they > put an generalized pub/sub event system in the middle > of their cluster frameworks. I've been pointing this > out for a while, but it doesn't seem to have gotten any > traction-- neither Tweedie's old proposal or Robertson's > here have one AFAICT. > > FWIW, xml-rpc is a perfectly fine mechanism for such > an event system, once you get the subscription mechanism > in place. You probably also want more authentication and > authorization than is used on things that assume private > networks for cluster traffic. > > -dB > > -- > Butterflies tell me to say: > "The statements and opinions expressed here are my own and do not necessarily > represent those of Oracle Corporation." > > ------------------------------------------------------------------------------ > Linux HA Web Site: > http://linux-ha.org/ > Linux HA HOWTO: > http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html > ------------------------------------------------------------------------------ > > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 01:26:52 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 01:26:43 +0200 Received: from perth.fpcc.net ([207.174.142.141]:17724 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 01:26:27 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id RAA11345; Mon, 4 Jun 2001 17:25:57 -0600 Message-ID: <3B1C18D0.CA5DCE6E@unix.sh> Date: Mon, 04 Jun 2001 17:25:04 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster CC: riel@conectiva.com.br, Anas Nashif Subject: Re: A proposal for a General Clustering Framework References: <3B1BA89D.313D31DD@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Hi, > > I've talked off-list to many of you about the prospects for creating a > general clustering framework - usable for many or all HA (and HPC) clusters. > > I'm currently in the process of writing a paper on these framework ideas. > If any of you are interested in providing me feedback on a VERY EARLY > version of the paper, you can find a copy of it in StarOffice format here: > > http://www.linux-ha.org/framework/HAFramework.sdw OK. At the request of a couple of folks, I've provided PS and PDF and HTML versions of the document in this same directory, http://www.linux-ha.org/framework/ Let's continue this discussion on linux-cluster, and *not* on the other lists, to avoid giving everyone a multi-cross-post-headache. I'll be continuing my discussion and replies to mails about this proposal on that list - ONLY. Please subscribe if you haven't already done so. Here's what I did to subscribe. Majordomo-heads can shorten this or correct it no doubt, (I'm a mailman-user myself ;-)). Subject: subscribe linux-cluster To: majordomo@nl.linux.org subscribe linux-cluster If you can't get subscribed to this list somehow, then send mail directly to me. Thanks! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 02:03:18 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 02:03:00 +0200 Received: from 239-MADR-X23.libre.retevision.es ([62.83.14.239]:27630 "EHLO carlos.mosix.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 02:02:55 +0200 Received: from carlos by carlos.mosix.net with local (Exim 3.12 #1 (Debian)) id 1574FT-0000S1-00; Tue, 05 Jun 2001 01:59:03 +0200 Subject: Re: A proposal for a General Clustering Framework From: Carlos Manzanedo To: Michael E Brown Cc: clustering In-Reply-To: Content-Type: text/plain X-Mailer: Evolution 0.5.1 (Developer Preview) Date: 04 Jun 2001 22:59:03 -0100 Mime-Version: 1.0 Message-Id: Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list do you think HP developers will agree with rpc ? ok , they should agree sharing components and may be xml , but i think that the net load of an HP cluster uses to be hight enough to overload it with rpc ... > xml-rpc and/or soap is a great idea. I've just been using it in one of my > projects to connect some C<-->Perl<-->PHP (and possibly VB on windows in > the future) and it is awesome. > -- > Michael Brown > > On Mon, 4 Jun 2001, David Brower wrote: > > > IMO, IBM and Compaq both Got It Right when they > > put an generalized pub/sub event system in the middle > > of their cluster frameworks. I've been pointing this > > out for a while, but it doesn't seem to have gotten any > > traction-- neither Tweedie's old proposal or Robertson's > > here have one AFAICT. > > > > FWIW, xml-rpc is a perfectly fine mechanism for such > > an event system, once you get the subscription mechanism > > in place. You probably also want more authentication and > > authorization than is used on things that assume private > > networks for cluster traffic. > > > > -dB > > > > -- > > Butterflies tell me to say: > > "The statements and opinions expressed here are my own and do not necessarily > > represent those of Oracle Corporation." > > > > ------------------------------------------------------------------------------ > > Linux HA Web Site: > > http://linux-ha.org/ > > Linux HA HOWTO: > > http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html > > ------------------------------------------------------------------------------ > > > > > > > Linux-cluster: generic cluster infrastructure for Linux > Archive: http://mail.nl.linux.org/linux-cluster/ > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 02:21:41 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 02:21:26 +0200 Received: from smtp6.us.dell.com ([143.166.83.101]:25610 "EHLO smtp6.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 02:21:20 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp6.us.dell.com (8.11.0/8.11.0) with ESMTP id f550LHG28297; Mon, 4 Jun 2001 19:21:17 -0500 Received: from xeon.michaels-house.net ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id TAA17899; Mon, 4 Jun 2001 19:21:15 -0500 (CDT) Date: Mon, 4 Jun 2001 19:21:15 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Alan Robertson cc: linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1C18D0.CA5DCE6E@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Thanks for the PDFs! I hadn't gotten around to installing StarOffice on my Debain box at home yet. After having looked at the paper, I see the purpose of the previouse xmlrpc/soap discussion. :-) So, now that I know you are actually going to use XML as your interprocess interchange format, I feel strongly that using the smallest subset of XML that is compatible with XMLRPC would greatly benefit the overall project. XMLRPC is a pretty simple representation, and it doesn't have all the extra bells and whistles that SOAP has. But what it does have is language bindings for just about every language out there. http://xmlrpc-c.sourceforge.net/xmlrpc-howto/xmlrpc-howto.html I see in the paper where you talk about the need for a way to configure each module, and having standards for configuration. In XMLRPC there are the beginnings of a standard for "introspection". In other words, a way for an XMLRPC client to figure out which methods an XMLRPC server supports. See http://xmlrpc.usefulinc.com/doc/reserved.html So far from what I have seen, everybody that uses XML uses expat. http://www.expat.org It's not really that big: xeon:/opt/ali-client/lib# ls -l *expat* -rw-r--r-- 1 root root 318738 Jun 3 01:04 libexpat.a -rwxr-xr-x 1 root root 653 Jun 3 01:04 libexpat.la lrwxrwxrwx 1 root root 17 Jun 3 01:04 libexpat.so -> libexpat.so.0.0.1 lrwxrwxrwx 1 root root 17 Jun 3 01:04 libexpat.so.0 -> libexpat.so.0.0.1 -rwxr-xr-x 1 root root 270786 Jun 3 01:04 libexpat.so.0.0.1 But given the lightweight-edness of the XMLRPC spec (see below), it would be relatively straightforward to write a lightweight/fast XML parser specifically for doing XMLRPC. And I would rather see that happen than see Yet Another XML Standard :-) As for the XMLRPC spec, it is pretty simple. Here is an example procedure call: POST /RPC2 HTTP/1.0 User-Agent: Frontier/5.1.2 (WinNT) Host: betty.userland.com Content-Type: text/xml Content-length: 181 examples.getStateName 41 And here is the spec for xmlrpc: http://www.xmlrpc.com/spec -- Michael Brown On Mon, 4 Jun 2001, Alan Robertson wrote: > Alan Robertson wrote: > > > > Hi, > > > > I've talked off-list to many of you about the prospects for creating a > > general clustering framework - usable for many or all HA (and HPC) clusters. > > > > I'm currently in the process of writing a paper on these framework ideas. > > If any of you are interested in providing me feedback on a VERY EARLY > > version of the paper, you can find a copy of it in StarOffice format here: > > > > http://www.linux-ha.org/framework/HAFramework.sdw > > OK. At the request of a couple of folks, I've provided PS and PDF and HTML > versions of the document in this same directory, > http://www.linux-ha.org/framework/ > > Let's continue this discussion on linux-cluster, and *not* on the other > lists, to avoid giving everyone a multi-cross-post-headache. > > I'll be continuing my discussion and replies to mails about this proposal on > that list - ONLY. Please subscribe if you haven't already done so. Here's > what I did to subscribe. Majordomo-heads can shorten this or correct it no > doubt, (I'm a mailman-user myself ;-)). > > Subject: subscribe linux-cluster > To: majordomo@nl.linux.org > > subscribe linux-cluster > > If you can't get subscribed to this list somehow, then send mail directly to > me. > > > Thanks! > > -- Alan Robertson > alanr@unix.sh > > ------------------------------------------------------------------------------ > Linux HA Web Site: > http://linux-ha.org/ > Linux HA HOWTO: > http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html > ------------------------------------------------------------------------------ > > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 02:50:55 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 02:50:46 +0200 Received: from cerebus.wirex.com ([216.161.55.93]:23278 "EHLO figure1.int.wirex.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 02:50:40 +0200 Received: (from chris@localhost) by figure1.int.wirex.com (8.11.0/8.11.0) id f550lsI05377 for linux-cluster@nl.linux.org; Mon, 4 Jun 2001 17:47:54 -0700 Date: Mon, 4 Jun 2001 17:47:54 -0700 From: Chris Wright To: linux-cluster Subject: Re: [Linux-ha-dev] Re: A proposal for a General Clustering Framework Message-ID: <20010604174754.B11187@figure1.int.wirex.com> References: <3B1BA89D.313D31DD@unix.sh> <3B1BFB8F.110F6ED6@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1BFB8F.110F6ED6@oracle.com>; from David.Brower@oracle.com on Mon, Jun 04, 2001 at 02:20:15PM -0700 X-Editor: Vim http://www.vim.org/ X-Info: http://www.wirex.com Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list * David Brower (David.Brower@oracle.com) wrote: > IMO, IBM and Compaq both Got It Right when they > put an generalized pub/sub event system in the middle > of their cluster frameworks. I've been pointing this > out for a while, but it doesn't seem to have gotten any > traction-- neither Tweedie's old proposal or Robertson's > here have one AFAICT. I fully agree. I've made similar suggestions (for linux-ha specifically) and had similar difficulty gaining traction. From a framework perspective it seems very useful to consider cluster or service state transitions as events. This ought to allow many interpretations of the meaning of the event (as I assume is the intention of developing a generic framework). -chris Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 03:00:16 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 03:00:07 +0200 Received: from mercury.mv.net ([199.125.85.40]:54534 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 02:59:54 +0200 Received: from filesrus (bnh-3-19.mv.com [199.125.99.147]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id UAA13143; Mon, 4 Jun 2001 20:59:42 -0400 (EDT) Message-ID: <017801c0ed5b$2852dd60$93637dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: Subject: Re: A proposal for a General Clustering Framework Date: Mon, 4 Jun 2001 21:02:12 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list As the newest of newbies (having joined my first Linux list today), I'm a bit hesitant to voice too strong an opinion yet. But I will suggest that at least some LAN-resident cluster implementations might prefer a base communication mechanism allowing lighter-weight (one-way) datagrams rather than requiring RPC-style responses for *all* messages: message loss in (at least switched) LANs can often be made sufficiently rare that the mechanisms one needs anyway to allow for node failure can handle message loss as well, making responses superfluous in the many cases that don't fit the request/response paradigm (there are quite a few, for example, in the implementation of distributed file systems and caches). - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 03:43:43 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 03:43:35 +0200 Received: from c004-h009.c004.sfo.cp.net ([209.228.14.66]:6561 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Tue, 5 Jun 2001 03:43:25 +0200 Received: (cpmta 23108 invoked from network); 4 Jun 2001 18:43:10 -0700 Received: from adsl-151-203-49-173.bostma.adsl.bellatlantic.net (HELO jdarcy6986nk) (151.203.49.173) by smtp.namezero.com (209.228.14.66) with SMTP; 4 Jun 2001 18:43:10 -0700 X-Sent: 5 Jun 2001 01:43:10 GMT Message-ID: <00f001c0ed60$4aaf2710$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "ha-linux List" , "Linux-HA Development List" , "LinuxFailSafe Mailing List" , "linux-cluster" References: <3B1BA89D.313D31DD@unix.sh> <3B1BFB8F.110F6ED6@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Mon, 4 Jun 2001 21:38:56 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list > IMO, IBM and Compaq both Got It Right when they > put an generalized pub/sub event system in the middle > of their cluster frameworks. A generalized event system is a great idea, but I'm not sure that "pub/sub" belongs in the middle of the cluster framework. One of the most critical functionalities for a cluster infrastructure is agreement on which of several events (which may be merely symptoms of a single actual failure) to process first. That process is difficult enough when everyone agrees on the list of candidates. At this level, there should be no subscription requirement; all nodes should get all events, period. That sort of filtering belongs at what I'd call the top of the framework, not the middle. > FWIW, xml-rpc is a perfectly fine mechanism for such > an event system, once you get the subscription mechanism > in place. You probably also want more authentication and > authorization than is used on things that assume private > networks for cluster traffic. I'm afraid you (all) will have to forgive me some of my biases. I spent a lot of time developing protocols for this kind of thing, in code that was later credited to IBM. One conclusion I reached during that work is that *RPC is evil* as soon as you depart from the two-party request-response model for which it was designed. Yes, yes, I know you can do broadcast RPC and stuff, but that's just a collection of two-way conversations and fundamentally different than the sort of true multi-way conversation you need for cluster stuff. XML is fine as a representation, and XML-RPC might be fine as part of a top-level notification API, but deep down in the guts of things I think all flavors of RPC should be avoided. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 03:47:55 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 03:47:47 +0200 Received: from perth.fpcc.net ([207.174.142.141]:33342 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 03:47:36 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA12092; Mon, 4 Jun 2001 19:47:18 -0600 Message-ID: <3B1C39F3.7C48A710@unix.sh> Date: Mon, 04 Jun 2001 19:46:27 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <017801c0ed5b$2852dd60$93637dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > As the newest of newbies (having joined my first Linux list today), I'm a > bit hesitant to voice too strong an opinion yet. But I will suggest that at > least some LAN-resident cluster implementations might prefer a base > communication mechanism allowing lighter-weight (one-way) datagrams rather > than requiring RPC-style responses for *all* messages: message loss in (at > least switched) LANs can often be made sufficiently rare that the mechanisms > one needs anyway to allow for node failure can handle message loss as well, > making responses superfluous in the many cases that don't fit the > request/response paradigm (there are quite a few, for example, in the > implementation of distributed file systems and caches). Message loss is no problem to this layer of the system. The underlying infrastructure must retransmit, etc, so that's not an issue. As an example, you can see how heartbeat does this by looking here: http://linux-ha.org/comm/ I really think that RPC is more often not what you want. I didn't mention RPC, but others did. In a cluster, there is often event notification (which is not bidirectional), and often other kinds of multicast operations. RPC is neither necessary nor sufficient for cluster communications. People use barriers, semaphores, transactions, etc as suitable models. I don't see RPC as the right paradigm. On the other hand, a cluster-aware can use RPC to communicate with a *local* daemon to get barrier, membership, transaction services, etc from a local process... But across the cluster, I don't think it's that suitable. The typical paradigm for a cluster communication doesn't look at all like normal client-server interactions... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 03:54:17 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 03:54:08 +0200 Received: from perth.fpcc.net ([207.174.142.141]:20031 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 03:53:57 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA12143; Mon, 4 Jun 2001 19:53:52 -0600 Message-ID: <3B1C3B7D.457349CA@unix.sh> Date: Mon, 04 Jun 2001 19:53:01 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Carlos Manzanedo CC: Michael E Brown , clustering Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Carlos Manzanedo wrote: > > do you think HP developers will agree with rpc ? > ok , they should agree sharing components and may be xml , but i think > that the net load of an HP cluster uses to be hight enough to overload > it with rpc ... > > > xml-rpc and/or soap is a great idea. I've just been using it in one of my > > projects to connect some C<-->Perl<-->PHP (and possibly VB on windows in > > the future) and it is awesome. I don't think RPC is what we need. What I need first and foremost is cluster messaging. The only case I can think of is *local* client<->server intereactions. I really believe in XML, but have not thought that RPC is the right thing to do for the cluster infrastructure. But, I'm willing to listen... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 03:54:27 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 03:54:10 +0200 Received: from perth.fpcc.net ([207.174.142.141]:53822 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 03:53:48 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA12129; Mon, 4 Jun 2001 19:52:42 -0600 Message-ID: <3B1C3B37.813C63D6@unix.sh> Date: Mon, 04 Jun 2001 19:51:51 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Darcy CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1BA89D.313D31DD@unix.sh> <3B1BFB8F.110F6ED6@oracle.com> <00f001c0ed60$4aaf2710$367b9fa8@lss.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Jeff Darcy wrote: > > > IMO, IBM and Compaq both Got It Right when they > > put an generalized pub/sub event system in the middle > > of their cluster frameworks. > > A generalized event system is a great idea, but I'm not sure that "pub/sub" > belongs in the middle of the cluster framework. One of the most critical > functionalities for a cluster infrastructure is agreement on which of > several events (which may be merely symptoms of a single actual failure) to > process first. That process is difficult enough when everyone agrees on the > list of candidates. At this level, there should be no subscription > requirement; all nodes should get all events, period. That sort of > filtering belongs at what I'd call the top of the framework, not the middle. > > > FWIW, xml-rpc is a perfectly fine mechanism for such > > an event system, once you get the subscription mechanism > > in place. You probably also want more authentication and > > authorization than is used on things that assume private > > networks for cluster traffic. > > I'm afraid you (all) will have to forgive me some of my biases. I spent a > lot of time developing protocols for this kind of thing, in code that was > later credited to IBM. One conclusion I reached during that work is that > *RPC is evil* as soon as you depart from the two-party request-response > model for which it was designed. Yes, yes, I know you can do broadcast RPC > and stuff, but that's just a collection of two-way conversations and > fundamentally different than the sort of true multi-way conversation you > need for cluster stuff. XML is fine as a representation, and XML-RPC might > be fine as part of a top-level notification API, but deep down in the guts > of things I think all flavors of RPC should be avoided. I agree with this. I didn't mention RPC and have thought it was the wrong tool for this job. It's the wrong paradigm, I think... But, for talking to local daemons, etc., some kind of RPC is necessary. I implemented a kind for heartbeat that could be generalized here. XML-RPC is too weak for the job (it only allows the simplest kinds of data to be passed around). I only sent this reply to linux-cluster as per my earlier email about flooding all these lists... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 04:03:38 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 04:03:29 +0200 Received: from perth.fpcc.net ([207.174.142.141]:14400 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 04:03:11 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id UAA12186; Mon, 4 Jun 2001 20:03:08 -0600 Message-ID: <3B1C3DA9.CD48229A@unix.sh> Date: Mon, 04 Jun 2001 20:02:17 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > Thanks for the PDFs! I hadn't gotten around to installing StarOffice on my > Debain box at home yet. > > After having looked at the paper, I see the purpose of the previouse > xmlrpc/soap discussion. :-) > So far from what I have seen, everybody that uses XML uses expat. > http://www.expat.org > expat.org is the URL for a french e-magazine... This looks like a better URL: http://sourceforge.net/projects/expat -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 04:05:01 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 04:04:51 +0200 Received: from perth.fpcc.net ([207.174.142.141]:15168 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 04:04:37 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id UAA12190; Mon, 4 Jun 2001 20:04:32 -0600 Message-ID: <3B1C3DFC.555CEDF7@unix.sh> Date: Mon, 04 Jun 2001 20:03:40 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > Thanks for the PDFs! I hadn't gotten around to installing StarOffice on my > Debain box at home yet. > > After having looked at the paper, I see the purpose of the previouse > xmlrpc/soap discussion. :-) > > So, now that I know you are actually going to use XML as your interprocess > interchange format, I feel strongly that using the smallest subset of XML > that is compatible with XMLRPC would greatly benefit the overall project. Unfrotuantely, XMLRPC is not nearly powerful enough. I need to send basically arbitrarily complex data structures back and forth. It doesn't have name/value pairs, or attributes, etc. So, XML-RPC isn't very attractive thing to build on for clusters... I'll see if I can come up with an example that I *know* I'll have to deal with in configuring Stonith modules... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 04:07:31 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 04:07:14 +0200 Received: from perth.fpcc.net ([207.174.142.141]:16960 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 04:07:02 +0200 Received: from unix.sh (09-060.026.popsite.net [64.24.108.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id UAA12221; Mon, 4 Jun 2001 20:06:59 -0600 Message-ID: <3B1C3E90.A3EE9A80@unix.sh> Date: Mon, 04 Jun 2001 20:06:08 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1C3DA9.CD48229A@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Michael E Brown wrote: > > > > Thanks for the PDFs! I hadn't gotten around to installing StarOffice on my > > Debain box at home yet. > > > > After having looked at the paper, I see the purpose of the previouse > > xmlrpc/soap discussion. :-) > > > > So far from what I have seen, everybody that uses XML uses expat. > > http://www.expat.org > > > > expat.org is the URL for a french e-magazine... > > This looks like a better URL: > > http://sourceforge.net/projects/expat Or here... http://www.jclark.com/xml/expat.html -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 04:07:33 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 04:07:13 +0200 Received: from web9207.mail.yahoo.com ([216.136.129.40]:62989 "HELO web9207.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Tue, 5 Jun 2001 04:07:07 +0200 Message-ID: <20010605020656.20219.qmail@web9207.mail.yahoo.com> Received: from [202.135.13.219] by web9207.mail.yahoo.com; Mon, 04 Jun 2001 19:06:56 PDT Date: Mon, 4 Jun 2001 19:06:56 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: Linux Cluster In-Reply-To: <3B1BFB8F.110F6ED6@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- David Brower wrote: > IMO, IBM and Compaq both Got It Right when they > put an generalized pub/sub event system in the middle > of their cluster frameworks. I've been pointing this > out for a while, but it doesn't seem to have gotten any > traction-- neither Tweedie's old proposal or Robertson's > here have one AFAICT. IBM's clustering has multiple possible subscription points. One is tightly synchronized, fully ordered, set semantics, etc. This is used to drive failover and such. This is either contained within the cluster manager and not a general API (HACMP) or exposed via the Group Services API. The other seems more what you're referring to, the Event Management API. This is not really in the middle, more on the top, as it doesn't provide the ordering guarantees and such. But, it does provide a very general mechanism for publishing and subscribing to almost any kind of event (with some work on the producer side, depending on what you want to monitor.) This is intended as a very extensible monitoring engine, not per se a recovery driver. > > FWIW, xml-rpc is a perfectly fine mechanism for such > an event system, once you get the subscription mechanism > in place. You probably also want more authentication and > authorization than is used on things that assume private > networks for cluster traffic. I, personally, am very uncomfortable with RPC except for local (intra-node) communications. I see it could be used as part of the event management type notification, but, as I'm familiar with it, the events are really just "here they are" messages, without the kind of execution implied for rpc. > > -dB > > -- > Butterflies tell me to say: > "The statements and opinions expressed here are my own and do not necessarily > > represent those of Oracle Corporation." Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 15:42:29 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 15:42:11 +0200 Received: from psasolar.colltech.com ([208.229.236.14]:40681 "EHLO psasolar.colltech.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 15:41:58 +0200 Received: from localhost by psasolar.colltech.com (8.9.3/8.9.3/not) with ESMTP id IAA14228; Tue, 5 Jun 2001 08:41:26 -0500 (CDT) Date: Tue, 5 Jun 2001 08:41:25 -0500 (CDT) From: James Mello X-Sender: kingjamm@psasolar.private.psa.pencom.com To: David Brower cc: ha-linux List , Linux-HA Development List , LinuxFailSafe Mailing List , linux-cluster Subject: Re: [LinuxFailSafe] Re: A proposal for a General Clustering Framework In-Reply-To: <3B1BFB8F.110F6ED6@oracle.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Just out of curiosity, what kind of clustering are you talking about? Are you talking specifically about HA, or the LB kind? -- Cheers -- James On Mon, 4 Jun 2001, David Brower wrote: > IMO, IBM and Compaq both Got It Right when they > put an generalized pub/sub event system in the middle > of their cluster frameworks. I've been pointing this > out for a while, but it doesn't seem to have gotten any > traction-- neither Tweedie's old proposal or Robertson's > here have one AFAICT. > > FWIW, xml-rpc is a perfectly fine mechanism for such > an event system, once you get the subscription mechanism > in place. You probably also want more authentication and > authorization than is used on things that assume private > networks for cluster traffic. > > -dB > > -- > Butterflies tell me to say: > "The statements and opinions expressed here are my own and do not necessarily > represent those of Oracle Corporation." > _______________________________________________ > LinuxFailSafe mailing list > LinuxFailSafe@lists.community.tummy.com > http://lists.community.tummy.com/mailman/listinfo/linuxfailsafe > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 16:10:08 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 16:09:47 +0200 Received: from web9208.mail.yahoo.com ([216.136.129.41]:4360 "HELO web9208.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Tue, 5 Jun 2001 16:09:35 +0200 Message-ID: <20010605140933.91932.qmail@web9208.mail.yahoo.com> Received: from [202.135.239.41] by web9208.mail.yahoo.com; Tue, 05 Jun 2001 07:09:33 PDT Date: Tue, 5 Jun 2001 07:09:33 -0700 (PDT) From: Peter Badovinatz Subject: Re: [LinuxFailSafe] Re: A proposal for a General Clustering Framework To: Linux Cluster In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- James Mello wrote: > Just out of curiosity, what kind of clustering are you talking about? Are > you talking specifically about HA, or the LB kind? > > -- Cheers > -- James This thread is oriented to HA, although the IBM work that is referenced by the text below was used in HA and in high performance/LB clusters (Beowulf-like, IBM SP MPP). > > On Mon, 4 Jun 2001, David Brower wrote: > > > IMO, IBM and Compaq both Got It Right when they > > put an generalized pub/sub event system in the middle > > of their cluster frameworks. I've been pointing this > > out for a while, but it doesn't seem to have gotten any > > traction-- neither Tweedie's old proposal or Robertson's > > here have one AFAICT. > > > > -dB > > > > -- > > Butterflies tell me to say: > > "The statements and opinions expressed here are my own and do not > necessarily > > represent those of Oracle Corporation." > Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 16:56:46 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 16:56:38 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:63967 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 16:56:33 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f55EpID29536; Tue, 5 Jun 2001 07:51:19 -0700 (PDT) Received: from oracle.com ([152.68.53.78]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f55EuJX07446; Tue, 5 Jun 2001 07:56:20 -0700 (PDT) Message-ID: <3B1CF30E.CC40CBFE@oracle.com> Date: Tue, 05 Jun 2001 07:56:15 -0700 From: David Brower Organization: Oracle X-Mailer: Mozilla 4.7 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Alan Robertson CC: Michael E Brown , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1C3DFC.555CEDF7@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > Unfrotuantely, XMLRPC is not nearly powerful enough. I need to send > basically arbitrarily complex data structures back and forth. It doesn't > have name/value pairs, or attributes, etc. So, XML-RPC isn't very > attractive thing to build on for clusters... I don't understand this conclusion at all. You can have pretty much anything you want as a parameter, and XML is nothing but a big list of name value pairs. The semantics of the parms (nvlist, attribute, etc. is up to the callers to agree upon; XML-RPC doesn't have an IDL that enforces any semantics. And I don't know what kind of complex data structure can't be represented. As with corba, you need to understand the abstraction to do linked lists or trees. Since you can't send pointers (which are language dependant in all cases), you need to abstract the link references and resconstruct them in the appropriate way on the receiving end. Someone else observed that xml-rpc was "an expensive round trip" for cases where one wished to use broadcast or multicast. I think the approach to take is that used by one interpretation of the corba "oneway": have the request contain a "no response needed" flag, so that the receiver doesn't send a message back. -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 17:17:59 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 17:17:43 +0200 Received: from smtp6.us.dell.com ([143.166.83.101]:47112 "EHLO smtp6.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 17:17:29 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp6.us.dell.com (8.11.0/8.11.0) with ESMTP id f55FHRG28300; Tue, 5 Jun 2001 10:17:27 -0500 Received: from F2400-1.linuxdev.us.dell.com ([10.180.92.240]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id KAA17485; Tue, 5 Jun 2001 10:17:22 -0500 (CDT) Date: Tue, 5 Jun 2001 10:17:22 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Alan Robertson cc: linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1C3DFC.555CEDF7@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list I'm sorry. I don't think I communicated an important point that I had on my mind when I was composing the original msg. I was not advocating that you use XMLRPC for certain things such as cluster heartbeat, which have to be slim/trim/no memory leaks/etc. But I do think it would be easier from a library standpoint if the messages that you were multicasting were in an XMLRPC-compatible encoding. I think David Brower's point that XMLRPC is very powerful and can encode just about any C struct or datatype is valid. What I was thinking of is another layer on top of the shared hearbeat/cluster manager/etc framework. An XMLRPC transport so that _third party_ applications have a well-defined interface into the core cluster managment interfact. For example, the API for an application to use that is sitting on top of the cluster, or for things like non-core cluster application monitoring scripts and programs. XMLRPC IS appropriate for (with a completely made-up example API :-) 1) Application cluster membership query ->ListClusterMembers() 2) Application status query ->QueryNodeStatus() 3) application monitoring/start/stop scripts ->BeginResourceStartup() ->ResourceRunning() ->ResourceFailure() 4) Configuration: ->ListStonithMethods() ->StonithConfig() ->NetworkConfig() XMLRPC IS NOT appropriate for 1) low-level heartbeat: SendHeartbeat() 2) cluster manager stuff/membership services: 3) actually performing stonith: StomithKillNode() So, what I would imagine is a core API that uses shared-libraries and XML as it's communication method for the "Highly-Available" part of the cluster management. Then there would be an XMLRPC wrapper around the "Administrative" part of the cluster managment. Right now, shell scripts certainly play a large role in cluster managment. I would envision moving towards a Python/Perl/C/whatever based structure, and the good thing about XMLRPC is that it would allow you to freely choose your implementation language for these functions, but your API would always stay the same. And with the added benefit that you would not have to write wrappers for any of these languages. Hopefully I have been a bit more clear this time :-) -- Michael On Mon, 4 Jun 2001, Alan Robertson wrote: > Michael E Brown wrote: > > > > Thanks for the PDFs! I hadn't gotten around to installing StarOffice on my > > Debain box at home yet. > > > > After having looked at the paper, I see the purpose of the previouse > > xmlrpc/soap discussion. :-) > > > > So, now that I know you are actually going to use XML as your interprocess > > interchange format, I feel strongly that using the smallest subset of XML > > that is compatible with XMLRPC would greatly benefit the overall project. > > Unfrotuantely, XMLRPC is not nearly powerful enough. I need to send > basically arbitrarily complex data structures back and forth. It doesn't > have name/value pairs, or attributes, etc. So, XML-RPC isn't very > attractive thing to build on for clusters... > > I'll see if I can come up with an example that I *know* I'll have to deal > with in configuring Stonith modules... > > -- Alan Robertson > alanr@unix.sh > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 17:32:10 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 17:32:03 +0200 Received: from perth.fpcc.net ([207.174.142.141]:11340 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 17:31:48 +0200 Received: from unix.sh (05-058.026.popsite.net [64.24.106.58]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id JAA16077; Tue, 5 Jun 2001 09:31:15 -0600 Message-ID: <3B1CFB0F.7096BE91@unix.sh> Date: Tue, 05 Jun 2001 09:30:23 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Brower CC: Michael E Brown , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1C3DFC.555CEDF7@unix.sh> <3B1CF30E.CC40CBFE@oracle.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list David Brower wrote: > > Alan Robertson wrote: > > > Unfrotuantely, XMLRPC is not nearly powerful enough. I need to send > > basically arbitrarily complex data structures back and forth. It doesn't > > have name/value pairs, or attributes, etc. So, XML-RPC isn't very > > attractive thing to build on for clusters... > > I don't understand this conclusion at all. You can have pretty > much anything you want as a parameter, and XML is nothing but a big > list of name value pairs. The semantics of the parms (nvlist, > attribute, > etc. is up to the callers to agree upon; XML-RPC doesn't have an IDL > that enforces any semantics. And I don't know what kind of complex > data structure can't be represented. As with corba, you need to > understand > the abstraction to do linked lists or trees. Since you can't send > pointers > (which are language dependant in all cases), you need to abstract the > link references and resconstruct them in the appropriate way on the > receiving end. > > Someone else observed that xml-rpc was "an expensive round trip" for > cases where one wished to use broadcast or multicast. I think the > approach to take is that used by one interpretation of the corba > "oneway": have the request contain a "no response needed" flag, > so that the receiver doesn't send a message back. According to the spec, XML-RPC is limited to the following data types: 32-bit int, boolean, string, double, date-time, base64, struct and array. I certainly don't *think* that structs are intended to represent arbitrary name/value pairs. On the other hand, perhaps that was what was intended... (?) I had been thinking in terms of name/value pairs, lists, and strings. I suppose one could translate name/value pairs to structs with no damage done. Hmmm... I don't think the interface needs data typing, but it doesn't hurt anything except for message bulk - unless you need 64-bit ints (which isn't that hard to imagine). If you use XML-RPC for *all* your messages, typing is definitely a problem - since it about doubles the size of messages over the simplest XML representation, which makes it about 3 times the size of current heartbeat messages - which are already thought to be too big... I'm not fond of the idea of compressing the messages just to get rid of text that could just be avoided in the first place... The spec only permits this being sent over HTTP, and there is no provision for "event" messages which have not reply. I suppose one could define certain sets method names to mean "no-reply-needed". Maybe they could even be systematically named - for example with a "-nr" suffix on the name. Maybe we could make the XML transforms be pluggagle, and allow the sender to send either "strict" XML-RPC or a more compact XML representation... If it were hidden under the covers, no one need care... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 17:49:39 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 17:49:32 +0200 Received: from perth.fpcc.net ([207.174.142.141]:19020 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 17:49:16 +0200 Received: from unix.sh (05-058.026.popsite.net [64.24.106.58]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id JAA16210; Tue, 5 Jun 2001 09:49:13 -0600 Message-ID: <3B1CFF44.64F9DFA7@unix.sh> Date: Tue, 05 Jun 2001 09:48:20 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > I'm sorry. I don't think I communicated an important point that I had on > my mind when I was composing the original msg. > > I was not advocating that you use XMLRPC for certain things such as > cluster heartbeat, which have to be slim/trim/no memory leaks/etc. But I > do think it would be easier from a library standpoint if the messages that > you were multicasting were in an XMLRPC-compatible encoding. I think David > Brower's point that XMLRPC is very powerful and can encode just about any > C struct or datatype is valid. What I need is something that encodes *more* than 'C' language datatypes. For example, 'C' has no concept of lists or name/value pairs. We need those. Even heartbeat needs them at the lowest levels. But, if one frees one's mind from thinking about "struct" as the 'C' keyword 'struct' and instead think name/value pairs, then things suddenly seem reasonable. I can see NO reason not to use the same encoding for everything. More to the point, I can see *many* reasons to use the same encoding everywhere - including heartbeats. > What I was thinking of is another layer on top of the shared > hearbeat/cluster manager/etc framework. An XMLRPC transport so that > _third party_ applications have a well-defined interface into the core > cluster managment interfact. I agree. But, I think there should only be one mechanism. Perhaps two encodings, but only one API (and one mechanism). > XMLRPC IS NOT appropriate for > 1) low-level heartbeat: SendHeartbeat() > 2) cluster manager stuff/membership services: > 3) actually performing stonith: StomithKillNode() Actually, I would VERY DEFINITELY want to use the same mechanism for everything. Without a doubt. It might be the case that the sender could select their preferred encoding. At this point, I think I understand enough to say that it's not a matter of semantics or the APIs, but of how the data gets encoded onto the wire that's a concern to me. I suspect that it would be easy to come up with XML-RPC-slim, and the sender could select whether they wanted to send XML-RPC or XML-RPC-slim encodings of their calls. And, it is equally important that calls be allowed to not require any kind of explicit reply, and that there be an addressing mechanism around it which allows multicast/broadcast/directed addressing of packets. Clearly whatever we do will BY DEFINITION be a significant extension over the XML-RPC spec. 1) we won't support HTTP 2) we will allow calls without explicit replies (event notification) 3) we need to indicate which domain the message is within. By that I mean which application program on the destination end will receive the call... This is kinda cool... I also thought of a third extension: 3) Allow calls to multiple nodes in the cluster. There would be another parameter for the list of machines in the cluster to be called, and one for a timeout value. The caller would be notified once all the replies came in, or timeouts had occurred, or (optionally) cluster membership had changed... This would be very nice. It's really what several parts of the HA code would like... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 18:58:49 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 18:58:42 +0200 Received: from inet-mail3.oracle.com ([148.87.2.203]:16300 "EHLO inet-mail3.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 18:58:28 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail3.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f55Gt3v15979; Tue, 5 Jun 2001 09:55:03 -0700 (PDT) Received: from oracle.com ([152.68.53.78]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f55GwLX15470; Tue, 5 Jun 2001 09:58:21 -0700 (PDT) Message-ID: <3B1D0FA8.D56462C0@oracle.com> Date: Tue, 05 Jun 2001 09:58:16 -0700 From: David Brower Organization: Oracle X-Mailer: Mozilla 4.7 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Alan Robertson CC: Michael E Brown , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list > Clearly whatever we do will BY DEFINITION be a significant extension over > the XML-RPC spec. > > 1) we won't support HTTP why not? > 2) we will allow calls without explicit replies (event notification) > 3) we need to indicate which domain the message is within. By that I mean > which application program on the destination end will receive the > call... that is just another argument; no extension needed. > This is kinda cool... > > I also thought of a third extension: > > 3) Allow calls to multiple nodes in the cluster. There would be another > parameter for the list of machines in the cluster to be called, and one for > a timeout value. The caller would be notified once all the replies came in, > or timeouts had occurred, or (optionally) cluster membership had changed... this is an api, not an encoding/message issue. the target list for such an api is a bunch of transport endpoints (host:port) that you expect to be able to process the messages; the api would send the same encoded message to all of them. Someone else complained about the byte size of the encoding. I think we need to be over this kind of issue. I don't believe the bandwidth utilization is particularly relevant anymore. The 3X factor of message size isn't what will kill in a cluster, it is the algorithms affecting the aggregate number of messages. And it's often not the heartbeat that goes kerblooey, it's often the membership determination steps that go off the deep end. So, I'm saying, don't worry about the XML encoding size. Michael Brown wrote: >XMLRPC IS NOT appropriate for > 1) low-level heartbeat: SendHeartbeat() > 2) cluster manager stuff/membership services: > 3) actually performing stonith: StomithKillNode() and Wombat exercsede reluctance to use RPC for low levels. I disagree with this, but maybe because I'm playing loosed with XML-RPC as a concept than he is. I see no reason not to use the encoding, and little reason not to allow it's actual use for just those things, even over http. It may not be efficient, but it does allow configurations that are difficult otherwise -- such as geoclusters with heartbeats tunneling through firewalls. Wombat is right on another point that the ordering needed during cluster transition (barrier services) aren't well served by an event system. I think that is an application for RPC. Now, it is the case that I'm comfortable with RPC systems, having built a corba ORB and gotten used to thinking that way. My impression is that many cluster people (including those I work with) aren't adequately comfortable or familiar with RPC systems, and tend to think in terms of sending raw structures as messages over a channel to a homogeneous system running the same code at the same version on the other end. (And usually over a private network so security issues can be punted). BTW, here are some links to IBM references, including events. Here is a good high-level picture: http://www.research.ibm.com/dss/html/phoenix_ext.html IBM High Availability Cluster Multi-Processor (HACMP) Software http://www.rs6000.ibm.com/software/Apps/hacmp/ http://www.rs6000.ibm.com/resource/technology/ha42ov.html IBM RS6000 Cluster Technology (RSCT) IBM RS6000 Parallel System Support Program (PSSP) Boot Info -- System Data Repository (SDR) http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/cmdsv2/spt1mst07.html Topology Services http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/admin/spa1mst35.html Group Services http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/admin/spa1mst36.html http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/grpsvcs/csg1mst.html Event Management http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/admin/spa1mst37.html http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/evmgt/cse1mst02.html Managing Shared Disks http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/mngdisks/msd1mst02.html General Parallel File System (GPFS) http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/gpfs/install_admin/gpfs1mst02.html -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:32:53 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:32:44 +0200 Received: from mercury.mv.net ([199.125.85.40]:63748 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 19:32:31 +0200 Received: from filesrus (bnh-3-47.mv.com [199.125.99.175]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id NAA26996; Tue, 5 Jun 2001 13:32:26 -0400 (EDT) Message-ID: <011b01c0ede5$d92be480$2d627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 13:34:59 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list An aspect that I haven't yet seen covered (though I might have missed it) is the efficient processing of large chunks of data - e.g., file/database pages (which these days can easily reach 128 KB in size and in the future might reasonably get even larger - in many cases, any size where the seek time/rotational latency eclipses the transfer time can be 'reasonable') and even larger chunks of streaming data. Facilities like (Illinois) Fast Messages at least potentially allow (FM's ability may depend upon the implementation and/or its whims of the moment, but one can get down-and-dirty in, e.g., NT and provide stronger guarantees) the receiver of such data to use the start of the message to recognize the contents and identify an appropriate receptacle (e.g., pre-allocated target buffer[s]) where the payload can be dumped without extraneous copy activity (copying being a non-negligible concern with payloads in the 100 KB - multi-MB range). It may be due to simple ignorance on my part, but it's not clear to me how a generic XML processor would allow such optimization - or other FM-like facilities such as the ability to process some short messages with 'handlers' at interrupt level when appropriate. - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:42:06 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:41:57 +0200 Received: from gw.xkey.com ([206.86.100.52]:9998 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 19:41:48 +0200 Received: (from smtp@localhost) by happy.xkey.com id KAA26212 for ; Tue, 5 Jun 2001 10:41:45 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma026196; Tue Jun 5 10:41:38 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f55Hi1t01764 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 13:44:01 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 13:44:01 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605134401.B1542@wumpus> Mail-Followup-To: linux-cluster References: <3B1CFF44.64F9DFA7@unix.sh> <011b01c0ede5$d92be480$2d627dc7@filesrus> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <011b01c0ede5$d92be480$2d627dc7@filesrus>; from billtodd@foo.mv.com on Tue, Jun 05, 2001 at 01:34:59PM -0400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 01:34:59PM -0400, Bill Todd wrote: > An aspect that I haven't yet seen covered (though I might have missed it) is > the efficient processing of large chunks of data - e.g., file/database pages > (which these days can easily reach 128 KB in size Gee, I didn't realize we were building such a general purpose protocol just to handle our cluster control messages. I'm sure we'll end up with something as elegant as Mach when we're done. -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:47:00 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:46:41 +0200 Received: from smtp7.us.dell.com ([143.166.224.233]:47633 "EHLO smtp7.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 19:46:22 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp7.us.dell.com (8.11.2/8.11.0) with ESMTP id f55HkGD05850; Tue, 5 Jun 2001 12:46:16 -0500 Received: from [143.166.133.124] ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id MAA03417; Tue, 5 Jun 2001 12:46:16 -0500 (CDT) Date: Tue, 5 Jun 2001 12:46:15 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: David Brower cc: Alan Robertson , linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1D0FA8.D56462C0@oracle.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list One point... SOAP is designed to be transport-independent. There is at least one implementation that lets you make (one-way) rpc calls via SMTP as a transport. Thus, it would be very easy to define a "multicast" transport for one-way SOAP calls. OK, that is SOAP, because I've read a bit about it and I know it supports these things. XMLRPC, on the other hand, I can't quite say that I know as much about this aspect, but I suspect it would be very easy to use XMLRPC protocol but specify another transport for delivery. The cool thing about the protocol concept is that you can have several "wrappers" that let you make procedure calls that all use a unified back-end library. What I mean is you can have (HTTP|SMTP|multicast|other) access methods without writing any more code (except the transport layer code, of course). On your other topic, you want Name=Value pairs. Easy. The XMLRPC spec defines: --begin quote-->http://www.xmlrpc.com/spec lowerBound 18 upperBound 139 s can be recursive, any may contain a or any other type, including an , described below. --end quote-- So, you simply define a "struct" such that: Key SomeDataName Value The Value The datatype can be any of the defined datatypes (string/i4/etc...) -- Michael Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:49:11 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:48:54 +0200 Received: from c004-h015.c004.sfo.cp.net ([209.228.14.102]:30194 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Tue, 5 Jun 2001 19:48:32 +0200 Received: (cpmta 5556 invoked from network); 5 Jun 2001 10:48:29 -0700 Received: from lca3054.lss.emc.com (HELO jdarcy6986nk) (168.159.123.54) by smtp.namezero.com (209.228.14.102) with SMTP; 5 Jun 2001 10:48:29 -0700 X-Sent: 5 Jun 2001 17:48:29 GMT Message-ID: <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "David Brower" , "Alan Robertson" Cc: "Michael E Brown" , "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 13:44:14 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list From: "David Brower" > The 3X factor of message > size isn't what will kill in a cluster, it is the algorithms affecting > the aggregate number of messages. I'll come back to this point in a little while, at a point where you seem to've forgotten it. > I see no reason not to use the > encoding I agree. Using an encoding for which parsing tools etc. already exist cannot be a bad thing. In the worst case you're no worse off than if you had "rolled your own", and often you'll be way ahead. > It may not be efficient, but it does > allow configurations that are difficult otherwise -- such as geoclusters > with heartbeats tunneling through firewalls. Anybody who can solve the many more difficult problems involved in implementing geographically-distributed clusters isn't going to lose a single moment of sleep over firewalls. As for efficiency, this is where we get back to your point highlighted previously. RPC forces *gross* algorithmic inefficiencies for operations needed by a cluster infrastructure. RPC is oriented toward point-to-point or broadcast communication, while the most useful conceptual structure for many things in a cluster does(membership, voting, some parts of heartbeating) is likely to be a ring. Aggregating point-to-point RPC into a star topology based around a temporarily-elected communications server doesn't work. Leader election is one of the services a cluster infrastructure *provides*, not one it consumes from elsewhere. The only alternative implementable via RPC is all-to-all, which simply doesn't scale. In short, the conceptual model of RPC forces logical-topology decisions in directions that are far from optimal for clusters. Please, look up some of these algorithms in Lynch or Mullender. Think for a while about whether they can be adapted to use RPC, efficiently and without creating SPOFs. Think about whether the kinds of timeouts and retransmissions inherent in the RPC model really match the requirements for those same sorts of things in a cluster environment. RPC is not a bad thing, but it's a technological mismatch for cluster requirements. > Wombat is right on another point that the ordering needed during > cluster transition (barrier services) aren't well served by an > event system. I think that is an application for RPC. See above for an explanation of why that is not so. > My > impression is that many cluster people (including those I > work with) aren't adequately comfortable or familiar with > RPC systems That's insulting. It would be just as true to say that "RPC people" aren't adequately familiar with anything *but* the RPC model, and in fact I'm tempted to do so, but I think this conversation can move along more productively without such pissing contests. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:53:31 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:53:23 +0200 Received: from mercury.mv.net ([199.125.85.40]:7690 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 19:53:05 +0200 Received: from filesrus (bnh-3-47.mv.com [199.125.99.175]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id NAA04394; Tue, 5 Jun 2001 13:52:58 -0400 (EDT) Message-ID: <012a01c0ede8$b7958940$2d627dc7@filesrus> From: "Bill Todd" To: "David Brower" Cc: "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 13:55:31 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "David Brower" To: "Alan Robertson" Cc: "Michael E Brown" ; "linux-cluster" Sent: Tuesday, June 05, 2001 12:58 PM Subject: Re: A proposal for a General Clustering Framework ... > Someone else complained about the byte size of the encoding. I think > we need to be over this kind of issue. I don't believe the bandwidth > utilization is particularly relevant anymore While bandwidth may be less relevant (though it's seldom *completely* irrelevant), switch-level buffering considerations sometimes make it desirable to be able to fit a message into a single Ethernet packet. ... > impression is that many cluster people (including those I > work with) aren't adequately comfortable or familiar with > RPC systems, and tend to think in terms of sending raw structures > as messages over a channel to a homogeneous system running the > same code at the same version on the other end. (And usually > over a private network so security issues can be punted). Sending raw structures works only in homogeneous-endian environments anyway (though file systems do need the ability to send raw file data as byte-strings, since they haven't a clue what the internal structure of it may be and must leave interpretation up to the application). My own reservations (which my impression is may be shared by others here) center on overheads - e.g., the arbitrary requirement for a response (though a flag requesting datagram-like behavior covers that) and the lack of fine control over marshalling activity (including the potential for [distributed] resource-allocation deadlocks). - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 19:56:22 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 19:56:04 +0200 Received: from smtp7.us.dell.com ([143.166.224.233]:6152 "EHLO smtp7.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 19:56:01 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp7.us.dell.com (8.11.2/8.11.0) with ESMTP id f55HtxD12181; Tue, 5 Jun 2001 12:55:59 -0500 Received: from [143.166.133.124] ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id MAA08206; Tue, 5 Jun 2001 12:55:59 -0500 (CDT) Date: Tue, 5 Jun 2001 12:55:59 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Alan Robertson cc: linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1CFF44.64F9DFA7@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, 5 Jun 2001, Alan Robertson wrote: > > Clearly whatever we do will BY DEFINITION be a significant extension over > the XML-RPC spec. > > 1) we won't support HTTP > 2) we will allow calls without explicit replies (event notification) > 3) we need to indicate which domain the message is within. By that I mean > which application program on the destination end will receive the > call... Alan, This is what I was talking about in my last message about transport-independence. I know SOAP has a clearly defined spec for it, I'm not sure about XMLRPC. Clearly, SOAP is too heavyweight for our task, so I would propose 'backporting' the transport-independence into the XMLRPC protocol. So, to answer your points line-by-line... 1) We simply define a new 'lightweight' transport. 2) Already supported. (or do you mean a UDP-like 'unreliable' transport mechanism?) 3) Trivially done within XMLRPC. You can do either of two ways: 1) have an XMLRPC 'listener' for each application, or define part of the api that specifies which app a message will go to. an example: I have two libraries that I distribute calls to from my XMLRPC wrapper. I do it like this (pseudo-code): xmlrpc->call( "library.ProcedureName", "procedure args" ); The wrapper splits up the name on the ".", and distributes the call to the appropriate library. > > > This is kinda cool... > > I also thought of a third extension: > > 3) Allow calls to multiple nodes in the cluster. There would be another > parameter for the list of machines in the cluster to be called, and one for > a timeout value. The caller would be notified once all the replies came in, > or timeouts had occurred, or (optionally) cluster membership had changed... > The above will be taken care of with a 'multicast' transport. The message payload will be the same as a normal XMLRPC message, it will just be multicast to a list of cluster members. -- Michael Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 20:02:24 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 20:02:15 +0200 Received: from smtp7.us.dell.com ([143.166.224.233]:13067 "EHLO smtp7.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 20:01:56 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp7.us.dell.com (8.11.2/8.11.0) with ESMTP id f55I1sD15896; Tue, 5 Jun 2001 13:01:54 -0500 Received: from [143.166.133.124] ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id NAA11042; Tue, 5 Jun 2001 13:01:54 -0500 (CDT) Date: Tue, 5 Jun 2001 13:01:53 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Jeff Darcy cc: David Brower , Alan Robertson , linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, 5 Jun 2001, Jeff Darcy wrote: > > As for efficiency, this is where we get back to your point highlighted > previously. RPC forces *gross* algorithmic inefficiencies for operations > needed by a cluster infrastructure. RPC is oriented toward point-to-point > or broadcast communication, while the most useful conceptual structure for > many things in a cluster does(membership, voting, some parts of > heartbeating) is likely to be a ring. Aggregating point-to-point RPC into a > star topology based around a temporarily-elected communications server > doesn't work. Leader election is one of the services a cluster > infrastructure *provides*, not one it consumes from elsewhere. The only > alternative implementable via RPC is all-to-all, which simply doesn't scale. > In short, the conceptual model of RPC forces logical-topology decisions in > directions that are far from optimal for clusters. Can't we all just be friends? I think it would be pretty easy to define a new 'multicast' protocol spec to distribute XMLRPC messages. At this point, XMLRPC ceases to be RPC, but just a generic message passing mechanism. You can then implement the star model you are talking about. The model that I am thinking about has libraries on the back-end that are accessible via XMLRPC through several 'transports' to remote systems. This gives the flexibility to use libraries in other languages that are already developed and in use, so we don't have to 'roll our own' protocol and message passing spec. Along with the 'roll our own' idea we would then have to write language bindings for several languages and keep them all up-to-date with the main library. And _that_ isn't scalable. HTH, Michael Brown Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 20:52:36 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 20:52:30 +0200 Received: from mercury.mv.net ([199.125.85.40]:46853 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 20:52:17 +0200 Received: from filesrus (bnh-4-06.mv.com [199.125.99.198]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id OAA02255; Tue, 5 Jun 2001 14:52:13 -0400 (EDT) Message-ID: <019101c0edf0$fecaa040$2d627dc7@filesrus> From: "Bill Todd" To: "Greg Lindahl" , "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <011b01c0ede5$d92be480$2d627dc7@filesrus> <20010605134401.B1542@wumpus> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 14:46:25 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Greg Lindahl" To: "linux-cluster" Sent: Tuesday, June 05, 2001 1:44 PM Subject: Re: A proposal for a General Clustering Framework > On Tue, Jun 05, 2001 at 01:34:59PM -0400, Bill Todd wrote: > > > An aspect that I haven't yet seen covered (though I might have missed it) is > > the efficient processing of large chunks of data - e.g., file/database pages > > (which these days can easily reach 128 KB in size > > Gee, I didn't realize we were building such a general purpose > protocol just to handle our cluster control messages. Perhaps my impression that we were discussing a more general messaging mechanism arose from comments about multiple language bindings and use by applications. Or from my suspicion that using something as general as even an XML subset seems like gross overkill if all we're talking about is a relatively limited set of messages under our own control. But if the assumption indeed is that things like cluster file systems will roll their own message facilities rather than use (or build upon) those under discussion here, then I withdraw my observations. - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 20:54:37 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 20:54:27 +0200 Received: from [213.98.27.110] ([213.98.27.110]:20236 "EHLO hermes.orcero.org") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 20:54:16 +0200 Received: from localhost (localhost.localdomain [127.0.0.1]) by hermes.orcero.org (8.11.0/8.11.0) with ESMTP id f55Kxvp23085 for ; Tue, 5 Jun 2001 20:59:57 GMT Date: Tue, 5 Jun 2001 20:59:57 +0000 (/etc/localtime) From: cc: linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <012a01c0ede8$b7958940$2d627dc7@filesrus> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII To: unlisted-recipients:; (no To-header on input) Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Hello, all! > > we need to be over this kind of issue. I don't believe the bandwidth > > utilization is particularly relevant anymore > > While bandwidth may be less relevant (though it's seldom *completely* > irrelevant), But bandwidth IS relevant to some of us. There is some people doing HP clustering with cheap hardware -that in some countries it is not SO cheap-. Yours: David --------------------- http://www.orcero.org irbis@orcero.org --------------------- Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:01:27 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:01:19 +0200 Received: from [213.98.27.110] ([213.98.27.110]:23308 "EHLO hermes.orcero.org") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:01:05 +0200 Received: from localhost (localhost.localdomain [127.0.0.1]) by hermes.orcero.org (8.11.0/8.11.0) with ESMTP id f55L6Zp23142; Tue, 5 Jun 2001 21:06:35 GMT Date: Tue, 5 Jun 2001 21:06:35 +0000 (/etc/localtime) From: To: Alan Robertson cc: linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1BA89D.313D31DD@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Hello, All! > I've talked off-list to many of you about the prospects for creating a > general clustering framework - usable for many or all HA (and HPC) clusters. > > I'm currently in the process of writing a paper on these framework ideas. IMHO the proponsal of the paper is trying to solve things of the 6th OSI level on the kernel, moving things that are now being resolved by the applications in the kernel. For me, it sounds strange with OSI glases to put the session layer on the kernel clustering framework -this is the overview proponsal main idea-. I also can not find how to but all this XML stuff on HP clusters. Maybe is my lack of understanding, but all the problems that is trying to solve the proponsal were already solved by the application libs that are being used. Cluster transparency and performance -the two main interests of HP clustering- are not improved with this proponsal. And performace can be penaltied. Yours: David --------------------- http://www.orcero.org irbis@orcero.org --------------------- Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:32:59 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:32:43 +0200 Received: from perth.fpcc.net ([207.174.142.141]:13138 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:32:27 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id NAA18058; Tue, 5 Jun 2001 13:32:09 -0600 Message-ID: <3B1D3382.C86658B6@unix.sh> Date: Tue, 05 Jun 2001 13:31:15 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: Jeff Darcy , David Brower , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > On Tue, 5 Jun 2001, Jeff Darcy wrote: > > > > > As for efficiency, this is where we get back to your point highlighted > > previously. RPC forces *gross* algorithmic inefficiencies for operations > > needed by a cluster infrastructure. RPC is oriented toward point-to-point > > or broadcast communication, while the most useful conceptual structure for > > many things in a cluster does(membership, voting, some parts of > > heartbeating) is likely to be a ring. Aggregating point-to-point RPC into a > > star topology based around a temporarily-elected communications server > > doesn't work. Leader election is one of the services a cluster > > infrastructure *provides*, not one it consumes from elsewhere. The only > > alternative implementable via RPC is all-to-all, which simply doesn't scale. > > In short, the conceptual model of RPC forces logical-topology decisions in > > directions that are far from optimal for clusters. > > > Can't we all just be friends? > > > I think it would be pretty easy to define a new 'multicast' protocol spec > to distribute XMLRPC messages. At this point, XMLRPC ceases to be RPC, but > just a generic message passing mechanism. You can then implement the star > model you are talking about. > > The model that I am thinking about has libraries on the back-end that are > accessible via XMLRPC through several 'transports' to remote systems. This > gives the flexibility to use libraries in other languages that are already > developed and in use, so we don't have to 'roll our own' protocol and > message passing spec. Along with the 'roll our own' idea we would then > have to write language bindings for several languages and keep them all > up-to-date with the main library. And _that_ isn't scalable. AFAIK, the existing libraries require HTTP, and expect it. Read the spec - it *certainly* requires it. Maybe we can write libraries in several languages which create/extract the XML out of the "envelope" and only keep those up to date. This wouldn't be so bad... If the libraries in all these other languages have ways of bypassing the HTTP protocol and only dealing with what the spec calls the "payload" along with information extracted from the "envelope", then they might be of use. If they assume HTTP they are useful only under the surgeon's knife ;-) Basically the envelope needs to have the following information in it when you receive it: (authenticated-originator-id, authenticated-origination machine , destination- , destlist (?)) When you send the packet, the originator only supplies the destination list and , and the other two are filled in automatically. I am cool with this idea except for one thing: XML-RPC retains the worst aspect of 'C' calling sequences - which is potentially deadly for an HA system - positional parameters. The usual problem with positional parameters is that they assume that everyone is compiled against the same version of the function definition. Within limits, this is fine for a single non-HA system. However, in an HA system, calls across nodes *cannot* make this assumption. This is because you can *never* take an HA system down all at once and upgrade all the pieces of software at once, then bring it back up. This is in contradiction with the requirements of an HA system never having to go down. For example, let's say we have an XML-RPC function called log_message() which takes two parameters: a format string and an array of arguments of various types. In version 2, we add a priority field to the call so that it is now like this: log_message(priority, format, args) With positional parameters the new code totally confuses the old code when it the new code sends a message to the old code to log something, since its expecting a format string as its first argument, and got an int instead. Yuck. If the code sent this via name/value pairs (or an XML-RPC struct), the message would arrive with a named field it didn't understand, and all the other arguments it could nicely understand: (priotity=3, format="foo", args=(1,2,3)). Now, this is perfectly easy to code, and easy to deal with, and much more expandable and robust, etc. than positional parameters. This is not to say that we shouldn't use XML-RPC, but that anyone who cares about version compatability should only send a single argument of a struct - no matter what else you want to send... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:35:02 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:34:52 +0200 Received: from perth.fpcc.net ([207.174.142.141]:14418 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:34:38 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id NAA18082; Tue, 5 Jun 2001 13:34:34 -0600 Message-ID: <3B1D3415.D3C33F5C@unix.sh> Date: Tue, 05 Jun 2001 13:33:41 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <011b01c0ede5$d92be480$2d627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > An aspect that I haven't yet seen covered (though I might have missed it) is > the efficient processing of large chunks of data - e.g., file/database pages > (which these days can easily reach 128 KB in size and in the future might > reasonably get even larger - in many cases, any size where the seek > time/rotational latency eclipses the transfer time can be 'reasonable') and > even larger chunks of streaming data. > > Facilities like (Illinois) Fast Messages at least potentially allow (FM's > ability may depend upon the implementation and/or its whims of the moment, > but one can get down-and-dirty in, e.g., NT and provide stronger guarantees) > the receiver of such data to use the start of the message to recognize the > contents and identify an appropriate receptacle (e.g., pre-allocated target > buffer[s]) where the payload can be dumped without extraneous copy activity > (copying being a non-negligible concern with payloads in the 100 KB - > multi-MB range). > > It may be due to simple ignorance on my part, but it's not clear to me how a > generic XML processor would allow such optimization - or other FM-like > facilities such as the ability to process some short messages with > 'handlers' at interrupt level when appropriate. The short answer to this is that this protocol is for control messages, not bulk data transfers. The needs of the two are somewhat different, in some ways, radically different. My belief is that no single mechanism can meet the needs of the two parts well. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:43:28 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:43:19 +0200 Received: from perth.fpcc.net ([207.174.142.141]:19538 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:43:06 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id NAA18148; Tue, 5 Jun 2001 13:43:00 -0600 Message-ID: <3B1D360F.2A9262BA@unix.sh> Date: Tue, 05 Jun 2001 13:42:07 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: David Brower , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <012a01c0ede8$b7958940$2d627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > > Sending raw structures works only in homogeneous-endian environments anyway > (though file systems do need the ability to send raw file data as > byte-strings, since they haven't a clue what the internal structure of it > may be and must leave interpretation up to the application). My own > reservations (which my impression is may be shared by others here) center on > overheads - e.g., the arbitrary requirement for a response (though a flag > requesting datagram-like behavior covers that) I think we agree that one can simply send an event-type RPC call, and that can take care of this expectation. > and the lack of fine control > over marshalling activity (including the potential for [distributed] > resource-allocation deadlocks). I'd like for you to elaborate some more on these two items: lack of fine control over marshalling activities potential for [distributed] resource-allocation deadlocks Thanks! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:55:43 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:55:34 +0200 Received: from smtp6.us.dell.com ([143.166.83.101]:65040 "EHLO smtp6.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:55:20 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp6.us.dell.com (8.11.0/8.11.0) with ESMTP id f55JtDG17415; Tue, 5 Jun 2001 14:55:13 -0500 Received: from [143.166.133.124] ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id OAA13152; Tue, 5 Jun 2001 14:55:07 -0500 (CDT) Date: Tue, 5 Jun 2001 14:55:07 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Alan Robertson cc: Jeff Darcy , David Brower , linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1D3382.C86658B6@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, 5 Jun 2001, Alan Robertson wrote: > > AFAIK, the existing libraries require HTTP, and expect it. Read the spec - > it *certainly* requires it. Maybe we can write libraries in several > languages which create/extract the XML out of the "envelope" and only keep > those up to date. This wouldn't be so bad... If the libraries in all these > other languages have ways of bypassing the HTTP protocol and only dealing > with what the spec calls the "payload" along with information extracted from > the "envelope", then they might be of use. If they assume HTTP they are > useful only under the surgeon's knife ;-) > I use the Perl code as an example because that is what I am familiar with. The Perl code implemented SOAP first, then the implementor went back and implemented XMLRPC because they share a lot of the same code. The perl SOAP/XMLRPC library is transport-independent. You basically say, "I want to make XMLRPC calls, and by the way, please use such-and-such transport." I haven't done it, so this might be a leap, but it looks pretty straightforward to implement another transport. So, the cluster framework would require a bit of surgery to be useful in another language (basically, reimplement whatever transport you come up with), but it would be a one-time cost. The transport code would not have to keep up with library API changes. -- Michael Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 21:57:56 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 21:57:48 +0200 Received: from perth.fpcc.net ([207.174.142.141]:23378 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 21:57:35 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id NAA18222; Tue, 5 Jun 2001 13:57:32 -0600 Message-ID: <3B1D3975.60FC764D@unix.sh> Date: Tue, 05 Jun 2001 13:56:37 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > On Tue, 5 Jun 2001, Alan Robertson wrote: > > > > > Clearly whatever we do will BY DEFINITION be a significant extension over > > the XML-RPC spec. > > > > 1) we won't support HTTP > > 2) we will allow calls without explicit replies (event notification) > > 3) we need to indicate which domain the message is within. By that I mean > > which application program on the destination end will receive the > > call... > > Alan, > This is what I was talking about in my last message about > transport-independence. I know SOAP has a clearly defined spec for it, I'm > not sure about XMLRPC. Clearly, SOAP is too heavyweight for our task, so I > would propose 'backporting' the transport-independence into the XMLRPC > protocol. > > So, to answer your points line-by-line... > > 1) We simply define a new 'lightweight' transport. The transport in the architecture heartbeat implements provides for multiple "pluggable" transports. The framework will do the same. So, we don't define a single transport, but a transport API, and leave the rest to the plugins. > 2) Already supported. (or do you mean a UDP-like 'unreliable' transport > mechanism?) Neither. I mean a sender-API difference. On the wire there's no difference. But, in the requestor, the API code needs to know not to wait for a reply. > 3) Trivially done within XMLRPC. You can do either of two ways: 1) have an > XMLRPC 'listener' for each application, or define part of the api that > specifies which app a message will go to. This is clearly allowed for by the spec (see my previous email). > > > > I also thought of a third extension: > > > > 3) Allow calls to multiple nodes in the cluster. There would be another > > parameter for the list of machines in the cluster to be called, and one for > > a timeout value. The caller would be notified once all the replies came in, > > or timeouts had occurred, or (optionally) cluster membership had changed... > > > > The above will be taken care of with a 'multicast' transport. The message > payload will be the same as a normal XMLRPC message, it will just be > multicast to a list of cluster members. On the wire, the payload's the same, but in the API, there's a huge difference in what has to happen... In my view, the application shouldn't specify transport, so from the point of view of the application, it's not a multicast transport, but simply a list of destinations. The transport layer needs to make them show up at their destinations somehow. It might ALL be through serial links if that's how you have it configured underneath. But, this is not the applications's concern. The conclusions I would draw are these: The XML-RPC payload is OK as is (but be careful about positional parameters - one parameter is best) The data on the wire will probably not match the XML-RPC spec's idea of what it should be. (different "envelope"). The API interface code will also have to be different. So, we've saved (kept) the parsing, but everything else is broken from the point of view of perfection. This is not intended as a complaint, but as a statement of fact as best I understand it. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:02:21 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:02:12 +0200 Received: from perth.fpcc.net ([207.174.142.141]:29522 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:02:02 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id OAA18308; Tue, 5 Jun 2001 14:01:53 -0600 Message-ID: <3B1D3A7D.BD4F0E27@unix.sh> Date: Tue, 05 Jun 2001 14:01:01 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: Greg Lindahl , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <011b01c0ede5$d92be480$2d627dc7@filesrus> <20010605134401.B1542@wumpus> <019101c0edf0$fecaa040$2d627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > ----- Original Message ----- > From: "Greg Lindahl" > To: "linux-cluster" > Sent: Tuesday, June 05, 2001 1:44 PM > Subject: Re: A proposal for a General Clustering Framework > > > On Tue, Jun 05, 2001 at 01:34:59PM -0400, Bill Todd wrote: > > > > > An aspect that I haven't yet seen covered (though I might have missed > it) is > > > the efficient processing of large chunks of data - e.g., file/database > pages > > > (which these days can easily reach 128 KB in size > > > > Gee, I didn't realize we were building such a general purpose > > protocol just to handle our cluster control messages. > > Perhaps my impression that we were discussing a more general messaging > mechanism arose from comments about multiple language bindings and use by > applications. Or from my suspicion that using something as general as even > an XML subset seems like gross overkill if all we're talking about is a > relatively limited set of messages under our own control. > > But if the assumption indeed is that things like cluster file systems will > roll their own message facilities rather than use (or build upon) those > under discussion here, then I withdraw my observations. The message transport under discussion is intended to be for control data, not bulk application data. And, most likely a filesystem would have to interface with these mechanisms, to get needed services. They might even want to use them for some of their own control needs. But, they're not intended for bulk data transport. Thanks for the question though, since we've all been discussing this over a period of years we forgot to state that assumption. Thanks! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:02:41 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:02:34 +0200 Received: from smtp6.us.dell.com ([143.166.83.101]:24581 "EHLO smtp6.us.dell.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:02:26 +0200 Received: from koskov.us.dell.com (koskov.us.dell.com [143.166.217.48]) by smtp6.us.dell.com (8.11.0/8.11.0) with ESMTP id f55K2PG21677; Tue, 5 Jun 2001 15:02:25 -0500 Received: from [143.166.133.124] ([143.166.133.124]) by koskov.us.dell.com (8.8.8+Sun/8.8.8) with ESMTP id PAA17669; Tue, 5 Jun 2001 15:02:24 -0500 (CDT) Date: Tue, 5 Jun 2001 15:02:24 -0500 (CDT) From: Michael E Brown X-X-Sender: Reply-To: Michael E Brown To: Alan Robertson cc: Jeff Darcy , David Brower , linux-cluster Subject: Re: A proposal for a General Clustering Framework In-Reply-To: <3B1D3382.C86658B6@unix.sh> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, 5 Jun 2001, Alan Robertson wrote: > > I am cool with this idea except for one thing: XML-RPC retains the worst > aspect of 'C' calling sequences - which is potentially deadly for an HA > system - positional parameters. > ??? lowerBound 18 I can clearly see the tage right there in the spec. You just define your API like this: log_message( xmlrpc_struct ); In version 1, your struct looks like this: format someformat args "something to log" "something else" In version 2 of your log_message API, your struct now looks like this: priority someformat format someformat args "something to log" "something else" As long as your code always access parameters in the struct by name, you are fine. -- Michael > The usual problem with positional parameters is that they assume that > everyone is compiled against the same version of the function definition. > Within limits, this is fine for a single non-HA system. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:04:02 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:03:54 +0200 Received: from gw.xkey.com ([206.86.100.52]:50949 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:03:35 +0200 Received: (from smtp@localhost) by happy.xkey.com id NAA08291 for ; Tue, 5 Jun 2001 13:03:33 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma008262; Tue Jun 5 13:03:24 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f55K5lK02320 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 16:05:47 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 16:05:47 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605160547.A2291@wumpus> Mail-Followup-To: linux-cluster References: <3B1D3975.60FC764D@unix.sh> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1D3975.60FC764D@unix.sh>; from alanr@unix.sh on Tue, Jun 05, 2001 at 01:56:37PM -0600 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 01:56:37PM -0600, Alan Robertson wrote: > > 1) We simply define a new 'lightweight' transport. > > The transport in the architecture heartbeat implements provides for multiple > "pluggable" transports. The framework will do the same. So, we don't > define a single transport, but a transport API, and leave the rest to the > plugins. This is good. In particular, you could imagine a simple pub/sub system that would broadcast all published data, and if the local system doesn't have a subscription, it discards it. Or you could have a central server that distributes data only to the actual subscribers. Can you write an API flexible enough to allow both implementations? -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:05:12 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:04:53 +0200 Received: from perth.fpcc.net ([207.174.142.141]:31570 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:04:41 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id OAA18334; Tue, 5 Jun 2001 14:04:13 -0600 Message-ID: <3B1D3B09.8464559D@unix.sh> Date: Tue, 05 Jun 2001 14:03:21 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: irbis@orcero.org CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list irbis@orcero.org wrote: > > Hello, all! > > > > we need to be over this kind of issue. I don't believe the bandwidth > > > utilization is particularly relevant anymore > > > > While bandwidth may be less relevant (though it's seldom *completely* > > irrelevant), > > But bandwidth IS relevant to some of us. There is some people doing HP > clustering with cheap hardware -that in some countries it is not SO > cheap-. Perhaps we should make the payload encoding a plugin, then you can invent your own, which might be smaller and faster. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:10:56 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:10:48 +0200 Received: from perth.fpcc.net ([207.174.142.141]:54610 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:10:37 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id OAA18406; Tue, 5 Jun 2001 14:10:31 -0600 Message-ID: <3B1D3C83.176E1B35@unix.sh> Date: Tue, 05 Jun 2001 14:09:39 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: irbis@orcero.org CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list irbis@orcero.org wrote: > > Hello, All! > > > I've talked off-list to many of you about the prospects for creating a > > general clustering framework - usable for many or all HA (and HPC) clusters. > > > > I'm currently in the process of writing a paper on these framework ideas. > > IMHO the proponsal of the paper is trying to solve things of the 6th OSI > level on the kernel, moving things that are now being resolved by the > applications in the kernel. > > For me, it sounds strange with OSI glases to put the session layer on the > kernel clustering framework -this is the overview proponsal main idea-.'' Kernel clustering? The proposal's main thrust isn't actually about XML, but it is one of several key pieces. This is only for *control* data. Not for application-level data at all. > I also can not find how to but all this XML stuff on HP clusters. Maybe > is my lack of understanding, but all the problems that is trying to solve > the proponsal were already solved by the application libs that are being > used. Not at all. > Cluster transparency and performance -the two main interests of HP > clustering- are not improved with this proponsal. And performace can be > penaltied. Performance might be slightly penalized. Probably not much - since the only part of the paper under discussion at the moment seems to be using XML to transport control data. IMHO, Cluster transperency is improved, not made worse by the proposal. Hopefully, control data doesn't constitute the majority of the work you are doing. If it does, then you might suffer a 20-30% slowdown. If not, it'll more likely be 1-2%. And, the tradeoff is that more developers will be developing more things that will work on your cluster. And, if you invent components that plug into the framework that make it work faster for you, then PLEASE, PLEASE, PLEASE do that! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:16:30 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:16:22 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:16077 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:16:14 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f55KB8D12839; Tue, 5 Jun 2001 13:11:09 -0700 (PDT) Received: from oracle.com (dbrower-sun.us.oracle.com [130.35.180.64]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f55KGAX01067; Tue, 5 Jun 2001 13:16:10 -0700 (PDT) Message-ID: <3B1D3E09.3A3976EA@oracle.com> Date: Tue, 05 Jun 2001 13:16:09 -0700 From: David Brower Organization: Oracle Corporation X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Darcy , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Jeff Darcy wrote: > > As for efficiency, this is where we get back to your point highlighted > previously. RPC forces *gross* algorithmic inefficiencies for operations > needed by a cluster infrastructure. RPC is oriented toward point-to-point > or broadcast communication, while the most useful conceptual structure for > many things in a cluster does(membership, voting, some parts of > heartbeating) is likely to be a ring. Aggregating point-to-point RPC into a > star topology based around a temporarily-elected communications server > doesn't work. Leader election is one of the services a cluster > infrastructure *provides*, not one it consumes from elsewhere. The only > alternative implementable via RPC is all-to-all, which simply doesn't scale. > In short, the conceptual model of RPC forces logical-topology decisions in > directions that are far from optimal for clusters. > > Please, look up some of these algorithms in Lynch or Mullender. Think for a > while about whether they can be adapted to use RPC, efficiently and without > creating SPOFs. Think about whether the kinds of timeouts and > retransmissions inherent in the RPC model really match the requirements for > those same sorts of things in a cluster environment. RPC is not a bad > thing, but it's a technological mismatch for cluster requirements. I will confess to not understanding what Mr. Darcy has written here. I do not see how a point to point RPC from one node to another is any different than a single hop message in a ring. I don't see where timeouts and retransmissions come into play; I don't understand what SPOF he thinks is present in an RPC. We've already been assuming the use of RPC-like encoding for messages that are unacknowledged and possibly broad- or multicast. These are capabilities present in many RPC systems (but not XML-RPC in present form). I think Mr. Darcy is making conclusions based on knowledge of some particular RPC systems. He may be assuming the presence of routing daemons and things like name services in the RPC path; there is nothing requiring such things in many systems. > > Wombat is right on another point that the ordering needed during > > cluster transition (barrier services) aren't well served by an > > event system. I think that is an application for RPC. > > See above for an explanation of why that is not so. I still don't see it. Elections involve messages between voters; those messages could be RPCs. Maybe they want to be reliable/acknowledged messagees, maybe not, depending on the election algorithm. > > My > > impression is that many cluster people (including those I > > work with) aren't adequately comfortable or familiar with > > RPC systems > > That's insulting. It would be just as true to say that "RPC people" aren't > adequately familiar with anything *but* the RPC model, and in fact I'm > tempted to do so, but I think this conversation can move along more > productively without such pissing contests. Sorry, you found it insulting. It wasn't intended to be anything but a prod to do more research before concluding RPC systems are inappropriate. If I were going to slam RPC, and XML-RPC/SOAP in particular, it would be the cpu expense of packing/unpacking the messages-- and that is only a problem for the heartbeats, which are frequent enough to cause background load. I still don't agree with bandwidth complaints, because I don't believe a cluster of really low-bandwidth is going to want to heartbeat at a rate that presses the speed of the link for many reasons. thanks, -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:18:50 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:18:41 +0200 Received: from perth.fpcc.net ([207.174.142.141]:8531 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:18:33 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id OAA18423; Tue, 5 Jun 2001 14:12:38 -0600 Message-ID: <3B1D3D00.17F6AA6A@unix.sh> Date: Tue, 05 Jun 2001 14:11:44 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Michael E Brown CC: Jeff Darcy , David Brower , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Michael E Brown wrote: > > On Tue, 5 Jun 2001, Alan Robertson wrote: [snip] > I haven't done it, so this might be a leap, but it looks pretty > straightforward to implement another transport. > > So, the cluster framework would require a bit of surgery to be useful in > another language (basically, reimplement whatever transport you come up > with), but it would be a one-time cost. The transport code would not have > to keep up with library API changes Cool. And all the different transports under you will meet the same API. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 22:23:01 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 22:22:43 +0200 Received: from perth.fpcc.net ([207.174.142.141]:41043 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 22:22:32 +0200 Received: from unix.sh (02-101.026.popsite.net [64.24.105.101]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id OAA18480; Tue, 5 Jun 2001 14:22:29 -0600 Message-ID: <3B1D3F51.537D23@unix.sh> Date: Tue, 05 Jun 2001 14:21:37 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Greg Lindahl CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Greg Lindahl wrote: > > On Tue, Jun 05, 2001 at 01:56:37PM -0600, Alan Robertson wrote: > > > > 1) We simply define a new 'lightweight' transport. > > > > The transport in the architecture heartbeat implements provides for multiple > > "pluggable" transports. The framework will do the same. So, we don't > > define a single transport, but a transport API, and leave the rest to the > > plugins. > > This is good. In particular, you could imagine a simple pub/sub system > that would broadcast all published data, and if the local system > doesn't have a subscription, it discards it. Or you could have a > central server that distributes data only to the actual > subscribers. Can you write an API flexible enough to allow both > implementations? The reason why I didn't worry too much about the event system was because it's pretty easy to write one as an application in the framework. You could even make it where all events are seen by all systems in the same order... In fact, you could even write one pretty easily on the heartbeat API. If you want events received in the same order everywhere, you might have to do a little more work, but it's not really that bad... Basically, you have an event daemon on each machine. Each of these daemons has knowledge of who their local clients are. Each daemon is responsible for broadcasting events out, and notifying their clients of new events that they have subscribed for. It's just another cluster-aware app, which you can start up when/if someone requests it. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:11:56 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:11:48 +0200 Received: from mercury.mv.net ([199.125.85.40]:5133 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 23:11:31 +0200 Received: from filesrus (bnh-7-20.mv.com [199.125.98.148]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id RAA19644; Tue, 5 Jun 2001 17:11:23 -0400 (EDT) Message-ID: <005a01c0ee04$71ab3760$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <3B1D3382.C86658B6@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 17:13:59 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Alan Robertson" To: "Michael E Brown" Cc: "Jeff Darcy" ; "David Brower" ; "linux-cluster" Sent: Tuesday, June 05, 2001 3:31 PM Subject: Re: A proposal for a General Clustering Framework ... > I am cool with this idea except for one thing: XML-RPC retains the worst > aspect of 'C' calling sequences - which is potentially deadly for an HA > system - positional parameters. > > The usual problem with positional parameters is that they assume that > everyone is compiled against the same version of the function definition. > Within limits, this is fine for a single non-HA system. > > However, in an HA system, calls across nodes *cannot* make this assumption. > This is because you can *never* take an HA system down all at once and > upgrade all the pieces of software at once, then bring it back up. This is > in contradiction with the requirements of an HA system never having to go > down. While I agree with the observation expressed elsewhere that name/value pairs would seem to address the problem you describe, I'll also point out that it is often easier (and/or safer) to define upgrades (at least those that affect protocol syntax or semantics) such that all nodes get upgraded (one by one, with no general outage) to new software that continues to use the old protocols - and then after all upgrades are complete, a synchronous cluster-wide transition to the new protocols occurs. Among other things, this avoids proliferation of run-time conditionals that depend on the capabilities of each member. In general, if one can avoid the requirement to support version heterogeneity across a cluster, life can be a lot simpler. While version heterogeneity is clearly required in general networks (where systems are under the control of completely independent organizations), it's not clear that (at least what I think of as) a cluster can't reasonably impose such a constraint. - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:17:59 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:17:49 +0200 Received: from mercury.mv.net ([199.125.85.40]:56078 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 23:17:37 +0200 Received: from filesrus (bnh-7-20.mv.com [199.125.98.148]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id RAA21542; Tue, 5 Jun 2001 17:17:27 -0400 (EDT) Message-ID: <006101c0ee05$49d7cae0$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <3B1D3382.C86658B6@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 17:20:03 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Please ignore the comment about positional parameters I just sent: on re-reading your post, I see that it was the RPC argument sequence rather than the XML representation that caused the issue. Of course, if all you're going to use is single-argument RPCs (to avoid that issue), that further calls into question whether the RPC framework is really buying you all that much. - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:25:44 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:25:27 +0200 Received: from c004-h005.c004.sfo.cp.net ([209.228.14.76]:63379 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Tue, 5 Jun 2001 23:25:23 +0200 Received: (cpmta 1415 invoked from network); 5 Jun 2001 14:25:16 -0700 Received: from lca3054.lss.emc.com (HELO jdarcy6986nk) (168.159.123.54) by smtp.namezero.com (209.228.14.76) with SMTP; 5 Jun 2001 14:25:16 -0700 X-Sent: 5 Jun 2001 21:25:16 GMT Message-ID: <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "David Brower" , "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 17:21:02 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list From: "David Brower" > I will confess to not understanding what Mr. Darcy has written > here. I do not see how a point to point RPC from one node to another > is any different than a single hop message in a ring. They're different because RPC, by definition, involves a reply to indicate completion (and possibly a result). Bear in mind that RPC is a general and often-abused term; there are many kinds of RPC, and some RPC packages contain non-RPC functionality. Whatever Sun RPC or ONC RPC or XML-RPC or any other RPC actually looks like is not the issue; the generic category "RPC" still represents certain concepts and communications models that many see as non-complementary to clustering needs. You could send RPCs around a ring, such that A calls B and B calls C and when C calls A it completes the cycle so the replies all propagate along the reverse path. There's no point, though. The hard part is setting up and maintaining the logical topology anyway; once you've done that, using RPC instead of simple messaging often doesn't buy anything. It just doubles your message traffic - only linearly, true - and slows things down. > I don't see where > timeouts and retransmissions come into play They come into play in several ways. Every protocol that one might use - e.g. for membership or consensus - has a very particular set of requirements regarding message reliability, ordering, etc. If you fail to meet those requirements, the protocol might fail. What's less obvious is that *exceeding* the protocol's requirements can be just as dangerous. For example, a protocol might be able to tolerate a certain amount of message loss, but not duplication. Add retransmission and you add a potential for duplication, and you might well break your consensus protocol. That might seem far-fetched to you, but I've had to spend days tracking down just such requirements mismatches in production environments. A closely related problem is when a too-ambitious low-level protocol constrains message behavior in such a way as to force a higher-level protocol into its least efficient paths. BTW, don't try to posit XID caches as the solution; they introduce their own set of rather nasty problems. > I don't understand what > SPOF he thinks is present in an RPC. The SPOF is not in the RPC itself, but in the topology that RPC - indirectly - forces upon the system. What I was thinking of, specifically, is the SPOF represented by the hub in a star topology. > I think Mr. Darcy is making conclusions based on knowledge of some > particular RPC systems. I really wish you wouldn't make such assumptions. You don't know me or my background at all. You have *no* reason to make assumptions about what I do or do not know. I'm trying not to make similar assumptions about you, except where your own words indicate presence or absence of specific knowledge. You'd do well to at least consider the possibility that those who disagree with you do so out of hard-won knowledge rather than ignorance. > I still don't see it. Elections involve messages between > voters; those messages could be RPCs. Maybe they want to > be reliable/acknowledged messagees, maybe not, depending on > the election algorithm. It's a chicken and egg problem. Before you can elect a leader, you have to determine membership. If your membership algorithm relies on already having a leader elected to act as a coordinator or communications hub, you'll never be able to bootstrap the whole thing into existence. The relationships between the protocols used in a cluster are numerous and subtle. Messaging code is affected by membership changes, which should be coordinated with event handling to allow graceful exit of a node, the event handling depends on consensus algorithms which in turn depend on membership...and so on. The essence of the problem is that the best-known and most reliable algorithms for these tasks are based on pure messaging. Adding RPC to this volatile mix doesn't really solve anything, and accomodating RPC's different "expectations" and behaviors just adds difficulty to an already-difficult problem set. > It wasn't intended to be > anything but a prod to do more research before concluding > RPC systems are inappropriate. And I'm hereby prodding you to do more research before concluding RPC *is* appropriate. Theories about how things "should" work don't cut much ice when several of the people in the group have been involved in making systems that *actually* work and those people seem pretty uniformly skeptical about the applicability of RPC. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:41:50 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:41:42 +0200 Received: from gw.xkey.com ([206.86.100.52]:18954 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 23:41:28 +0200 Received: (from smtp@localhost) by happy.xkey.com id OAA00834 for ; Tue, 5 Jun 2001 14:41:26 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma000803; Tue Jun 5 14:41:21 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f55Lhio02667 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 17:43:44 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 17:43:43 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605174343.A2650@wumpus> Mail-Followup-To: linux-cluster References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com>; from linuxguy@tambreet.com on Tue, Jun 05, 2001 at 05:21:02PM -0400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 05:21:02PM -0400, Jeff Darcy wrote: > They're different because RPC, by definition, involves a reply to indicate > completion (and possibly a result). Bear in mind that RPC is a general and > often-abused term; there are many kinds of RPC, and some RPC packages > contain non-RPC functionality. I suspect most people here are using the non-RPC form of RPC, which would explain why they went "Huh?" to your note. And that's why there's more heat than light happening. > They come into play in several ways. Every protocol that one might use - > e.g. for membership or consensus - has a very particular set of requirements > regarding message reliability, ordering, etc. That's correct. And if you make these requirements explicit and available to the lower layers, they can provide appropriate services. The usual problem is that these requirements aren't written down anywhere or are simply not known by the programmer. > What's less obvious is that > *exceeding* the protocol's requirements can be just as dangerous. For > example, a protocol might be able to tolerate a certain amount of message > loss, but not duplication. Add retransmission and you add a potential for > duplication, and you might well break your consensus protocol. This is an odd example. The requirement of "no duplicates" was simply not specified. I wouldn't calling that "exceeding the protocol's requirements". Perhaps a better example is an application which does not require ordered delivery using a protocol which only provides ordered delivery. The application may see significantly higher latency when a message is lost. Legion (a distributed system whose lowest layer is dataflow) introduced a taxonomy of requirements, but it could be that there are other existing taxonomies. I'm not up on the literature. > The SPOF is not in the RPC itself, but in the topology that RPC - > indirectly - forces upon the system. What I was thinking of, specifically, > is the SPOF represented by the hub in a star topology. And that depends on what your definition of RPC is. > The > essence of the problem is that the best-known and most reliable algorithms > for these tasks are based on pure messaging. Adding RPC to this volatile > mix doesn't really solve anything, and accomodating RPC's different > "expectations" and behaviors just adds difficulty to an already-difficult > problem set. And that depends on what your definition of RPC is. -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:57:46 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:57:38 +0200 Received: from mercury.mv.net ([199.125.85.40]:65033 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 23:57:23 +0200 Received: from filesrus (bnh-3-15.mv.com [199.125.99.143]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id RAA03483; Tue, 5 Jun 2001 17:57:19 -0400 (EDT) Message-ID: <009601c0ee0a$dbb798a0$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <012a01c0ede8$b7958940$2d627dc7@filesrus> <3B1D360F.2A9262BA@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 17:59:55 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Alan Robertson" To: "Bill Todd" Cc: "David Brower" ; "linux-cluster" Sent: Tuesday, June 05, 2001 3:42 PM Subject: Re: A proposal for a General Clustering Framework > Bill Todd wrote: ... > > and the lack of fine control > > over marshalling activity (including the potential for [distributed] > > resource-allocation deadlocks). > > I'd like for you to elaborate some more on these two items: The comment was made in the context of a more general-purpose mechanism than in fact seems to be under consideration here, and in particular in the context of potentially medium-to-large messages (or portions of messages) where at least the simpler credit-based communication mechanisms may not work well (e.g., specific space may be pre-allocated at the receiver). It's been many years since I thought about distributed resource allocation deadlocks in detail, but in general they involve binary (or longer-cycle) deadly embraces where backed-up request activity consumes the (usually buffer) resources required to support shipment and/or receipt of the responses that would free up the tied-up request resources. One common approach to the problem includes having requestors reserve response-receipt space prior to sending requests and having responders always reserve sufficient space to send at least one response (which is guaranteed not to block, since the requestor has guaranteed the space to receive it). The responder must be able to 'stream' larger responses through buffers of limited size if the reserved response space requirement is to be bounded. It would certainly be possible to create an RPC mechanism that would do this, but it's not something I'd assume was there without checking first. Sorry for the fuzziness above - as I said, it's been a long time. And in any event, with relatively small messages (in particular, with a known upper size bound) the problem is much more tractable. - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 5 23:58:57 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 5 Jun 2001 23:58:47 +0200 Received: from mercury.mv.net ([199.125.85.40]:16650 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 5 Jun 2001 23:58:34 +0200 Received: from filesrus (bnh-3-15.mv.com [199.125.99.143]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id RAA03670; Tue, 5 Jun 2001 17:57:59 -0400 (EDT) Message-ID: <009701c0ee0a$f362dc80$94627dc7@filesrus> From: "Bill Todd" To: "Jeff Darcy" , "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 18:00:35 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list While I tend to agree that for the kinds of messages under discussion here (as well as the kinds I'm interested in for a cluster FS) RPC adds complexity while offering minimal (at best) value, the existing discussion has made it pretty clear that the 'RPC' they're talking about includes a datagram-like facility (inappropriate as that may be for something called 'RPC'). Given that (and the ability to use it to avoid the kinds of problems uncontrolled retries and consequently lengthier time-outs can cause), the use of this kind of 'RPC' seems at worst a bit silly. And since the facility won't be designed to be useful for the kinds of things I'm interested in anyway, I can't get too excited about that - though I'd hoped it would be a lower-level, more generally-useful facility that I *could* use. - bill ----- Original Message ----- From: "Jeff Darcy" To: "David Brower" ; "linux-cluster" Sent: Tuesday, June 05, 2001 5:21 PM Subject: Re: A proposal for a General Clustering Framework Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 00:09:32 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 00:09:24 +0200 Received: from perth.fpcc.net ([207.174.142.141]:24406 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 00:09:07 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id QAA19345; Tue, 5 Jun 2001 16:08:59 -0600 Message-ID: <3B1D5841.C099E0E@unix.sh> Date: Tue, 05 Jun 2001 16:08:01 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Greg Lindahl , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Greg Lindahl wrote: > > > > On Tue, Jun 05, 2001 at 01:56:37PM -0600, Alan Robertson wrote: > > > > > > 1) We simply define a new 'lightweight' transport. > > > > > > The transport in the architecture heartbeat implements provides for multiple > > > "pluggable" transports. The framework will do the same. So, we don't > > > define a single transport, but a transport API, and leave the rest to the > > > plugins. > > > > This is good. In particular, you could imagine a simple pub/sub system > > that would broadcast all published data, and if the local system > > doesn't have a subscription, it discards it. Or you could have a > > central server that distributes data only to the actual > > subscribers. Can you write an API flexible enough to allow both > > implementations? > > The reason why I didn't worry too much about the event system was because > it's pretty easy to write one as an application in the framework. You could > even make it where all events are seen by all systems in the same order... > > In fact, you could even write one pretty easily on the heartbeat API. If > you want events received in the same order everywhere, you might have to do > a little more work, but it's not really that bad... > > Basically, you have an event daemon on each machine. Each of these daemons > has knowledge of who their local clients are. > > Each daemon is responsible for broadcasting events out, and notifying their > clients of new events that they have subscribed for. > > It's just another cluster-aware app, which you can start up when/if someone > requests it. It was just pointed out to me that I didn't answer the question as asked. Maybe this is a better answer :-) Sorry. You could build either type of implementation on top of the heartbeat API easily. Actually there are three kinds that come to mind: A master-node concept, where all events are serialized through this one machine. A distributed version with event-order coherency, where all events are seen in the same order by all machines. A distributed version without event-order coherency. I suppose you could install all types simultaneously, but it sounds like it wouldn't be very useful, and would confuse the apps. If you design the API properly, you should be able to hide the differences between the first two from the application. The third one has different semantics, and shouldn't be made to look like the other two... [and it's probably not that useful anyway...] -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 00:19:42 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 00:19:35 +0200 Received: from perth.fpcc.net ([207.174.142.141]:26710 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 00:19:19 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id QAA19377; Tue, 5 Jun 2001 16:19:15 -0600 Message-ID: <3B1D5AAE.E1375580@unix.sh> Date: Tue, 05 Jun 2001 16:18:22 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3382.C86658B6@unix.sh> <005a01c0ee04$71ab3760$94627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > ----- Original Message ----- > From: "Alan Robertson" > To: "Michael E Brown" > Cc: "Jeff Darcy" ; "David Brower" > ; "linux-cluster" > Sent: Tuesday, June 05, 2001 3:31 PM > Subject: Re: A proposal for a General Clustering Framework > > ... > > > I am cool with this idea except for one thing: XML-RPC retains the worst > > aspect of 'C' calling sequences - which is potentially deadly for an HA > > system - positional parameters. > > > > The usual problem with positional parameters is that they assume that > > everyone is compiled against the same version of the function definition. > > Within limits, this is fine for a single non-HA system. > > > > However, in an HA system, calls across nodes *cannot* make this > assumption. > > This is because you can *never* take an HA system down all at once and > > upgrade all the pieces of software at once, then bring it back up. This > is > > in contradiction with the requirements of an HA system never having to go > > down. > > While I agree with the observation expressed elsewhere that name/value pairs > would seem to address the problem you describe, I'll also point out that it > is often easier (and/or safer) to define upgrades (at least those that > affect protocol syntax or semantics) such that all nodes get upgraded (one > by one, with no general outage) to new software that continues to use the > old protocols - and then after all upgrades are complete, a synchronous > cluster-wide transition to the new protocols occurs. Among other things, > this avoids proliferation of run-time conditionals that depend on the > capabilities of each member. This is exactly what this technique is designed to support - allowing the new code to use the old data formats. > In general, if one can avoid the requirement to support version > heterogeneity across a cluster, life can be a lot simpler. While version > heterogeneity is clearly required in general networks (where systems are > under the control of completely independent organizations), it's not clear > that (at least what I think of as) a cluster can't reasonably impose such a > constraint. It's very clear that, in general, one cannot avoid this requirement. It's like saying "if one could avoid machines ever going down, one wouldn't need an HA system". OK. What you said is true -- it's just not very helpful. Because, it's clear that the infrastructure must support this obvious and common need. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 00:24:28 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 00:24:20 +0200 Received: from perth.fpcc.net ([207.174.142.141]:29270 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 00:24:05 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id QAA19406; Tue, 5 Jun 2001 16:24:01 -0600 Message-ID: <3B1D5BCB.938DB852@unix.sh> Date: Tue, 05 Jun 2001 16:23:07 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3382.C86658B6@unix.sh> <006101c0ee05$49d7cae0$94627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > Please ignore the comment about positional parameters I just sent: on > re-reading your post, I see that it was the RPC argument sequence rather > than the XML representation that caused the issue. Of course, if all you're > going to use is single-argument RPCs (to avoid that issue), that further > calls into question whether the RPC framework is really buying you all that > much. RPC is inherent in some parts of the world in some places. Having rpc-with-replies and rpc-without-replies is a unifying concept. Unifying concepts are almost always good. RPC isn't such a big deal such that losing some or all of it would lose much in the first place ;-) You can use more than one argument if you want -- but you're asking for trouble in the case of upgrades. Having a single structure with arbitrary data below it is *at least* as powerful as having an ordered list of unnamed arguments each of can be of arbitrary data. Having positional parameters adds no power or simplicity to RPC. What it adds is familarity. Which is not such a big deal to trade off for upgradeability. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 00:55:53 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 00:55:36 +0200 Received: from mercury.mv.net ([199.125.85.40]:61959 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 00:55:19 +0200 Received: from filesrus (bnh-4-08.mv.com [199.125.99.200]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id SAA18121; Tue, 5 Jun 2001 18:55:11 -0400 (EDT) Message-ID: <00ca01c0ee12$f139e860$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <3B1D3382.C86658B6@unix.sh> <005a01c0ee04$71ab3760$94627dc7@filesrus> <3B1D5AAE.E1375580@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 18:57:47 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Alan Robertson" To: "Bill Todd" Cc: "linux-cluster" Sent: Tuesday, June 05, 2001 6:18 PM Subject: Re: A proposal for a General Clustering Framework > Bill Todd wrote: ... I'll also point out that it > > is often easier (and/or safer) to define upgrades (at least those that > > affect protocol syntax or semantics) such that all nodes get upgraded (one > > by one, with no general outage) to new software that continues to use the > > old protocols - and then after all upgrades are complete, a synchronous > > cluster-wide transition to the new protocols occurs. Among other things, > > this avoids proliferation of run-time conditionals that depend on the > > capabilities of each member. > > This is exactly what this technique is designed to support - allowing the > new code to use the old data formats. > > > In general, if one can avoid the requirement to support *protocol* version > > heterogeneity across a cluster, life can be a lot simpler. While *protocol* version > > heterogeneity is clearly required in general networks (where systems are > > under the control of completely independent organizations), it's not clear > > that (at least what I think of as) a cluster can't reasonably impose such a > > constraint. > > It's very clear that, in general, one cannot avoid this requirement. Please reassess your statement in the light of my clarification above: if you still believe it, either you don't understand what I was saying (since what I was saying has no adverse impact on availability) or you appear to believe that a 'cluster' must support the same degree of node protocol-version diversity that a WAN must (which I consider a marginal 'might be nice but is likely hell-on-wheels to support' rather than a 'clear requirement'). - bill It's > like saying "if one could avoid machines ever going down, one wouldn't need > an HA system". OK. What you said is true -- it's just not very helpful. > Because, it's clear that the infrastructure must support this obvious and > common need. > > > -- Alan Robertson > alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 00:59:56 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 00:59:47 +0200 Received: from gw.xkey.com ([206.86.100.52]:42253 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 00:59:31 +0200 Received: (from smtp@localhost) by happy.xkey.com id PAA10760 for ; Tue, 5 Jun 2001 15:59:25 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma009523; Tue Jun 5 15:38:11 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f55MeYb02821 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 18:40:34 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 18:40:34 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605184034.A2725@wumpus> Mail-Followup-To: linux-cluster References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1D5841.C099E0E@unix.sh>; from alanr@unix.sh on Tue, Jun 05, 2001 at 04:08:01PM -0600 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 04:08:01PM -0600, Alan Robertson wrote: > The third one has different > semantics, and shouldn't be made to look like the other two... [and it's > probably not that useful anyway...] No! Distributed delivery without order coherancy is fine for things like performance data, where you happen to not mind occasional missing data. And there can be a significant savings to not have to deliver things in order, or reliably. -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 01:19:35 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 01:19:26 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:65466 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 01:19:06 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f55NE1D17056; Tue, 5 Jun 2001 16:14:01 -0700 (PDT) Received: from oracle.com (dbrower-sun.us.oracle.com [130.35.180.64]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f55NJ3X29856; Tue, 5 Jun 2001 16:19:03 -0700 (PDT) Message-ID: <3B1D68E7.405634A7@oracle.com> Date: Tue, 05 Jun 2001 16:19:03 -0700 From: David Brower Organization: Oracle Corporation X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Darcy CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Jeff Darcy wrote: > > I think Mr. Darcy is making conclusions based on knowledge of some > > particular RPC systems. > > I really wish you wouldn't make such assumptions. You don't know me or my > background at all. You have *no* reason to make assumptions about what I do > or do not know. I'm trying not to make similar assumptions about you, > except where your own words indicate presence or absence of specific > knowledge. You'd do well to at least consider the possibility that those > who disagree with you do so out of hard-won knowledge rather than ignorance. I have built message based RPC systems that didn't require replies; they were RPC because they completely conformed to CORBA interface definitions and "oneway" semantics. They were perfectly usable for the sorts of things being discussed here. I even talked about "oneway" in one of the earliest messages on the topic. I will note that the kernel components of one commercial cluster system I'm working with use RPC interfaces to do membership elections, so there is proof by existance. Another membership system I see uses hand-rolled messaging. Given an unconstrained choice, I'd use a viable RPC system-- it would keep me from hand rolling messages and keep me focused on the more important algorithmic issues. The main downside to any particular RPC system is the learning curve necessary to make it jump through the hoops you want, the way you want; and with some you can't do anything but get it to play dead. YMMV. If I were -really- religious, I'd say to take one of the free ORBs and use that with a new transport underneath. But I think that is overkill, and something else will do. cheers, -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 02:46:00 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 02:45:51 +0200 Received: from web9207.mail.yahoo.com ([216.136.129.40]:7687 "HELO web9207.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 02:45:44 +0200 Message-ID: <20010606004542.61128.qmail@web9207.mail.yahoo.com> Received: from [202.135.239.240] by web9207.mail.yahoo.com; Tue, 05 Jun 2001 17:45:42 PDT Date: Tue, 5 Jun 2001 17:45:42 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: linux-cluster In-Reply-To: <3B1D3415.D3C33F5C@unix.sh> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- Alan Robertson wrote: > Bill Todd wrote: > > > > An aspect that I haven't yet seen covered (though I might have missed it) > is > > the efficient processing of large chunks of data - e.g., file/database > pages > > (which these days can easily reach 128 KB in size and in the future might > > reasonably get even larger - in many cases, any size where the seek > > time/rotational latency eclipses the transfer time can be 'reasonable') and > > even larger chunks of streaming data. > The short answer to this is that this protocol is for control messages, not > bulk data transfers. The needs of the two are somewhat different, in some > ways, radically different. My belief is that no single mechanism can meet > the needs of the two parts well. > > -- Alan Robertson > alanr@unix.sh > Our experience with the RSCT clustering on IBM MPP systems was that our control level protocols (David Brower's message a few back in this thread had all of the relevant references to this work!) were NOT useful to the data transfer layers. We support both failover (HACMP, RVSD) and cluster file systems (GPFS) with the RSCT clustering. Our 'clients' do their own bulk data transfer. Our protocols are oriented to multi-cast, small size, reliable, minimal overhead, etc., kinds of things we're talking about here. The bulk data was mostly point-to-point, high performance, streaming type data. We provided consistent membership, failover, events; they moved their data and used our notifications to do recovery. Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 02:50:43 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 02:50:35 +0200 Received: from c004-h006.c004.sfo.cp.net ([209.228.14.77]:12954 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 02:50:20 +0200 Received: (cpmta 4303 invoked from network); 5 Jun 2001 17:50:16 -0700 Received: from adsl-151-203-49-173.bostma.adsl.bellatlantic.net (HELO jdarcy6986nk) (151.203.49.173) by smtp.namezero.com (209.228.14.77) with SMTP; 5 Jun 2001 17:50:16 -0700 X-Sent: 6 Jun 2001 00:50:16 GMT Message-ID: <02e801c0ee22$0eb0a410$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "David Brower" , "Linux Cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <3B1D68E7.405634A7@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 20:45:42 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list > I have built message based RPC systems that didn't require replies; they > were RPC because they completely conformed to CORBA interface definitions > and "oneway" semantics. If it's one-way in the sense of not waiting for a reply, it's not RPC according to any meaningful definition of the term. I think Greg Lindahl hit it right on the head when he made a distinction between a loose definition of RPC and a definition that actually distinguishes it from other kinds of messaging. > I will note that > the kernel components of one commercial cluster system I'm working with > use RPC interfaces to do membership elections, so there is proof by > existance. Not so fast, cowboy. Is this the one-way RPC-that-is-not-RPC of which you spoke? What product is it? How many nodes does it support? When you say you "work with it" does that mean you're intimately involved with the protocol it *really* uses, or do you work with it in some more distant way? It's not that I distrust you personally, despite your manner. It's more that HA is full of bogus claims, "highly available" systems that aren't highly available, buzzword compliance, inflated claims, etc. I'm a little skeptical of convenient claims like that one. > Another membership system I see uses hand-rolled > messaging. Given an unconstrained choice, I'd use a viable RPC system-- > it would keep me from hand rolling messages Nobody's saying you should "hand-roll" messages. Well, OK, I'm not. ;-) While we're quoting earlier messages in the thread, you might notice that my very first response said that XML was a great idea *as a representation*. What I don't think works so well is tying implementors' hands with a "send a request to X, wait for a response from X" communications paradigm that's an ill fit for N-way coordination. > If I were -really- religious, I'd say to take one of the free ORBs > and use that with a new transport underneath. Are any of those free ORBs HA-capable, capable at least of failover if not of outright distributed operation? If not, they're simply not usable for many people. Yes, I know there are people out there interested in non-HA clustering. With all due respect, I consider that a pretty different problem. The world's very different when failure is the "normal" case, worth optimizing to minimize downtime, rather than a rare case in the shadow of optimizing for throughput. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 02:54:43 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 02:54:35 +0200 Received: from web9208.mail.yahoo.com ([216.136.129.41]:8467 "HELO web9208.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 02:54:26 +0200 Message-ID: <20010606005424.93142.qmail@web9208.mail.yahoo.com> Received: from [202.135.239.240] by web9208.mail.yahoo.com; Tue, 05 Jun 2001 17:54:24 PDT Date: Tue, 5 Jun 2001 17:54:24 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: Alan Robertson , Bill Todd Cc: David Brower , linux-cluster In-Reply-To: <3B1D360F.2A9262BA@unix.sh> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- Alan Robertson wrote: > Bill Todd wrote: > > > > > > Sending raw structures works only in homogeneous-endian environments anyway > > (though file systems do need the ability to send raw file data as > > byte-strings, since they haven't a clue what the internal structure of it > > may be and must leave interpretation up to the application). I can't keep up with all of the sub-threads here :) So I'll answer here, since I had a comment on this. I'd like to reiterate on this comment, and on others, that avoiding raw, low-level data format is key here. Been there and done that. While it allowed us to pack messages down to minimal sizes, it requires us to play fun games with future development and upward compatibility - i.e., as each node is migrated to a new level of code, the message tricks to keep everything straight get very complex. We do this on the RSCT clustering (we've had to coexist up to four - major - releases of code, so it's doable. But, ugly. This doesn't mean that you get this upward compatiblity for "free" with a more open (e.g., XML) representation, but you do get a much more visible process, and it should be easier. So, ok, I don't think this representation side is what's causing it to go off hereabouts... Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 02:57:16 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 02:57:08 +0200 Received: from c004-h015.c004.sfo.cp.net ([209.228.14.102]:48580 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 02:56:57 +0200 Received: (cpmta 14929 invoked from network); 5 Jun 2001 17:56:55 -0700 Received: from adsl-151-203-49-173.bostma.adsl.bellatlantic.net (HELO jdarcy6986nk) (151.203.49.173) by smtp.namezero.com (209.228.14.102) with SMTP; 5 Jun 2001 17:56:55 -0700 X-Sent: 6 Jun 2001 00:56:55 GMT Message-ID: <02e901c0ee22$fc0e0590$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "Greg Lindahl" , "linux-cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <20010605174343.A2650@wumpus> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 20:52:37 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list From: "Greg Lindahl" > I suspect most people here are using the non-RPC form of RPC, which > would explain why they went "Huh?" to your note. And that's why > there's more heat than light happening. You're probably right about that part. Some people seem to be using RPC - "Remote Procedure Call" - in reference to separable representation issues, or "Remote Coroutine" paradigms, or even to plain old vanilla messaging. > > Every protocol that one might use - > > e.g. for membership or consensus - has a very particular set of requirements > > regarding message reliability, ordering, etc. > > That's correct. And if you make these requirements explicit and > available to the lower layers, they can provide appropriate > services. I sort of agree, but also sort of not. Communications protocols are rarely so modular that you can turn every individual feature on and off with a switch. That being the case, sometimes there's a mismatch that is only highlighted but not solved by making the higher-level protocol's needs explicit. > Perhaps a better example is an application which does > not require ordered delivery using a protocol which only provides > ordered delivery. Perhaps that is a better example. It's certainly a common one, and sort of what I was getting at when I mentioned lower-level protocols forcing higher-level ones into less efficient modes or subsets. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:18:57 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:18:49 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:16382 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:18:36 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f561DRD11454; Tue, 5 Jun 2001 18:13:27 -0700 (PDT) Received: from oracle.com (dbrower-sun.us.oracle.com [130.35.180.64]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f561IUX00734; Tue, 5 Jun 2001 18:18:30 -0700 (PDT) Message-ID: <3B1D84E5.791753B7@oracle.com> Date: Tue, 05 Jun 2001 18:18:29 -0700 From: David Brower Organization: Oracle Corporation X-Mailer: Mozilla 4.7 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Jeff Darcy CC: Linux Cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <3B1D68E7.405634A7@oracle.com> <02e801c0ee22$0eb0a410$367b9fa8@lss.emc.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Jeff Darcy wrote: > > > I have built message based RPC systems that didn't require replies; they > > were RPC because they completely conformed to CORBA interface definitions > > and "oneway" semantics. > > If it's one-way in the sense of not waiting for a reply, it's not RPC > according to any meaningful definition of the term. Please tell the OMG that CORBA is not RPC, http://www.omg.org. > > I will note that > > the kernel components of one commercial cluster system I'm working with > > use RPC interfaces to do membership elections, so there is proof by > > existance. > > Not so fast, cowboy. Is this the one-way RPC-that-is-not-RPC of which you > spoke? What product is it? How many nodes does it support? When you say > you "work with it" does that mean you're intimately involved with the > protocol it *really* uses, or do you work with it in some more distant way? The wording means "if I felt I could say, I would." > It's not that I distrust you personally, despite your manner. ditto, mr. "work I did that IBM claimed credit for". > Nobody's saying you should "hand-roll" messages. Well, OK, I'm not. ;-) > While we're quoting earlier messages in the thread, you might notice that my > very first response said that XML was a great idea *as a representation*. > What I don't think works so well is tying implementors' hands with a "send a > request to X, wait for a response from X" communications paradigm that's an > ill fit for N-way coordination. Good, we agree on something. > > If I were -really- religious, I'd say to take one of the free ORBs > > and use that with a new transport underneath. > > Are any of those free ORBs HA-capable, capable at least of failover if not > of outright distributed operation? If not, they're simply not usable for > many people. Do you -need- HA on the point-to-point messaging internal to the infrastructure? If it doesn't work, you may have a partition and a need to reconfigure. Or maybe you can explicitly try another path (object ref in corba terms). It's hard to have HA messaging if you don't get a response back, so no "oneway" datagram messaging scheme is HA. It -is- a perfectly good debate topic whether the support of multiple paths between nodes should be exposed or hidden from the layer doing the service message exchange. I think sometimes you'll want to take any path, and sometimes you want to require -this- path. > Yes, I know there are people out there interested in non-HA clustering. > With all due respect, I consider that a pretty different problem. The > world's very different when failure is the "normal" case, worth optimizing > to minimize downtime, rather than a rare case in the shadow of optimizing > for throughput. Right. "Only the failure cases are interesting." I like splitting hairs at least as much as the next pedant, but do we really need a holy war about the definition of "RPC"? "RPC" is like "Cluster" -- it means what the speaker wants it to mean, and no one can honestly say he was wrong. Hell, I choke thinking of XML-RPC or SOAP as RPC. Where's the IDL? Where's the language binding specifications? They're just message formatting standards, and the "O" in SOAP is a lie. -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:26:55 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:26:45 +0200 Received: from gw.xkey.com ([206.86.100.52]:30727 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:26:28 +0200 Received: (from smtp@localhost) by happy.xkey.com id SAA29103 for ; Tue, 5 Jun 2001 18:26:26 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma029098; Tue Jun 5 18:26:17 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f561SdW03142 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 21:28:39 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 21:28:39 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605212839.A3138@wumpus> Mail-Followup-To: linux-cluster References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <20010605174343.A2650@wumpus> <02e901c0ee22$fc0e0590$367b9fa8@lss.emc.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <02e901c0ee22$fc0e0590$367b9fa8@lss.emc.com>; from linuxguy@tambreet.com on Tue, Jun 05, 2001 at 08:52:37PM -0400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list > > That's correct. And if you make these requirements explicit and > > available to the lower layers, they can provide appropriate > > services. > > I sort of agree, but also sort of not. Communications protocols are rarely > so modular that you can turn every individual feature on and off with a > switch. That being the case, sometimes there's a mismatch that is only > highlighted but not solved by making the higher-level protocol's needs > explicit. I never said that every set of lower layers will provide exactly what every set of higher layers will want. I was talking about the right interface that can let the right thing happen if people care. The point of the current discussion is to define that interface. I think the other part, data representation, pretty much got agreement from everyone. -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:35:39 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:35:31 +0200 Received: from perth.fpcc.net ([207.174.142.141]:17498 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:35:18 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA20491; Tue, 5 Jun 2001 19:35:14 -0600 Message-ID: <3B1D889E.A827CBD1@unix.sh> Date: Tue, 05 Jun 2001 19:34:22 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Greg Lindahl CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> <20010605184034.A2725@wumpus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Greg Lindahl wrote: > > On Tue, Jun 05, 2001 at 04:08:01PM -0600, Alan Robertson wrote: > > > The third one has different > > semantics, and shouldn't be made to look like the other two... [and it's > > probably not that useful anyway...] > > No! Distributed delivery without order coherancy is fine for things > like performance data, where you happen to not mind occasional missing > data. And there can be a significant savings to not have to deliver > things in order, or reliably. Actually, delivering things reliably is practically free with heartbeat (it uses NAKs, not ACKs) see http://linux-ha.org/comm/ . Doing it with ordering is harder, and reasonably expensive. Performance data seems different from event data (which is what was under discussion). Performance data is another whole can of worms. Another day for that ... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:37:31 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:37:23 +0200 Received: from perth.fpcc.net ([207.174.142.141]:19802 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:37:10 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA20508; Tue, 5 Jun 2001 19:37:04 -0600 Message-ID: <3B1D890B.C2734849@unix.sh> Date: Tue, 05 Jun 2001 19:36:11 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Peter Badovinatz CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <20010606004542.61128.qmail@web9207.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Peter Badovinatz wrote: > > --- Alan Robertson wrote: > > Bill Todd wrote: > > > > > > An aspect that I haven't yet seen covered (though I might have missed it) > > is > > > the efficient processing of large chunks of data - e.g., file/database > > pages > > > (which these days can easily reach 128 KB in size and in the future might > > > reasonably get even larger - in many cases, any size where the seek > > > time/rotational latency eclipses the transfer time can be 'reasonable') and > > > even larger chunks of streaming data. > > > The short answer to this is that this protocol is for control messages, not > > bulk data transfers. The needs of the two are somewhat different, in some > > ways, radically different. My belief is that no single mechanism can meet > > the needs of the two parts well. > > > > -- Alan Robertson > > alanr@unix.sh > > > Our experience with the RSCT clustering on IBM MPP systems was that our control > level protocols (David Brower's message a few back in this thread had all of > the relevant references to this work!) were NOT useful to the data transfer > layers. We support both failover (HACMP, RVSD) and cluster file systems (GPFS) > with the RSCT clustering. Our 'clients' do their own bulk data transfer. > > Our protocols are oriented to multi-cast, small size, reliable, minimal > overhead, etc., kinds of things we're talking about here. The bulk data was > mostly point-to-point, high performance, streaming type data. We provided > consistent membership, failover, events; they moved their data and used our > notifications to do recovery. Well said. This is also the approach used by heartbeat, and the approach I'd recommend here also... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:45:46 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:45:36 +0200 Received: from web9208.mail.yahoo.com ([216.136.129.41]:57613 "HELO web9208.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 03:45:29 +0200 Message-ID: <20010606014528.1065.qmail@web9208.mail.yahoo.com> Received: from [202.135.239.240] by web9208.mail.yahoo.com; Tue, 05 Jun 2001 18:45:28 PDT Date: Tue, 5 Jun 2001 18:45:28 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: linux-cluster In-Reply-To: <3B1D3F51.537D23@unix.sh> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- Alan Robertson wrote: > Greg Lindahl wrote: > > > > On Tue, Jun 05, 2001 at 01:56:37PM -0600, Alan Robertson wrote: > > > > This is good. In particular, you could imagine a simple pub/sub system > > that would broadcast all published data, and if the local system > > doesn't have a subscription, it discards it. Or you could have a > > central server that distributes data only to the actual > > subscribers. Can you write an API flexible enough to allow both > > implementations? > > The reason why I didn't worry too much about the event system was because > it's pretty easy to write one as an application in the framework. You could > even make it where all events are seen by all systems in the same order... I'd rather say it's straightforward to write, "easy" is not the right word. In fact, full ordering of all global events is definitely non-trivial, and gets us into terrain other than the general events system I'm thinking about here. Ordering of local events is more reasonable, but in a fully general event system "order" is an interesting question. > > In fact, you could even write one pretty easily on the heartbeat API. If > you want events received in the same order everywhere, you might have to do > a little more work, but it's not really that bad... > > Basically, you have an event daemon on each machine. Each of these daemons > has knowledge of who their local clients are. Going beyond the 'basic' cluster events (node up/down, IP address up/down), is where the event system becomes interesting. First assumption is a basic plug-in style for producers and consumers. The consumers are as described here, they connect to a local "event" daemon. They set up subscriptions with that daemon for events of interest, the key is that these are both local and cluster global events. The daemon passes the events on. Producers monitor the cluster (for some things the event daemon itself may be the producer). These are plug-ins customised to the appropriate cluster entity for which they monitor. They can be quiescent until a consumer decides to request events for their target, the event daemon then 'tickles' the producer to start monitoring. It will then pass on information to the event daemon. > > Each daemon is responsible for broadcasting events out, and notifying their > clients of new events that they have subscribed for. > Key question: filtering (for want of a better word.) What I'm used to is that the event daemons do the filtering: - the subscriber passes in parameters that specify the conditions about which they care, e.g., 'tell me if paging space used on any node is > 75%'. In this case, an 'event' only occurs when page space utilisation exceeds this, prior to its happening, a consumer sees NOTHING. - the local event daemon distributes the request to its peers on the cluster (or not, this can all work purely in local mode). - each event daemon 'wakes up' the appropriate monitor for paging space. At this point, it gets more interesting. Does the paging space monitor: - report EVERY change in page space utilisation? - sample every X seconds? - report every sample? - report only when threshold exceeded? In the IBM event manager model the producers are 'dumb'. They do NOT know about the event thresholds. They report on some fixed basis, the local event daemon decides whether to pass anything on. Back to order. Data is generated by independent producers, so, even on the local node the event daemon can only generate an order that may, or may not, match the true order of the events. > It's just another cluster-aware app, which you can start up when/if someone > requests it. Yes, pretty well describes it. But by the time you define the APIs and the models and all that, it gets rather complex to make it fully general to allow plug-ins for anything you care about, to handle local and global events, to try and keep the message flow minimal and efficient, and to determine where to put the function. > > -- Alan Robertson > alanr@unix.sh Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:46:03 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:45:44 +0200 Received: from perth.fpcc.net ([207.174.142.141]:22362 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:45:34 +0200 Received: from unix.sh (03-158.026.popsite.net [64.24.105.158]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id TAA20535; Tue, 5 Jun 2001 19:43:28 -0600 Message-ID: <3B1D8A8C.78859C64@unix.sh> Date: Tue, 05 Jun 2001 19:42:36 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Brower CC: Jeff Darcy , Linux Cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <3B1D68E7.405634A7@oracle.com> <02e801c0ee22$0eb0a410$367b9fa8@lss.emc.com> <3B1D84E5.791753B7@oracle.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list David Brower wrote: > > > Hell, I choke thinking of XML-RPC or SOAP as RPC. Where's the IDL? > Where's the language binding specifications? They're just message > formatting standards, and the "O" in SOAP is a lie. That's probably why I find XML-RPC to be attractive ;-) It actually seems like it might be a very good match for what we need here. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:46:45 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:46:35 +0200 Received: from c004-h005.c004.sfo.cp.net ([209.228.14.76]:36025 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 03:46:21 +0200 Received: (cpmta 10345 invoked from network); 5 Jun 2001 18:46:14 -0700 Received: from adsl-151-203-49-173.bostma.adsl.bellatlantic.net (HELO jdarcy6986nk) (151.203.49.173) by smtp.namezero.com (209.228.14.76) with SMTP; 5 Jun 2001 18:46:14 -0700 X-Sent: 6 Jun 2001 01:46:14 GMT Message-ID: <030b01c0ee29$e0072b90$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "David Brower" Cc: "Linux Cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <3B1D68E7.405634A7@oracle.com> <02e801c0ee22$0eb0a410$367b9fa8@lss.emc.com> <3B1D84E5.791753B7@oracle.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 21:41:57 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list > ditto, mr. "work I did that IBM claimed credit for". Unlike your claim, mine can be verified. Ask Peter Badovinatz; he inherited some of the code I wrote. Ask Steve Todd, or Alan Robertson; they're both familiar with my work too. I was the chief architect and implementor of the cluster manager for HACMP 3.1 (pre-Phoenix) and one of three for the corresponding lock manager (yes, the one open-sourced by IBM). At the time, HACMP was the only open-systems clustering software to support eight nodes, and even now I have yet to hear about a lock manager that provides better four-node OPS scalability using an off-the-shelf interconnect. Much of my work from that period has survived to this day, which is pretty good for the industry in general and particularly in light of the continual political wars within IBM. The HACMP concepts and terminology that I helped develop have been adopted by many other HA projects, including one Linux HA project that was a transparent ripoff of the 3.1 design by an IBM field engineer. In short, I'd say my credentials in this area are extremely solid and quite verifiable...unlike yours, so you can keep your snide ad-hominem BS to yourself. Next time find out who the hell you're talking to before you assume you know more than they do. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:52:05 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:51:55 +0200 Received: from c004-h007.c004.sfo.cp.net ([209.228.14.63]:54189 "HELO c004.sfo.cp.net") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 03:51:43 +0200 Received: (cpmta 20651 invoked from network); 5 Jun 2001 18:51:40 -0700 Received: from adsl-151-203-49-173.bostma.adsl.bellatlantic.net (HELO jdarcy6986nk) (151.203.49.173) by smtp.namezero.com (209.228.14.63) with SMTP; 5 Jun 2001 18:51:40 -0700 X-Sent: 6 Jun 2001 01:51:40 GMT Message-ID: <031301c0ee2a$a23c93d0$367b9fa8@lss.emc.com> From: "Jeff Darcy" To: "Jeff Darcy" , "David Brower" Cc: "Linux Cluster" References: <3B1CFF44.64F9DFA7@unix.sh> <3B1D0FA8.D56462C0@oracle.com> <021401c0ede7$23b7a4c0$367b9fa8@lss.emc.com> <3B1D3E09.3A3976EA@oracle.com> <02ad01c0ee05$6cba4dd0$367b9fa8@lss.emc.com> <3B1D68E7.405634A7@oracle.com> <02e801c0ee22$0eb0a410$367b9fa8@lss.emc.com> <3B1D84E5.791753B7@oracle.com> <030b01c0ee29$e0072b90$367b9fa8@lss.emc.com> Subject: Re: A proposal for a General Clustering Framework Date: Tue, 5 Jun 2001 21:47:23 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4133.2400 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list My apologies for sending that to the list. I meant to send it privately. Obviously, someone's 'tude got under my skin a little. Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:53:25 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:53:16 +0200 Received: from gw.xkey.com ([206.86.100.52]:7433 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 03:53:07 +0200 Received: (from smtp@localhost) by happy.xkey.com id SAA32357 for ; Tue, 5 Jun 2001 18:53:05 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma032353; Tue Jun 5 18:52:55 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f561tHj03216 for linux-cluster@nl.linux.org; Tue, 5 Jun 2001 21:55:17 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Tue, 5 Jun 2001 21:55:17 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010605215517.A3213@wumpus> Mail-Followup-To: linux-cluster References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> <20010605184034.A2725@wumpus> <3B1D889E.A827CBD1@unix.sh> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1D889E.A827CBD1@unix.sh>; from alanr@unix.sh on Tue, Jun 05, 2001 at 07:34:22PM -0600 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 07:34:22PM -0600, Alan Robertson wrote: > Performance data seems different from event data (which is what was under > discussion). It may seem different to you, but if you provide pub/sub services, it fits it quite nicely. Perhaps you don't know what kind of data I'm talking about. Say you're doing load balancing and need to make distributed decisions about which member to talk to. One easy and good way to do that is to distribute an idea of what the "load" of various machines is. It is control information, of a sort. -- g Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 03:58:39 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 03:58:21 +0200 Received: from web9204.mail.yahoo.com ([216.136.129.27]:60165 "HELO web9204.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 03:58:08 +0200 Message-ID: <20010606015801.18290.qmail@web9204.mail.yahoo.com> Received: from [202.135.239.240] by web9204.mail.yahoo.com; Tue, 05 Jun 2001 18:58:01 PDT Date: Tue, 5 Jun 2001 18:58:01 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: linux-cluster In-Reply-To: <005a01c0ee04$71ab3760$94627dc7@filesrus> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- Bill Todd wrote: > While I agree with the observation expressed elsewhere that name/value pairs > would seem to address the problem you describe, I'll also point out that it > is often easier (and/or safer) to define upgrades (at least those that > affect protocol syntax or semantics) such that all nodes get upgraded (one > by one, with no general outage) to new software that continues to use the > old protocols - and then after all upgrades are complete, a synchronous > cluster-wide transition to the new protocols occurs. Among other things, > this avoids proliferation of run-time conditionals that depend on the > capabilities of each member. I've done this. Our normal recommendation for truly HA systems is to upgrade them as "fast as possible" although this does mean one node at a time, to get through this period in as timely a fashion as you can. We also stated that you should not perform any configuration changes during this window (i.e., define new nodes, modify resource definitions, etc.) Not that we strictly disallowed this, just try to discourage it. The code would switch over to new protocols once all nodes were upgraded. You still had to maintain the coexistence code though until you'd gotten rid of all levels which could coexist. > > In general, if one can avoid the requirement to support version > heterogeneity across a cluster, life can be a lot simpler. While version > heterogeneity is clearly required in general networks (where systems are > under the control of completely independent organizations), it's not clear > that (at least what I think of as) a cluster can't reasonably impose such a > constraint. > > - bill > Depends... We could push customers who really cared about strict HA to avoid version heterogeneity except during actual node-by-node upgrade. It was customers who were more oriented to cluster file system - where we still needed (a bit looser form of) HA and failover but they could be up to 500 node clusters - that we had no choice but to be very flexible as to supporting multiple versions across nodes because upgrading 500 nodes a few at a time takes a long time, and unlike more controlled HA clusters the workload was quite varied so many more things to upgrade/test/migrate. The better choice is to be sure the framework lets you handle multiple active version levels from the beginning, and you work to minimise their use by decree (as above) or by only supporting some limited number of levels (N and N+1 or N+2.) Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 04:13:02 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 04:12:54 +0200 Received: from web9208.mail.yahoo.com ([216.136.129.41]:50955 "HELO web9208.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 04:12:42 +0200 Message-ID: <20010606021240.5241.qmail@web9208.mail.yahoo.com> Received: from [202.135.239.240] by web9208.mail.yahoo.com; Tue, 05 Jun 2001 19:12:40 PDT Date: Tue, 5 Jun 2001 19:12:40 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: linux-cluster In-Reply-To: <3B1D5841.C099E0E@unix.sh> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- Alan Robertson wrote: > Alan Robertson wrote: > Sorry. > > You could build either type of implementation on top of the heartbeat API > easily. > > Actually there are three kinds that come to mind: > > A master-node concept, where all events are serialized through this one > machine. > > A distributed version with event-order coherency, where all events are seen > in the same order by all machines. These first two often break down to the same underlying implementation, or the latter is a distributed agreement protocol of some sort. Trade-offs here for scaling, messages, etc. I've built these kinds of systems. > > A distributed version without event-order coherency. This IS useful. It is just useful for different things than the first two. It is very scalable, very flexible, and so long as it is separate from the ordered mechanisms so the consumer can tell the message streams apart, you do want them all present. > > I suppose you could install all types simultaneously, but it sounds like it > wouldn't be very useful, and would confuse the apps. > > If you design the API properly, you should be able to hide the differences > between the first two from the application. The third one has different > semantics, and shouldn't be made to look like the other two... [and it's > probably not that useful anyway...] The first two should be invisible to the consumers, they get events, it is not important to them how the ordering is controlled, so long as it works. The latter needs to be clearly separated such that consumers can get events about which they care, but which may not indicate "bad" things, and they know that this channel is unordered. You want to be able to pass through large numbers of events (10,000s) through this channel to support monitoring as well as recovery. To try and order all of these is a scaling nightmare. > > > -- Alan Robertson > alanr@unix.sh Peter ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:04:31 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:04:23 +0200 Received: from web9207.mail.yahoo.com ([216.136.129.40]:49419 "HELO web9207.mail.yahoo.com") by humbolt.nl.linux.org with SMTP id ; Wed, 6 Jun 2001 06:04:15 +0200 Message-ID: <20010606040407.80487.qmail@web9207.mail.yahoo.com> Received: from [202.135.239.36] by web9207.mail.yahoo.com; Tue, 05 Jun 2001 21:04:07 PDT Date: Tue, 5 Jun 2001 21:04:07 -0700 (PDT) From: Peter Badovinatz Subject: Re: A proposal for a General Clustering Framework To: linux-cluster In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list --- irbis@orcero.org wrote: > > IMHO the proponsal of the paper is trying to solve things of the 6th OSI > level on the kernel, moving things that are now being resolved by the > applications in the kernel. > > For me, it sounds strange with OSI glases to put the session layer on the > kernel clustering framework -this is the overview proponsal main idea-. > > I also can not find how to but all this XML stuff on HP clusters. Maybe > is my lack of understanding, but all the problems that is trying to solve > the proponsal were already solved by the application libs that are being > used. > > Cluster transparency and performance -the two main interests of HP > clustering- are not improved with this proponsal. And performace can be > penaltied. By transparency do you mean not having to program to an MPP model? Or are you just concerned with administration or job scheduling? In any case, if you look at the IBM SP running PSSP+RSCT, you have an HP system (PSSP) with an included HA framework (RSCT, aka Phoenix). We had to do a number of things with RSCT to deal with differing environments: - tight HA, strict failover, shared data, etc. - HP, with some HA needed due to clustered file system and remote data access The latter assumed that we did not perform failover and recovery within 10s of seconds, rather in terms of a minute or two. This was for two reasons: - avoid false failure indications due to longer message delays with large numbers of nodes. - absolutely minimise performance impacts to allow the programs (usually distributed MPI jobs) to not be interfered with. Note that to get this right did require a fair amount of skill by the system admins and users to work out job characteristics. RSCT provided a wide variety of tuning handles, not all of them sufficiently explained. Benefits gained with RSCT across an HP cluster were consistent notification of failures, scalable (500 node) cluster file system, event monitoring (events in terms of what's been discussed in this thread.) But yes, avoiding ANY performance hit was impossible. The key was to achieve the level of monitoring and consistency you desired while minimising impact on the system, a labour-intensive task unfortunately. This does not say that the framework being proposed/discussed would be immediately extensible to HP clusters, but it is not out of the realm of possibility to consider it happening eventually. Peter > > Yours: > > David > > --------------------- > http://www.orcero.org > irbis@orcero.org > --------------------- > ===== These have been the opinions of: Peter R. Badovinatz -- (503)578-5530 (TL 775) wombat@us.ibm.com/tabmowzo@yahoo.com and in no way should be construed as official opinion of IBM, Corp. __________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail - only $35 a year! http://personal.mail.yahoo.com/ Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:11:23 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:11:04 +0200 Received: from perth.fpcc.net ([207.174.142.141]:51293 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:10:55 +0200 Received: from unix.sh (04-196.026.popsite.net [64.24.105.196]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA21401; Tue, 5 Jun 2001 22:10:51 -0600 Message-ID: <3B1DAD17.4319AA92@unix.sh> Date: Tue, 05 Jun 2001 22:09:59 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Greg Lindahl CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> <20010605184034.A2725@wumpus> <3B1D889E.A827CBD1@unix.sh> <20010605215517.A3213@wumpus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Greg Lindahl wrote: > > On Tue, Jun 05, 2001 at 07:34:22PM -0600, Alan Robertson wrote: > > > Performance data seems different from event data (which is what was under > > discussion). > > It may seem different to you, but if you provide pub/sub services, it > fits it quite nicely. Perhaps you don't know what kind of data I'm > talking about. Say you're doing load balancing and need to make > distributed decisions about which member to talk to. One easy and good > way to do that is to distribute an idea of what the "load" of various > machines is. It is control information, of a sort. This ticked off a minor flame war the last time I mentioned it, but heartbeat distributes basic load information with every packet. It's only a few bytes, and compared to the overhead in the RPC-XML format, a very small amount of data indeed ;-) The right thing to figure out is how to create an API / design which allows for this kind of data distribution, and yet also allows for other methods of it being distributed. See the paper (or better yet module.c) for my definitions of "module" and "plugin"... Perhaps the heartbeat module can register both as a transport plugin and as a performance data plugin... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:15:15 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:14:56 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:62909 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:14:45 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f5649dD02116 for ; Tue, 5 Jun 2001 21:09:39 -0700 (PDT) Received: from oracle.com ([152.68.53.78]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f564EfX26350 for ; Tue, 5 Jun 2001 21:14:41 -0700 (PDT) Message-ID: <3B1DAE2C.9659A8FB@oracle.com> Date: Tue, 05 Jun 2001 21:14:36 -0700 From: David Brower Organization: Oracle X-Mailer: Mozilla 4.7 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: linux-cluster Subject: An apology offerred References: <20010606021240.5241.qmail@web9208.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list to those who got caught up by an escalating series of snipes, I'm sorry for my part. When the linux stuff isn't being fun, it's time to chill. cheers, -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:16:48 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:16:39 +0200 Received: from perth.fpcc.net ([207.174.142.141]:5214 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:16:32 +0200 Received: from unix.sh (04-196.026.popsite.net [64.24.105.196]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA21427; Tue, 5 Jun 2001 22:15:27 -0600 Message-ID: <3B1DAE2B.43771C47@unix.sh> Date: Tue, 05 Jun 2001 22:14:35 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Peter Badovinatz CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <20010606015801.18290.qmail@web9204.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Peter Badovinatz wrote: > [snip] > Depends... We could push customers who really cared about strict HA to avoid > version heterogeneity except during actual node-by-node upgrade. It was > customers who were more oriented to cluster file system - where we still needed > (a bit looser form of) HA and failover but they could be up to 500 node > clusters - that we had no choice but to be very flexible as to supporting > multiple versions across nodes because upgrading 500 nodes a few at a time > takes a long time, and unlike more controlled HA clusters the workload was > quite varied so many more things to upgrade/test/migrate. > > The better choice is to be sure the framework lets you handle multiple active > version levels from the beginning, and you work to minimise their use by decree > (as above) or by only supporting some limited number of levels (N and N+1 or > N+2.) The conclusions I would draw from this are: The real world is ugly. Technology had better be equipped to deal with it. Sensible policies help you keep out of trouble. Policies are not guarantees. I view these as all perfectly consistent with what I said before. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:28:09 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:28:00 +0200 Received: from perth.fpcc.net ([207.174.142.141]:7262 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:27:50 +0200 Received: from unix.sh (04-196.026.popsite.net [64.24.105.196]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA21475; Tue, 5 Jun 2001 22:27:46 -0600 Message-ID: <3B1DB10C.4FB60712@unix.sh> Date: Tue, 05 Jun 2001 22:26:52 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Peter Badovinatz CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <20010606040407.80487.qmail@web9207.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Peter Badovinatz wrote: > [snip] > This does not say that the framework being proposed/discussed would be > immediately extensible to HP clusters, but it is not out of the realm of > possibility to consider it happening eventually. It should be considered a design critera of the *framework*, but not necessarily of any particular module/plugin in the framework. The framework itself is mainly a collection of infrastructure pieces and APIs. One can have HPC implementations of plugins and HA implementations of the plugins. Some plugins may work equally well for both HA and HPC clusters. The interesting questions for the framework come down to these: Can a reasonable HA implementation of API "x" be created? Can a reasonable HPC implementation of API "x" be created? Does the framework cover every interesting HPC API? Does the framework cover every interesting HA API? If the answer to these is "yes" for every interesting API, then the framework is a success. This is not to say that by day "X" that you will have everything you need in the framework. But, if you participate in its creation, and write some components, then you stand a much better chance of getting it by the date you need it ;-) -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:34:23 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:34:03 +0200 Received: from inet-mail4.oracle.com ([148.87.2.204]:26564 "EHLO inet-mail4.oracle.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:34:01 +0200 Received: from gmgw01.us.oracle.com (gmgw01.us.oracle.com [130.35.249.115]) by inet-mail4.oracle.com (Switch-2.1.3/Switch-2.1.0) with ESMTP id f564SuD07803; Tue, 5 Jun 2001 21:28:56 -0700 (PDT) Received: from oracle.com ([152.68.53.78]) by gmgw01.us.oracle.com (Switch-2.1.1/Switch-2.1.0) with ESMTP id f564XwX10969; Tue, 5 Jun 2001 21:33:58 -0700 (PDT) Message-ID: <3B1DB2B0.E79F0D2A@oracle.com> Date: Tue, 05 Jun 2001 21:33:52 -0700 From: David Brower Organization: Oracle X-Mailer: Mozilla 4.7 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Alan Robertson CC: Greg Lindahl , linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> <20010605184034.A2725@wumpus> <3B1D889E.A827CBD1@unix.sh> <20010605215517.A3213@wumpus> <3B1DAD17.4319AA92@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > The right thing to figure out is how to create an API / design which allows > for this kind of data distribution, and yet also allows for other methods of > it being distributed. > event systems shine for this. -- see the ibm stuff http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/sp/ssp/admin/spa1mst37.html or the compaq stuff; all doc at http://tru64unix.compaq.com/faqs/publications/base_doc/DOCUMENTATION/V51_HTML/MAN/MAN5/0013____.HTM all compaq doc at: http://tru64unix.compaq.com/faqs/publications/pub_page/pubs_page.html -dB and events at: -dB Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 06:48:55 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 06:48:46 +0200 Received: from perth.fpcc.net ([207.174.142.141]:14430 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 06:48:35 +0200 Received: from unix.sh (04-196.026.popsite.net [64.24.105.196]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA21580; Tue, 5 Jun 2001 22:48:31 -0600 Message-ID: <3B1DB5E9.64513E7C@unix.sh> Date: Tue, 05 Jun 2001 22:47:37 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Brower , linux-cluster Subject: Re: An apology offerred References: <20010606021240.5241.qmail@web9208.mail.yahoo.com> <3B1DAE2C.9659A8FB@oracle.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list David Brower wrote: > > to those who got caught up by an escalating > series of snipes, I'm sorry for my part. > When the linux stuff isn't being fun, it's > time to chill. By the way, I thought about how a wise friend of mine once sent me an email when I was getting bent out of shape with one Jerome Etienne. The subject line was "Now, Now" and the body said "I sense you're getting testy". It was the perfect thing to say. I thought about quoting it in the email I sent to both of you, but only you would have gotten it ;-) I probably would have sent out something a little sooner, but I was busy preparing an extended abstract for ALS in Oakland. Today's the deadline ;-) -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 07:45:17 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 07:44:59 +0200 Received: from mercury.mv.net ([199.125.85.40]:62726 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 07:44:51 +0200 Received: from filesrus (bnh-6-08.mv.com [199.125.98.72]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id BAA19971; Wed, 6 Jun 2001 01:44:45 -0400 (EDT) Message-ID: <02bd01c0ee4c$276c2d60$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <20010606015801.18290.qmail@web9204.mail.yahoo.com> <3B1DAE2B.43771C47@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Wed, 6 Jun 2001 01:47:19 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Alan Robertson" To: "Peter Badovinatz" Cc: "linux-cluster" Sent: Wednesday, June 06, 2001 12:14 AM Subject: Re: A proposal for a General Clustering Framework > Peter Badovinatz wrote: > > > > [snip] > > > Depends... We could push customers who really cared about strict HA to avoid > > version heterogeneity except during actual node-by-node upgrade. It was > > customers who were more oriented to cluster file system - where we still needed > > (a bit looser form of) HA and failover but they could be up to 500 node > > clusters - that we had no choice but to be very flexible as to supporting > > multiple versions across nodes because upgrading 500 nodes a few at a time > > takes a long time, and unlike more controlled HA clusters the workload was > > quite varied so many more things to upgrade/test/migrate. > > > > The better choice is to be sure the framework lets you handle multiple active > > version levels from the beginning, and you work to minimise their use by decree > > (as above) or by only supporting some limited number of levels (N and N+1 or > > N+2.) Since Alan hasn't responded to my request for a reassessment (I understand that he's been busy today), and since the comments above may reflect a similar lack of understanding, I'll reiterate: Supporting at least version N+1 of the *software* along with version N is clearly required to achieve on-line rolling upgrades. What is not required is to support (concurrently) multiple versions of the communication protocols: you can instead support only the old protocol set until all nodes have been upgraded to version N+1, and then perform a cluster-wide synchronous cut-over to the new protocol version. If the cluster can be assumed to be under the control of a single organization (in contrast to the nodes on the Internet, for example), then requiring that all nodes be upgraded before the communication protocols change is not prima facie unreasonable: indeed, it is difficult to create examples where a protocol-upgrade is urgently needed but only by some subset of the cluster's members, since a cluster is a considerably more tightly-bound entity than the nodes on the Internet. - bill > > The conclusions I would draw from this are: > > The real world is ugly. > > Technology had better be equipped to deal with it. > > Sensible policies help you keep out of trouble. > > Policies are not guarantees. > > I view these as all perfectly consistent with what I said before. > > -- Alan Robertson > alanr@unix.sh > > Linux-cluster: generic cluster infrastructure for Linux > Archive: http://mail.nl.linux.org/linux-cluster/ > Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 07:47:57 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 07:47:50 +0200 Received: from mercury.mv.net ([199.125.85.40]:17159 "EHLO mercury.mv.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 07:47:28 +0200 Received: from filesrus (bnh-6-08.mv.com [199.125.98.72]) by mercury.mv.net (8.8.8/mem-971025) with SMTP id BAA20324; Wed, 6 Jun 2001 01:47:23 -0400 (EDT) Message-ID: <02be01c0ee4c$85b60e40$94627dc7@filesrus> From: "Bill Todd" To: "Alan Robertson" Cc: "linux-cluster" References: <20010606040407.80487.qmail@web9207.mail.yahoo.com> <3B1DB10C.4FB60712@unix.sh> Subject: Re: A proposal for a General Clustering Framework Date: Wed, 6 Jun 2001 01:49:58 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.50.4522.1200 X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4522.1200 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list ----- Original Message ----- From: "Alan Robertson" To: "Peter Badovinatz" Cc: "linux-cluster" Sent: Wednesday, June 06, 2001 12:26 AM Subject: Re: A proposal for a General Clustering Framework > Peter Badovinatz wrote: > > > > [snip] > > > This does not say that the framework being proposed/discussed would be > > immediately extensible to HP clusters, but it is not out of the realm of > > possibility to consider it happening eventually. > > It should be considered a design critera of the *framework*, but not > necessarily of any particular module/plugin in the framework. > > The framework itself is mainly a collection of infrastructure pieces and > APIs. I'll make one more plug for having *underlying* mechanisms that can be used in a more general manner (and then drop the subject). E.g., something that allows a heartbeat message to be handled completely at interrupt level rather than consume more processing resources by being dispatched for later processing (which might be especially important if heartbeats are broadcast rather than distributed via some kind of ring or hierarchy). Defining such general-purpose underpinnings benefits everyone - and you'll need *something* for the specific purposes you're discussing here anyway. Making it general-purpose does require more design, but if your goal is the creation of generally-useful core software for clusters then a general-purpose inter-node communication mechanism ought to be fairly high on the list anyway. The important aspects of such a low-level mechanism include performance flexibility (see previous examples I've given), representational agnosticism (e.g., the decision to use something like XML belongs to higher layers: down here, messages or message segments are just byte streams), a namespace independent of lower-level identifiers such as IP addresses and process IDs (e.g., so that messages can be addressed to 'Facility Foo on Node Fred', or perhaps just to 'Facility Foo', wherever it is; any deeper addressing can be addressed by the registered Facility Foo handler at the destination if implementing a full address-hierarchy seems excessive), and support for features such as encryption and source-node-authentication for interconnect environments that require such things. Fast Messages offer one paradigm that seems to fit the bill (with the caveat that I'm familiar with them only from casual Web-browsing), and there are likely others. At a slightly higher level come things like event-notification (membership changes being an obvious biggie) to facilities - but the internal communication mechanisms they use aren't of that much interest to the applications they serve (as long as they work well). I'm no comm expert (so the above may be naive in some respects), but I have a pretty good idea of what I need for a distributed file system and cache and my suspicion is that at least some of what I need would be useful to other applications as well. However, if you're just going to use existing facilities such as UDP over kernel sockets then there may indeed not be all that much code that would be common (in which case an efficient, high-performance, low-level message facility might well be a good separate project - and something you might eventually find useful yourselves). - bill Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 08:31:13 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 08:31:06 +0200 Received: from perth.fpcc.net ([207.174.142.141]:19808 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 08:30:49 +0200 Received: from unix.sh (01-033.026.popsite.net [64.24.105.33]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id AAA22244; Wed, 6 Jun 2001 00:30:41 -0600 Message-ID: <3B1DCDDA.8803B4E2@unix.sh> Date: Wed, 06 Jun 2001 00:29:46 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Bill Todd CC: linux-cluster Subject: Re: A proposal for a General Clustering Framework References: <20010606015801.18290.qmail@web9204.mail.yahoo.com> <3B1DAE2B.43771C47@unix.sh> <02bd01c0ee4c$276c2d60$94627dc7@filesrus> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Bill Todd wrote: > > ----- Original Message ----- > From: "Alan Robertson" > To: "Peter Badovinatz" > Cc: "linux-cluster" > Sent: Wednesday, June 06, 2001 12:14 AM > Subject: Re: A proposal for a General Clustering Framework > > > Peter Badovinatz wrote: > > > > > > > [snip] > > > > > Depends... We could push customers who really cared about strict HA to > avoid > > > version heterogeneity except during actual node-by-node upgrade. It was > > > customers who were more oriented to cluster file system - where we still > needed > > > (a bit looser form of) HA and failover but they could be up to 500 node > > > clusters - that we had no choice but to be very flexible as to > supporting > > > multiple versions across nodes because upgrading 500 nodes a few at a > time > > > takes a long time, and unlike more controlled HA clusters the workload > was > > > quite varied so many more things to upgrade/test/migrate. > > > > > > The better choice is to be sure the framework lets you handle multiple > active > > > version levels from the beginning, and you work to minimise their use by > decree > > > (as above) or by only supporting some limited number of levels (N and > N+1 or > > > N+2.) > > Since Alan hasn't responded to my request for a reassessment (I understand > that he's been busy today), and since the comments above may reflect a > similar lack of understanding, I'll reiterate: > > Supporting at least version N+1 of the *software* along with version N is > clearly required to achieve on-line rolling upgrades. What is not required > is to support (concurrently) multiple versions of the communication > protocols: you can instead support only the old protocol set until all > nodes have been upgraded to version N+1, and then perform a cluster-wide > synchronous cut-over to the new protocol version. > > If the cluster can be assumed to be under the control of a single > organization (in contrast to the nodes on the Internet, for example), then > requiring that all nodes be upgraded before the communication protocols > change is not prima facie unreasonable: indeed, it is difficult to create > examples where a protocol-upgrade is urgently needed but only by some subset > of the cluster's members, since a cluster is a considerably more > tightly-bound entity than the nodes on the Internet. You're right that an HA cluster owner has much more control over their cluster than the average "n" random computers on the internet. You're right about probably not needing to have more than two versions of any one given entity in the cluster at the same time. However, I would disagree that they are only one version apart. Most HA customers are extremely conservative. Many of them upgrade only when forced to by something outside their control. This sometimes results in only upgrading every 5 years. This might mean the versions of various software pieces would be 3-15 versions apart. I'm not sure if there's a big technical difference between being able to support any two releases that are 15 versions apart from saying that you're going to support 15 versions at once. Administratively, there is a *huge* difference. "This way lies madness". Also, I'm not sure if all the technologies we develop will only be used on clusters whose administration is tightly coupled. I think that the mechanisms need to be capable of dealing with large differences - and that in practice you only support certain combinations *that you've tested*. Those supported combinations are probably dictated more by politics and money than technical judgement ;-) -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 6 16:55:05 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 6 Jun 2001 16:54:55 +0200 Received: from gw.xkey.com ([206.86.100.52]:61958 "EHLO happy.xkey.com") by humbolt.nl.linux.org with ESMTP id ; Wed, 6 Jun 2001 16:54:39 +0200 Received: (from smtp@localhost) by happy.xkey.com id HAA19578 for ; Wed, 6 Jun 2001 07:54:37 -0700 Received: from happy(127.0.0.1) by happy.xkey.com via smtp (V1.3) id sma019572; Wed Jun 6 07:54:29 2001 Received: (from lindahl@localhost) by localhost.hpti.com (8.11.0/8.11.0) id f56Eusr01712 for linux-cluster@nl.linux.org; Wed, 6 Jun 2001 10:56:54 -0400 X-Authentication-Warning: localhost.hpti.com: lindahl set sender to lindahl@conservativecomputer.com using -f Date: Wed, 6 Jun 2001 10:56:54 -0400 From: Greg Lindahl To: linux-cluster Subject: Re: A proposal for a General Clustering Framework Message-ID: <20010606105654.B1696@wumpus> Mail-Followup-To: linux-cluster References: <3B1D3975.60FC764D@unix.sh> <20010605160547.A2291@wumpus> <3B1D3F51.537D23@unix.sh> <3B1D5841.C099E0E@unix.sh> <20010605184034.A2725@wumpus> <3B1D889E.A827CBD1@unix.sh> <20010605215517.A3213@wumpus> <3B1DAD17.4319AA92@unix.sh> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B1DAD17.4319AA92@unix.sh>; from alanr@unix.sh on Tue, Jun 05, 2001 at 10:09:59PM -0600 Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list On Tue, Jun 05, 2001 at 10:09:59PM -0600, Alan Robertson wrote: > This ticked off a minor flame war the last time I mentioned it, but > heartbeat distributes basic load information with every packet. It's only a > few bytes, and compared to the overhead in the RPC-XML format, a very small > amount of data indeed ;-) That's fine as long as pub/sub is the interface presented to higher layers. I may or may not want to transmit load info as frequently as heartbeats; what you're saying is that using this heartbeat+info module means that I get another choice. The flamewar last time was from people saying THEY didn't want to transmit info that way. Fine, that's a policy issue, not a mechanism issue. One of the slogans of Legion, a distributed OS that I used to work on, is "mechanism, not policy!" -- greg Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Sun Jun 17 15:03:07 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Sun, 17 Jun 2001 15:02:58 +0200 Received: from perth.fpcc.net ([207.174.142.141]:12828 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Sun, 17 Jun 2001 15:02:46 +0200 Received: from unix.sh (02-080.026.popsite.net [64.24.105.80]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id HAA05227 for ; Sun, 17 Jun 2001 07:02:42 -0600 Message-ID: <3B2CAA3B.629BAE78@unix.sh> Date: Sun, 17 Jun 2001 07:01:47 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: linux-cluster Subject: Updated General Cluster framework draft document Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Hi, I have incorporated the latest comments from the last round of discussions on the framework document into it. As before, comments and discussions are encouraged. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Sun Jun 17 15:06:18 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Sun, 17 Jun 2001 15:06:09 +0200 Received: from perth.fpcc.net ([207.174.142.141]:15388 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Sun, 17 Jun 2001 15:05:57 +0200 Received: from unix.sh (02-080.026.popsite.net [64.24.105.80]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id HAA05249; Sun, 17 Jun 2001 07:05:53 -0600 Message-ID: <3B2CAAFB.E7353B96@unix.sh> Date: Sun, 17 Jun 2001 07:04:59 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: David Brower , linux-cluster Subject: Re: An apology offerred References: <20010606021240.5241.qmail@web9208.mail.yahoo.com> <3B1DAE2C.9659A8FB@oracle.com> <3B1DB5E9.64513E7C@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > [various things] > > -- Alan Robertson > alanr@unix.sh I also have to offer an apology. My last letter with this subject line was not intended to be sent to the list either. My apologies to all concerned. -- Alan [reply-to-all-blind] Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Sun Jun 17 15:08:19 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Sun, 17 Jun 2001 15:08:10 +0200 Received: from perth.fpcc.net ([207.174.142.141]:16668 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Sun, 17 Jun 2001 15:08:03 +0200 Received: from unix.sh (02-080.026.popsite.net [64.24.105.80]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id HAA05257 for ; Sun, 17 Jun 2001 07:08:01 -0600 Message-ID: <3B2CAB7A.B12A1359@unix.sh> Date: Sun, 17 Jun 2001 07:07:06 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: linux-cluster Subject: Re: Updated General Cluster framework draft document References: <3B2CAA3B.629BAE78@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Hi, > > I have incorporated the latest comments from the last round of discussions > on the framework document into it. > > As before, comments and discussions are encouraged. > > -- Alan Robertson > alanr@unix.sh Would you like to tell us where you put this document? Oh, yeah... http://linux-ha.org/framework/ Sorry... -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 18 06:19:28 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 18 Jun 2001 06:19:20 +0200 Received: from [207.174.142.141] ([207.174.142.141]:5928 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Mon, 18 Jun 2001 06:19:15 +0200 Received: from unix.sh (01-060.026.popsite.net [64.24.105.60]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA08419 for ; Sun, 17 Jun 2001 22:19:12 -0600 Message-ID: <3B2D8108.2D595CF4@unix.sh> Date: Sun, 17 Jun 2001 22:18:16 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: linux-cluster Subject: Re: Updated General Cluster framework draft document References: <3B2CAA3B.629BAE78@unix.sh> <3B2CAB7A.B12A1359@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Alan Robertson wrote: > > > > Hi, > > > > I have incorporated the latest comments from the last round of discussions > > on the framework document into it. > > > > As before, comments and discussions are encouraged. > > > > -- Alan Robertson > > alanr@unix.sh > > Would you like to tell us where you put this document? > > Oh, yeah... > > http://linux-ha.org/framework/ More dumbness... I forgot to push the new versions out to the web. They're there now. Sorry!! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 18 21:36:52 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 18 Jun 2001 21:36:43 +0200 Received: from f129.law9.hotmail.com ([64.4.9.129]:39698 "EHLO hotmail.com") by humbolt.nl.linux.org with ESMTP id ; Mon, 18 Jun 2001 21:36:29 +0200 Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Mon, 18 Jun 2001 12:24:50 -0700 Received: from 63.228.102.79 by lw9fd.law9.hotmail.msn.com with HTTP; Mon, 18 Jun 2001 19:24:50 GMT X-Originating-IP: [63.228.102.79] From: "Charly Jones" To: alanr@unix.sh, linux-cluster@nl.linux.org Subject: clusters of computers - that are all the same? Date: Mon, 18 Jun 2001 19:24:50 -0000 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 18 Jun 2001 19:24:50.0612 (UTC) FILETIME=[5810CF40:01C0F82C] Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Is the framework intended to include clusters of homogenous computers and operating systems? Or is it intended to include different hardware and operating systems? Different releases of Linux? Linux machines clustered with Unix machines? I am a newcomer to this topic... Charly Jones Vision Solutions Inc 253 265-6244 cjones@visionsolutions.com _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 19 02:25:20 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 19 Jun 2001 02:25:12 +0200 Received: from perth.fpcc.net ([207.174.142.141]:11310 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 19 Jun 2001 02:25:04 +0200 Received: from unix.sh (07-133.026.popsite.net [64.24.106.133]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id SAA17385; Mon, 18 Jun 2001 18:24:47 -0600 Message-ID: <3B2E9B96.AF59DCCB@unix.sh> Date: Mon, 18 Jun 2001 18:23:50 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Charly Jones CC: linux-cluster@nl.linux.org Subject: Re: clusters of computers - that are all the same? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Charly Jones wrote: > > Is the framework intended to include clusters of homogenous computers and > operating systems? Or is it intended to include different hardware > and operating systems? Different releases of Linux? Linux machines > clustered with Unix machines? > > I am a newcomer to this topic... Good question! I've never answered it directly before (this is a new proposal). The framework itself is agnostic. It does not / should not enforce homogeneity of any kind, nor rely on it. Framework *components* may or not be suitable for a heterogeneous environment - depending on the component. It's up to the component. Myself, I like components that allow heterogeneous nodes in the cluster, but the implementator of a component has the freedom to allow or disallow any particular mix of nodes. For example, high-bandwidth HPC interconnects may only work between homogeneous systems. There are good performance reasons why this may be required - and the framework doesn't dictate how the components work under the covers. However, HA components should work across a mix of different releases of OSes, and different releases of application software, and framework components. This is because HA systems have to work in an environment of rolling upgrades, where different software releases introduced into the cluster one node at a time. This is guaranteed to introduce all kinds of interesting differences into the cluster - and these differences have to be handled well. The fundamental control messaging system is XML-based which is very amenable to dealing with these kinds of differences, and providing tools and infrastructure which makes it easier and more natural for the components to do the same. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 19 02:41:28 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 19 Jun 2001 02:41:20 +0200 Received: from perth.fpcc.net ([207.174.142.141]:17966 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 19 Jun 2001 02:41:13 +0200 Received: from unix.sh (07-133.026.popsite.net [64.24.106.133]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id SAA17474; Mon, 18 Jun 2001 18:41:09 -0600 Message-ID: <3B2E9F6C.C807425C@unix.sh> Date: Mon, 18 Jun 2001 18:40:12 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Charly Jones , linux-cluster@nl.linux.org Subject: Re: clusters of computers - that are all the same? References: <3B2E9B96.AF59DCCB@unix.sh> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Alan Robertson wrote: > > Charly Jones wrote: > > > > Is the framework intended to include clusters of homogenous computers and > > operating systems? Or is it intended to include different hardware > > and operating systems? Different releases of Linux? Linux machines > > clustered with Unix machines? > > > > I am a newcomer to this topic... > > Good question! I've never answered it directly before (this is a new > proposal). > > The framework itself is agnostic. > > It does not / should not enforce homogeneity of any kind, nor rely on it. > > Framework *components* may or not be suitable for a heterogeneous > environment - depending on the component. It's up to the component. > Myself, I like components that allow heterogeneous nodes in the cluster, but > the implementator of a component has the freedom to allow or disallow any > particular mix of nodes. I don't think I answered part of your question: I'm currently targeting the framework itself to be built using automake, and support Linux and at least a few non-Linux platforms. The framework itself should work on any of the supported platforms, and should treat all platforms roughly the same. I currently expect the list of supported platforms to include these OSes: Linux, OpenBSD, FreeBSD, Solaris. This list could expand or contract depending on the resources the development community applies to the project. Some components may require OS modules which are only available on certain platforms. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 19 06:54:25 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 19 Jun 2001 06:54:18 +0200 Received: from perth.fpcc.net ([207.174.142.141]:6196 "EHLO perth.fpcc.net") by humbolt.nl.linux.org with ESMTP id ; Tue, 19 Jun 2001 06:54:10 +0200 Received: from unix.sh (01-026.026.popsite.net [64.24.105.26]) by perth.fpcc.net (8.9.3/8.9.3) with ESMTP id WAA18845; Mon, 18 Jun 2001 22:53:56 -0600 Message-ID: <3B2EDAA7.FD74F59F@unix.sh> Date: Mon, 18 Jun 2001 22:52:55 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: linux-cluster CC: Lars Marowsky-Bree , Anas Nashif Subject: HA working group meeting at OLS Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Hi, Lars Marowsky-Bree of SuSE has scheduled a working group meeting for the Ottawa Linux Symposium (OLS) http://www.linuxsymposium.org/ The dates are July 25-28. The exact timing for this working group meeting has not been finalized but is expected to be between 1.5 and 3 hours. The working group topic is High-Availability clusters. I have asked for some time during the working group meeting to discuss the cluster framework proposal which I have discussed on this mailing list in recent days. As you know, this proposal is not restricted to HA clusters, but will eventually cover services for HPC clusters as well. My funding for attending OLS is dependent on there being people for me to discuss the framework proposal with at the working group meeting. PLEASE consider attending, and let me know ASAP (like Tuesday or Wednesday) if you plan on attending the working group. An incomplete rough draft of the cluster framework proposal can be found here: http://linux-ha.org/framework/ -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Sat Jun 23 03:03:44 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Sat, 23 Jun 2001 03:03:37 +0200 Received: from [192.216.221.8] ([192.216.221.8]:35748 "EHLO suntan.tandem.com") by humbolt.nl.linux.org with ESMTP id ; Sat, 23 Jun 2001 03:03:29 +0200 Received: from kahuna.cag.cpqcorp.net (kahuna.cag.cpqcorp.net [16.61.168.50]) by suntan.tandem.com (8.9.3/2.0.1) with ESMTP id SAA19933 for ; Fri, 22 Jun 2001 18:03:17 -0700 (PDT) From: bruce@kahuna.cag.cpqcorp.net Received: (from bruce@localhost) by kahuna.cag.cpqcorp.net (8.10.1/UW7.1.1-NSC) id f5N0sDJ02195; Fri, 22 Jun 2001 17:54:13 -0700 (PDT) Message-Id: <200106230054.f5N0sDJ02195@kahuna.cag.cpqcorp.net> To: linux-cluster@nl.linux.org, linux-kernel@vger.kernel.org Date: Fri, 22 Jun 2001 17:50 PDT Subject: Compaq launches Open SSI Cluster Projects Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Compaq has launched two open source technology projects under the GPL license. They are briefly described below and can be found through www.opensource.compaq.com. We are actively looking for technology partners, contributors, consultants and general kibitzers to participate via the email lists set up for each project. Those that just want to monitor the projects are welcome as well. Cluster Infrastructure for Linux (CI) The goal of this project is to develop a common infrastructure for many if not all forms of Linux clustering by extending the Cluster Membership and Inter-node Communication Subsystems from Compaq's NonStop Clusters for Unixware code base. This project also provides the basis for the Open SSI Clusters for Linux project. A developers download is available via www.opensource.compaq.com for Intel-32, along with build, boot, hook, interface and api documentation. We will put the CVS repository on the web when we can. A port to the alpha chip has already succeeded and patches for that are available. Open Single System Image (SSI) Clusters for Linux Project The Open SSI project leverages both Compaq's NonStop Clusters for Unixware technology and other open source technology to provide a full, highly available SSI environment for Linux. Goals for SSI Clusters include availability, scalability and manageability, built from standard servers. Technology pieces will include: membership, single root and single init, cluster filesystems and DLM, single process space and process migration, load leveling, availability monitors and failover, single namespace and shared access for all forms of IPC, devices and networking, and a single management space. The SSI project will leverage the Cluster Infrastructure for Linux project. Source beyond the CI base is not yet available. We are aiming for a developers release of much of functionality in July. In the meantime there is a presentation on SSI Clustering on the web. An initial list of component requirements will soon be posted for discussion and refinement. Join the mail alias via www.opensource.compaq.com to stay updated. bruce walker SSI Cluster Architect Linux Program Office Compaq Computers Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Mon Jun 25 14:48:09 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Mon, 25 Jun 2001 14:48:02 +0200 Received: from 01-025.026.popsite.net ([64.24.105.25]:55806 "EHLO host.domain.name") by humbolt.nl.linux.org with ESMTP id ; Mon, 25 Jun 2001 14:47:46 +0200 Received: from unix.sh (localhost [127.0.0.1]) by host.domain.name (Postfix) with ESMTP id 785A3263E7; Mon, 25 Jun 2001 06:46:57 -0600 (MDT) Message-ID: <3B3732C1.D65ECE46@unix.sh> Date: Mon, 25 Jun 2001 06:46:57 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: bruce@kahuna.cag.cpqcorp.net Cc: linux-cluster@nl.linux.org, linux-kernel@vger.kernel.org, ha-linux List Subject: Re: Compaq launches Open SSI Cluster Projects References: <200106230054.f5N0sDJ02195@kahuna.cag.cpqcorp.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list bruce@kahuna.cag.cpqcorp.net wrote: > > Compaq has launched two open source technology projects > under the GPL license. [snip] > Cluster Infrastructure for Linux (CI) > The goal of this project is to develop a common > infrastructure for many if not all forms of Linux > clustering by extending the Cluster Membership and > Inter-node Communication Subsystems from Compaq's > NonStop Clusters for Unixware code base. Hi Bruce, Welcome to the growing ranks of companies who've decided to create a new open source high-availability project! In the Linux community, we seem to have an embarassment of riches when it comes to HA software. As you are aware, there is an effort to create a framework of common APIs and infrastructure for cluster systems and cluster-aware apps. It covers HA and HPC clustering, and is intended to encourage sharing of components, and creation of common interfaces for cluster-aware apps in order to maximize the benefits of the open source development model, and to create a little sanity for developers who wish to write cluster-aware apps for Linux and other environments supported by the Cluster Framework project. The cluster framework project will support (at least) Linux, OpenBSD, Solaris, FreeBSD. It will probably also support other OSes (such as Tru64 UNIX), depending on community interest. It sounds like there is an excellent match between your desires and those of the cluster framework project. There is a need for different implementations of cluster components to meet the diverse needs of cluster users. I'm confident that your COMPAQ CI for Linux system would make an excellent addition to the list of cluster components available with the framework. As a note, it is not necessary for any given cluster component (like the COMPAQ CI for Linux (CCIL?)) to be available for every platform. Since the APIs are common, applications need not be aware of the differences. There is an work-in-progress paper on the Cluster Framework available on http://linux-ha.org/framework/ Congratulations and Welcome! -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Tue Jun 26 21:37:52 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Tue, 26 Jun 2001 21:37:34 +0200 Received: from f163.law9.hotmail.com ([64.4.9.163]:47115 "EHLO hotmail.com") by humbolt.nl.linux.org with ESMTP id ; Tue, 26 Jun 2001 21:37:25 +0200 Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Tue, 26 Jun 2001 12:25:55 -0700 Received: from 204.1.130.113 by lw9fd.law9.hotmail.msn.com with HTTP; Tue, 26 Jun 2001 19:25:54 GMT X-Originating-IP: [204.1.130.113] From: "Charly Jones" To: charly3005@hotmail.com, alanr@unix.sh, linux-cluster@nl.linux.org Subject: XML support of strange languages Date: Tue, 26 Jun 2001 19:25:54 -0000 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 26 Jun 2001 19:25:55.0159 (UTC) FILETIME=[D1D7E670:01C0FE75] Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list I am interested in the XML aspect of the Cluster Framework. I haven't yet studied XML, but I am going to. Between now and when I finally do that, maybe someone would be kind enough to answer a few questions... Is there a good open source version of an XML parser? Does XML support unicode? Do you think the Cluster Framework should support configuration in a language other than English? If yes, and if we had a cluster with a node with a primary language of French, would you think it is reasonable to have another node in the same cluster that has a primary language of English? If yes, would you think the entire cluster would have to be configured in one language, or would it be possible to configure one node in French and the other in English? Let me know if I am smoking bad dope... Charly Jones Vision Solutions Inc 253 265-6244 949 253-6515 (message only) _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/ From owner-linux-cluster@nl.linux.org Wed Jun 27 17:33:08 2001 Received: (root@humbolt.nl.linux.org) by humbolt.nl.linux.org id ; Wed, 27 Jun 2001 17:32:51 +0200 Received: from gull.mail.pas.earthlink.net ([207.217.121.85]:53211 "EHLO gull.mail.pas.earthlink.net") by humbolt.nl.linux.org with ESMTP id ; Wed, 27 Jun 2001 17:32:44 +0200 Received: from unix.sh (cpe-24-221-212-80.co.sprintbbd.net [24.221.212.80]) by gull.mail.pas.earthlink.net (EL-8_9_3_3/8.9.3) with ESMTP id IAA23407; Wed, 27 Jun 2001 08:32:40 -0700 (PDT) Message-ID: <3B3A4036.52EAD729@unix.sh> Date: Wed, 27 Jun 2001 14:21:10 -0600 From: Alan Robertson Organization: IBM Linux Technology Center X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.2.18 i686) X-Accept-Language: en MIME-Version: 1.0 To: Charly Jones CC: linux-cluster@nl.linux.org Subject: Re: XML support of strange languages References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-cluster@nl.linux.org Precedence: bulk Return-Path: X-Envelope-To: <"| /bin/marchive -a -m -f /home/majordomo/public_html/linux-cluster/folders/linux-cluster"> (uid 0) X-Orcpt: rfc822;linux-cluster-list Charly Jones wrote: > > I am interested in the XML aspect of the Cluster Framework. > I haven't yet studied XML, but I am going to. Between now and > when I finally do that, maybe someone would be kind enough to > answer a few questions... As best I can ;-) > Is there a good open source version of an XML parser? There are apparently several, but only two that I know of which are native 'C': libXML from the Gnome project - about 3 times the size of the current heartbeat cluster system expat - only about twice the size of the current heartbeat code So, both are way too large to be locked into memory. We have a prototype XML parser which is 5% the size of libXML. It looks pretty reasonable. > Does XML support unicode? Yes. I don't plan on supporting any Unicode encoding except for UTF-8, because it's so 'C' friendly, and much easier to program. > Do you think the Cluster Framework should support configuration > in a language other than English? Without a doubt. > If yes, and if we had a > cluster with a node with a primary language of French, would > you think it is reasonable to have another node in the same > cluster that has a primary language of English? If yes, > would you think the entire cluster would have to be configured > in one language, or would it be possible to configure one node > in French and the other in English? I have a different paradigm... All cluster members are configured the the same (as nearly to identically as possible). Configuration does not necessarily take place on one of the cluster nodes. It probably takes place from a client which is not a member of the cluster. Otherwise remote administration can be difficult. For example, it could be done by Java programs/applets (like FailSafe). Any given cluster configuration client program is configured in some national language. For example, if I'm running a configuration client on a Java client on a desktop that's configured in French, then the APIs need to support passing that config information (language = French) to the back end, and acting on it. If someone else is connected from a different client to the same node in the same cluster, they could get a different set of messages. This is necessary, and should be controlled by the environment the configuration/status client in - not the node in the cluster that's servicing the request. > Let me know if I am smoking bad dope... I don't think so. ;-) On a related subject: I don't imagine that many (if any) messages between cluster nodes (over cluster transport) will be sensitive to national language. That is, they wouldn't often contain such messages. Those would occur most often between a configuration client and the cluster. I don't envision such communication to occur over cluster message transport communication paths. -- Alan Robertson alanr@unix.sh Linux-cluster: generic cluster infrastructure for Linux Archive: http://mail.nl.linux.org/linux-cluster/