Seeing as the specific data in question could be considered readonly to the majority of clients, why not have them use an automount map with both filers in.
We do that for our centrally maintained and served binaries and db's and it works very well.
Hi,
We have two netapp filers serving mainly static data (updated overnight) to several hp-ux clients.
If one server fails we can swap the clients onto the other server by mounting and switching symbollic links.
This will help for any new mounts (so you don't have to go through the bother of relinking) but won't solve the problem of processes hanging on already mounted filesystems.
As already stated, for the "live" server to take over it would have to have an interface with both the IP address and MAC address changed and the NVRAM state copied. But also, I don't think that the NFS filehandles for the "live" filesystem would match the "dead" filesystem and you would get stale filehandles so this would only work if you were sharing the same filesystem between the two systems (which is what the new Netapp failover solution does).
-Steve gremban@ti.com
Mark D Simmons wrote:
Seeing as the specific data in question could be considered readonly to the majority of clients, why not have them use an automount map with both filers in.
We do that for our centrally maintained and served binaries and db's and it works very well.
Hi,
We have two netapp filers serving mainly static data (updated overnight) to several hp-ux clients.
If one server fails we can swap the clients onto the other server by mounting and switching symbollic links.
So, are you saying that the new NetApp failover system has two toasters hooked up to the same RAID set, and then the secondary toaster will mirror the state of the primary toaster? Can they share the same NFS state, including file handles and such, so that there are no dead/stale handles? Ok, I have a more abstract question about HA (high availability) in general. Here's what I know. There are many delicate issues involved in HA, that stem from the fact that you're using technology that's primarily designed as standalone. In other words, it's a group of _individual_ hosts. The issues happen because you're trying to get one host to dynamically assume the identity of another host, on some level or all levels. Am I wrong on any of that? So my question is, what are the main factors involved? I know what happens when you have a high-level router, switch, bridge, etc and you swap an ethernet card in a machine. It won't work until you call your router admins and have them reset the arp cache :) I'm guessing that this makes HA impossible. It seems to me that we have to do the following:
1) configure complimentary equipment such as routers, to not be so picky. is this possible? a low-level ethernet device like a router/switch needs to map ethernet hardware mac addresses to IP addresses.
2) design each piece of equipment in the HA environment to be HA-aware. no more plugging in devices one by one. that requires a paradigm switch between the object-oriented to the relationship-oriented. it's no longer primarily all about boxes and addresses -- it's about how they relate -- who's responsible for who.
3) i dont know! point me to where I can read about HA theory, and what's been tried, and who else is in the game. i've built linux software RAIDs and i'm on linux-raid@vger.rutgers.edu.
On Thu, 10 Sep 1998, Steve Gremban wrote:
As already stated, for the "live" server to take over it would have to have an interface with both the IP address and MAC address changed and the NVRAM state copied. But also, I don't think that the NFS filehandles for the "live" filesystem would match the "dead" filesystem and you would get stale filehandles so this would only work if you were sharing the same filesystem between the two systems (which is what the new Netapp failover solution does).
` ~ ^ ' ~ ' ` ^ ~ ~ ^ ' ~ ` ` ~ ' ^ ` ~ ~ ` " ` ~ ' ^ " ^ ` ~ ' ~ ^ " ' ~ ` ^ ~ "If I seem too inconclusive, well it's just because it's so elusive." This email has been licensed by the GPL (http://www.gnu.org/philosophy) Dan ((((now seeking a Linux/Unix sysadmin job in Silicon Valley!)))) Bethe
At the www.netapp.com web site is a white paper on netapp's clustered failover (CF) design. We have two F630s that we plan to convert to CF. Netapp has not released CF yet, but we have read the white paper and had some of our questions answered by tech folks.
Clustered failover connects two filers to the same stack of disk shelves. Each filer has its own set of volumes built on its own set of disks, so during normal operation, the cluster behaves like (and has the performance of) two separate filers. If one filer fails, the healthy filer takes over the failed filer's volumes and starts serving them.
It looks like a failover will be about as disruptive as a reboot, i.e., NFS mounts via udp will survive, but all CIFS connections will be lost (on the failed filer). Presumably the healthy filer will not have its service interrupted. Failover will take a few minutes, so it will appear as if the failed filer simply rebooted.
Once the failed hardware is repaired, I'm pretty sure that going back to normal operation requires rebooting the healthy filer, because it has to "offline" the volumes that it took over and that requires a reboot.
So CF doesn't eliminate service disruptions. It just means that certain hardware failures become no more disruptive than one unscheduled reboot and one scheduled reboot. Plus you have poorer performance while one filer does the work of two.
Designing CF so that there are absolutely no service disruptions might very well require one filer to be in "standby" mode at all times and do no work during normal operations. Since hardware failures are rare, this is wasteful.
The key to clustered failover is that each filer uses only half of its own NVRAM and mirrors its NVRAM state onto the unused half of its partner's NVRAM. So in the case of a hardware failure, the healty filer takes over the volumes of the failed filer, takes over the IP address(es) of the failed filer, and starts serving the volumes. The takeover procedure is similar to when a standalone filer recovers from an abrupt power outage. The filesystem resumes at the latest WAFL consistency point, and the NVRAM log is replayed to perform transactions that happened after the consistency point.
Clustered failover requires fibre channel disks because you can't hook SCSI disk shelves up to multiple filers. Each fibre channel disk shelf in a CF system has two fibre channel interfaces, and each filer has its own daisy chain linking it to all the shelves. During normal operation each disk is "owned" by one filer. It appears that disks from the same shelf can be owned by different filers, so you can add a disk to any shelf and assign it to either filer. I don't know if hot spares can be left unassigned, so it may be necessary for each filer to have its own hot spare(s) assigned to it. In the event of a failover, the healthy filer takes over all the disks.
Netapp's CF supports only two filers. This makes sense since each disk shelf must have a fibre channel interface for each filer in the cluster. This puts a rather severe limit on the theoretical size of a cluster using this hardware.
Steve Losen scl@virginia.edu phone: 804-924-0640
University of Virginia ITC Unix Support
A couple of things I just gotta throw in here since Adaptec dumped Sun/Veritas HA for the future NetApp CF solution ;^)
1) Failover will take a few minuted indeed. Just no way around that. 2) Clustered failover should be possible on FWD SCSI. That was the way Sun did it. You are limited to your implementation of drive attachment. Sun used a dual initiated SCSI processor (now the StorEDGE arrays) that attached to all the other disks. It works, but it's also really slooooowwww :^) Known as Dual Initiated SCSI. 3) My understanding of HA and CF during failover is this: * The failed server goes away (don't care how or why, it just does) * The second server picks this up via the interconnect and begins a few key things, i.e. spoof the network interfaces of the failed server, grab its vols, etc. Now, unless NetApp is implementing some very strange code (which I don't think they are doing <grin> ) the backup NICs on the second server will pick up hostname, ip, and MAC address. As far as I know that's the only way to implement it. 4) Reverting back to normal modes should be the reverse of failing over. It should only take a few minutes, but you should be able to do it live. Otherwise what's the point. If you have to schedule a reboot why have CF? I know this is a simplistic view of the decision process, but if the failover isn't relatively transparent to my NFS and CIFS clients I don't need CF. Sorry, coming back... So you should be able to signal the Dual service machine that the failed machine is coming back online, have it let go of the NIC (basically ifconfig down) and volumes, and have the new machine pick up any NVRAM updates for itself and continue chugging away.
Failover and failback should be that easy. I'll admit that it's a pain to set up any HA or CF solution, but once you get it configured it's a breeze.
BTW having a backup machine idle will still take an apparent "reboot" to bring on-line, and not all your clients will reconnect due to MAC address changes.
I hope this info helps a bit. Sorry for the soap box :^)
--Steve
Stephen C. Losen wrote:
At the www.netapp.com web site is a white paper on netapp's clustered failover (CF) design. We have two F630s that we plan to convert to CF. Netapp has not released CF yet, but we have read the white paper and had some of our questions answered by tech folks.
Clustered failover connects two filers to the same stack of disk shelves. Each filer has its own set of volumes built on its own set of disks, so during normal operation, the cluster behaves like (and has the performance of) two separate filers. If one filer fails, the healthy filer takes over the failed filer's volumes and starts serving them.
It looks like a failover will be about as disruptive as a reboot, i.e., NFS mounts via udp will survive, but all CIFS connections will be lost (on the failed filer). Presumably the healthy filer will not have its service interrupted. Failover will take a few minutes, so it will appear as if the failed filer simply rebooted.
Once the failed hardware is repaired, I'm pretty sure that going back to normal operation requires rebooting the healthy filer, because it has to "offline" the volumes that it took over and that requires a reboot.
So CF doesn't eliminate service disruptions. It just means that certain hardware failures become no more disruptive than one unscheduled reboot and one scheduled reboot. Plus you have poorer performance while one filer does the work of two.
Designing CF so that there are absolutely no service disruptions might very well require one filer to be in "standby" mode at all times and do no work during normal operations. Since hardware failures are rare, this is wasteful.
The key to clustered failover is that each filer uses only half of its own NVRAM and mirrors its NVRAM state onto the unused half of its partner's NVRAM. So in the case of a hardware failure, the healty filer takes over the volumes of the failed filer, takes over the IP address(es) of the failed filer, and starts serving the volumes. The takeover procedure is similar to when a standalone filer recovers from an abrupt power outage. The filesystem resumes at the latest WAFL consistency point, and the NVRAM log is replayed to perform transactions that happened after the consistency point.
Clustered failover requires fibre channel disks because you can't hook SCSI disk shelves up to multiple filers. Each fibre channel disk shelf in a CF system has two fibre channel interfaces, and each filer has its own daisy chain linking it to all the shelves. During normal operation each disk is "owned" by one filer. It appears that disks from the same shelf can be owned by different filers, so you can add a disk to any shelf and assign it to either filer. I don't know if hot spares can be left unassigned, so it may be necessary for each filer to have its own hot spare(s) assigned to it. In the event of a failover, the healthy filer takes over all the disks.
Netapp's CF supports only two filers. This makes sense since each disk shelf must have a fibre channel interface for each filer in the cluster. This puts a rather severe limit on the theoretical size of a cluster using this hardware.
Steve Losen scl@virginia.edu phone: 804-924-0640
University of Virginia ITC Unix Support
____________________________________________________
Engineering Systems and Network Administrator
phone: 408 957 2351 fax: 408 957 4895 email: nevets@eng.adaptec.com
Adaptec Milpitas, CA, USA ____________________________________________________
Now, unless NetApp is implementing some very strange code (which Idon't think they are doing <grin> ) the backup NICs on the second server will pick up hostname, ip, and MAC address. As far as I know that's the only way to implement it.
Yup, that's what happens.
- Reverting back to normal modes should be the reverse of failing over. It
should only take a few minutes, but you should be able to do it live.
You can, as per my other mail.
So you should be able to signal the Dual service machine that the failedmachine is coming back online, have it let go of the NIC (basically ifconfig down) and volumes,
Yup - that's done with a "cf giveback" command on the machine that took over ffor the other machine.
It looks like a failover will be about as disruptive as a reboot, i.e., NFS mounts via udp will survive,
NFS mounts via TCP, if you have NFS-over-TCP enabled, should also survive, assuming your NFS-over-TCP client isn't broken - NFS-over-TCP clients shouldn't assume that TCP connections persist forever, they should be able to re-establish them as necessary (especially given that I know of three NFS servers, off the top of my head, that will close idle NFS-over-TCP connections:
NetApp
Digital UNIX
Solaris 2.x
).
but all CIFS connections will be lost (on the failed filer).
In *some* situations, this doesn't cause a problem - at least some of Microsoft's CIFS clients will re-establish connections - but if there are any open files on the connection, there's too much state on the server for Microsoft's clients to recover (I don't know if they have plans to change that - it can be tricky, given that the client might've had the file open for exclusive use and that you don't want somebody else to be able to sneak in and open the file before the client in question has a chance to reopen it).
I.e., CIFS sessions *might* survive a takeover or giveback, but, if one does, consider yourself lucky - don't depend on it.
Presumably the healthy filer will not have its service interrupted.
It shouldn't.
Once the failed hardware is repaired, I'm pretty sure that going back to normal operation requires rebooting the healthy filer,
Nope.
because it has to "offline" the volumes that it took over and that requires a reboot.
No, it just has to give them back to the filer from which it took them; that, unlike taking the volumes offline, doesn't require a reboot. (I don't *think* the first CF release will lift the restriction that you can't take a volume offline without doing a reboot, but it might; we don't view that as a permanent feature, we want to lift that restriction eventually.)
I don't know if hot spares can be left unassigned, so it may be necessary for each filer to have its own hot spare(s) assigned to it.
Hot spares are, as I remember, owned by particular members of the CF pair.
(No, I don't know whether the people here who dubbed it "CF" realized that "CF" could stand for something else. :-))
From what I read about the Cluster, all connections survive. NFS, UDP and TCP, is stateless and can therefore easily re-establish itself. And the Cluster software syncs CIFS state information between the two filers so after a failover the remaining filer just picks up where the first left off. If this wasn't the case, why pay for the automatic failover...?
jason
In message 199809182111.OAA03435@tooting.netapp.com, Guy Harris writes:
It looks like a failover will be about as disruptive as a reboot, i.e., NFS mounts via udp will survive,
NFS mounts via TCP, if you have NFS-over-TCP enabled, should also survive, assuming your NFS-over-TCP client isn't broken - NFS-over-TCP clients shouldn't assume that TCP connections persist forever, they should be able to re-establish them as necessary (especially given that I know of three NFS servers, off the top of my head, that will close idle NFS-over-TCP connections:
NetApp
Digital UNIX
Solaris 2.x
).
but all CIFS connections will be lost (on the failed filer).
In *some* situations, this doesn't cause a problem - at least some of Microsoft's CIFS clients will re-establish connections - but if there are any open files on the connection, there's too much state on the server for Microsoft's clients to recover (I don't know if they have plans to change that - it can be tricky, given that the client might've had the file open for exclusive use and that you don't want somebody else to be able to sneak in and open the file before the client in question has a chance to reopen it).
I.e., CIFS sessions *might* survive a takeover or giveback, but, if one does, consider yourself lucky - don't depend on it.
Presumably the healthy filer will not have its service interrupted.
It shouldn't.
Once the failed hardware is repaired, I'm pretty sure that going back to normal operation requires rebooting the healthy filer,
Nope.
because it has to "offline" the volumes that it took over and that requires a reboot.
No, it just has to give them back to the filer from which it took them; that, unlike taking the volumes offline, doesn't require a reboot. (I don't *think* the first CF release will lift the restriction that you can't take a volume offline without doing a reboot, but it might; we don't view that as a permanent feature, we want to lift that restriction eventually.)
I don't know if hot spares can be left unassigned, so it may be necessary for each filer to have its own hot spare(s) assigned to it.
Hot spares are, as I remember, owned by particular members of the CF pair.
(No, I don't know whether the people here who dubbed it "CF" realized that "CF" could stand for something else. :-))
From what I read about the Cluster, all connections survive. NFS, UDP and TCP, is stateless and can therefore easily re-establish itself.
Yes (which I said in my mail), but...
And the Cluster software syncs CIFS state information between the two filers
It doesn't - whatever you read that claims that is mistaken.
Stephen;
An excellent understanding on how CF works.
We're looking forward to working with you on your implementation.
Regards
Lew Kirschner -------------
At 11:08 AM 9/18/1998 -0400, Stephen C. Losen wrote:
At the www.netapp.com web site is a white paper on netapp's clustered failover (CF) design. We have two F630s that we plan to convert to CF. Netapp has not released CF yet, but we have read the white paper and had some of our questions answered by tech folks.
Clustered failover connects two filers to the same stack of disk shelves. Each filer has its own set of volumes built on its own set of disks, so during normal operation, the cluster behaves like (and has the performance of) two separate filers. If one filer fails, the healthy filer takes over the failed filer's volumes and starts serving them.
It looks like a failover will be about as disruptive as a reboot, i.e., NFS mounts via udp will survive, but all CIFS connections will be lost (on the failed filer). Presumably the healthy filer will not have its service interrupted. Failover will take a few minutes, so it will appear as if the failed filer simply rebooted.
Once the failed hardware is repaired, I'm pretty sure that going back to normal operation requires rebooting the healthy filer, because it has to "offline" the volumes that it took over and that requires a reboot.
So CF doesn't eliminate service disruptions. It just means that certain hardware failures become no more disruptive than one unscheduled reboot and one scheduled reboot. Plus you have poorer performance while one filer does the work of two.
Designing CF so that there are absolutely no service disruptions might very well require one filer to be in "standby" mode at all times and do no work during normal operations. Since hardware failures are rare, this is wasteful.
The key to clustered failover is that each filer uses only half of its own NVRAM and mirrors its NVRAM state onto the unused half of its partner's NVRAM. So in the case of a hardware failure, the healty filer takes over the volumes of the failed filer, takes over the IP address(es) of the failed filer, and starts serving the volumes. The takeover procedure is similar to when a standalone filer recovers from an abrupt power outage. The filesystem resumes at the latest WAFL consistency point, and the NVRAM log is replayed to perform transactions that happened after the consistency point.
Clustered failover requires fibre channel disks because you can't hook SCSI disk shelves up to multiple filers. Each fibre channel disk shelf in a CF system has two fibre channel interfaces, and each filer has its own daisy chain linking it to all the shelves. During normal operation each disk is "owned" by one filer. It appears that disks from the same shelf can be owned by different filers, so you can add a disk to any shelf and assign it to either filer. I don't know if hot spares can be left unassigned, so it may be necessary for each filer to have its own hot spare(s) assigned to it. In the event of a failover, the healthy filer takes over all the disks.
Netapp's CF supports only two filers. This makes sense since each disk shelf must have a fibre channel interface for each filer in the cluster. This puts a rather severe limit on the theoretical size of a cluster using this hardware.
Steve Losen scl@virginia.edu phone: 804-924-0640
University of Virginia ITC Unix Support
NETWORK APPLIANCE ====================================================================== FAST! SIMPLE! RELIABLE! MULTIPROTOCOL!! ********************************************************************** Lew Kirschner - Eastern Area V-Mail: 732 603 7330 Reseller Mgr (V-Mail Only) Network Appliance Reach #: 914-369-3830 35 Sagamore Avenue Fax #: 914-369-3832 Suffern, New York 10901 e-mail: lewisk@netapp.com http://www.netapp.com ********************************************************************** The Market Leader in Network Attached Storage for Multiprotocol Environments!
-
Dan Bethe -
guy＠netapp.com -
Jason D. Kelleher -
Lewis M. Kirschner -
mds＠ns.gbnet.net -
Stephen C. Losen -
Steve Gremban -
Steven D'Aurora x2351