toasters

toasters@lists.teaparty.net

13517 discussions

Re: Problem mounting NFS from a Fedora Core 6 machine
by Gustavo Chaves 23 Nov '06

23 Nov '06

>>>>> On Thu, 23 Nov 2006 01:45:37 +0100, Åke Beckman <Ake.Beckman(a)proact.se> said: > I suppose the clients that aren't directly connected will specify > the interface that the netapp uses as its default route for their > mount requests, in which case the reply will always come back from > that interface too. Hence, there shouldn't be a problem? The reply comes from the same router, but the source IP address is different. Gustavo.

1 0

RE: simple SAN?!
by Davies,Matt 23 Nov '06

23 Nov '06

On the VMware side, as far as I understand you can only use FC to the physical machine, not to a virtual machine, as the VM's do not support FC. I believe you can still use VCB with Iscsi, it just needs a bit more scripting, I would check out the VMware discussion groups on the VMware website, I have always found them very helpful. Cheers Matt ________________________________ From: Willeke, Jochen [mailto:Jochen.Willeke@wincor-nixdorf.com] Sent: 23 November 2006 11:07 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: simple SAN?! We already use ISCSI, but we aer thinking about VMWare and VMWare has some nice features (e.g. VCB) which need an FC-SAN. Thanks anyway Jochen ________________________________ From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Thursday, November 23, 2006 11:59 AM To: Willeke, Jochen; toasters(a)mathworks.com Subject: RE: simple SAN?! If you are looking at low cost san, I would go Iscsi, as long as everything you need on Iscsi is on Gig ethernet ports, it should work fine. Cheers Matt ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Willeke, Jochen Sent: 23 November 2006 10:11 To: toasters(a)mathworks.com Subject: simple SAN?! Hi, as we use not too much SAN at our site i am not really used to this topic. Yesterday i read an article about "simple SAN". "Simple SAN" seems to mean as an plug-n-play solution. But what about the components espacially the SAN-switches? Do we see really new (and cheap :D) SAN-switches or only old products "reassembled" as a solution for small companies. Regards Jochen

2 1

RE: simple SAN?!
by Davies,Matt 23 Nov '06

23 Nov '06

If you are looking at low cost san, I would go Iscsi, as long as everything you need on Iscsi is on Gig ethernet ports, it should work fine. Cheers Matt ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Willeke, Jochen Sent: 23 November 2006 10:11 To: toasters(a)mathworks.com Subject: simple SAN?! Hi, as we use not too much SAN at our site i am not really used to this topic. Yesterday i read an article about "simple SAN". "Simple SAN" seems to mean as an plug-n-play solution. But what about the components espacially the SAN-switches? Do we see really new (and cheap :D) SAN-switches or only old products "reassembled" as a solution for small companies. Regards Jochen

2 1

iSCSI SAN Queries
by Raj Patel 23 Nov '06

23 Nov '06

Hi, We're in the preliminary scoping phase for a low/mid-range iSCSI SAN solution. I was wondering if anyone had any positive or negative real-world experiences with - NetApp 270c HDS AMS200 EMC CX320 At the moment the NetApp is the most familiar to me as I have used one of their NAS boxes before and the simulator provides a pretty good indication of how it works. However the iSCSI seems a bit of a 'bolt-on' and its not clear if it will handle tiered storage as well as the other vendors (then again does it matter?). The HDS & EMC are unknown quantities (other than what I can glean from the web). Any feedback concerning ease of use, expansion, licensing, snapshot mechanism (the EMC seems clunky from their literature but I don't know if that's the case in operation). Also does anyone have any iSCSI 'gotchas' ? Is a TOE one of those 'nice to have but not really necessary' things on a modern server or should it be factored into the solution ? Cheers, Raj.

4 4

RE: iSCSI SAN Queries
by Learmonth, Peter 22 Nov '06

22 Nov '06

That page is mostly links to other docs, most of which are external. http://www.netapp.com/library/tr/3250.pdf http://www.netapp.com/library/tr/3338.pdf http://www.netapp.com/library/tr/3357.pdf http://www.netapp.com/library/tr/3323.pdf http://www.netapp.com/library/tr/3376.pdf http://support.microsoft.com/default.aspx?scid=kb;en-us;305547 http://www.netapp.com/library/tr/3302.pdf http://www.netapp.com/library/tr/3280.pdf Share and enjoy! Peter ________________________________ From: Raj Patel [mailto:phigmov@gmail.com] Sent: Wednesday, November 22, 2006 11:24 AM To: toasters(a)mathworks.com Subject: Re: iSCSI SAN Queries >http://now.netapp.com/NOW/knowledge/docs/bpg/san_ipsan/man_admin_iscsi. shtml One slight annoyance is that this site is member only and basic membership doesn't let you get to this page (Cisco has something similar on their site and its just as aggravating). The EMC page did come straight up though. I'll see if I can get the doc out of a local NetApp vendor. Cheers, Raj. On 11/23/06, Julio Calderon <jcalderon(a)agami.com> wrote: Thanks for sharing your findings. Rule of thumb, regardless of what vendor you choose to go with, review their specific best practices Netapp has them here: http://now.netapp.com/NOW/knowledge/docs/bpg/san_ipsan/man_admin_iscsi.s html EMC has them here: http://www.emc.com/techlib/pdf/H1784_Celerra_iSCSI_Solutions_MS_Exch_ldv .pdf Hitachi, I looked but could not find their best practices. My personal recommendation is to review what other vendors have to offer as well. If you do a quick search on the net, you will find many. Best Regards, Julio C.

1 0

RE: iSCSI SAN Queries
by Pichardo, Socrates 22 Nov '06

22 Nov '06

Greetings and salutations, We have deployed, quite successfully, Netapp 3020 with iSCSI for all our Exchange and SQL servers. This solution has been in prod for more than a year now and the issues that we have seen are not directly related to iSCSI. Of course, we did put in place some safe guards that we believe has made the whole solution more reliable like: - Run complete separate network for our iSCSI traffic. - TOE cards for our heavy-duty servers. - Overcapacity on our filer heads. We did this since our iSCSI decision had nothing to do with cost (which at the time of the RFP FC was still significantly more expensive than iSCSI) but with our desire to keep our installation simple and not having to bring a new protocol/environment (FC) under our management. iSCSI has proven to be as fast, reliable and cost effective as we expected it without the steep learning curve of FC. Here are a few findings: - TOE: On our lab we saw a 7%to 12% CPU savings by using them therefore we did not justify them for performance reasons but we ended up using them for SAN booting requirements. - iSCSI Performance: on our lab/RPF we noticed better performance out of FC under heavy load conditions. Under our expected loads (low to average), we did not noticed any significance performance gains on FC over iSCSI. - iSCSI Implementation: We also noticed that a year ago iSCSI looked like an "add-on" to most of the SAN vendors we spoke with. We selected Netapp for their simplicity and easy of management and both have been proven to be true. Regards, SPV ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Julio Calderon Sent: Tuesday, November 21, 2006 9:21 PM To: Raj Patel; toasters(a)mathworks.com Subject: RE: iSCSI SAN Queries Greetings, I've had first hand experience with Netapp 270s / iSCSI on couple of windows host. (past life) Nutshell: SLOW, don't try exchange on it, it kept timing out on me. I've also seen a large client have issues with Lotus notes when using Gfilers 980c and HDS as the backend storage. (this was a poor configuration issue, too many LUNs over a 1 gig connection) I've also tried it for another purposes and I lost my LUN a few times. The only way of getting it back then was by restarting the client! not a fun thing to do when you are running multiple apps on the same server. Suggestion would be, directly attach application servers via GigE across to your iSCSI device. If direct connecting your app servers is not realistic. use VLANs to segregate traffic, dedicate iSCSI traffic to a set of bonded ports. Start slow, make sure that the number of LUNs being provisioned to the app servers are provisioned in a control manner. (meaning, see how your application behaves in a controlled environment before doing any big role out) Monitor performance, use a testing tool (iozone) to see how much performance you would gain or loose from moving away from current solution. Also, consider the advantages and disadvantages an iSCSI solution would bring your environment. (replication, HA, speed, easy of management, backup, etc) TOE ? I think that the purpose of having iSCSI in your environment is to lower the investment of FC (SAN) solution by leveraging your current network infrastructure and current staff expertise so, if you start purchasing the TOE cards, you will in fact start cutting into the savings a properly architected and managed IP SAN (iscsi) solution would offer your company. I have not had the opportunity to try a TOE card nor have I needed it to yet since I am able to reach almost wire speeds on GigE at the moment. Can not comment on the HDS and EMC solutions but I can't believe they would be better than what Netapp has to offer since their main business is in the FC world. Hope this helps guide you! Best Regards, Julio Calderon West Region - Systems Engineer agami <http://www.agami.com/> 1269 Innsbruck Drive <http://maps.google.com/maps?q=1269+Innsbruck+Drive%2CSunnyvale%2CCA+940 89%2CUSA&hl=en> Sunnyvale, CA 94089 USA Work: 408.349.0414 Mobile: 408.394.5638 Fax: 408.349.0393 Email: jcalderon(a)agami.com <mailto:jcalderon@agami.com> IM: juliocus (Yahoo) Want a signature like this? <http://www.linkedin.com/e/sig/2105155/> ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Tuesday, November 21, 2006 11:40 AM To: toasters(a)mathworks.com Subject: iSCSI SAN Queries Hi, We're in the preliminary scoping phase for a low/mid-range iSCSI SAN solution. I was wondering if anyone had any positive or negative real-world experiences with - NetApp 270c HDS AMS200 EMC CX320 At the moment the NetApp is the most familiar to me as I have used one of their NAS boxes before and the simulator provides a pretty good indication of how it works. However the iSCSI seems a bit of a 'bolt-on' and its not clear if it will handle tiered storage as well as the other vendors (then again does it matter?). The HDS & EMC are unknown quantities (other than what I can glean from the web). Any feedback concerning ease of use, expansion, licensing, snapshot mechanism (the EMC seems clunky from their literature but I don't know if that's the case in operation). Also does anyone have any iSCSI 'gotchas' ? Is a TOE one of those 'nice to have but not really necessary' things on a modern server or should it be factored into the solution ? Cheers, Raj. <font size="2">"IMPORTANT NOTICE: The information in this email (and any attachments hereto) is confidential and may be protected by legal privileges and work product immunities. If you are not the intended recipient, you must not use or disseminate the information. Receipt by anyone other than the intended recipient is not a waiver of any attorney-client or work product privilege. If you have received this email in error, please immediately notify me by "Reply" command and permanently delete the original and any copies or printouts thereof. Although this email and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Transatlantic Reinsurance Company or its subsidiaries or affiliates either jointly or severally, for any loss or damage arising in any way from its use."</font>

3 2

RE: iSCSI SAN Queries
by Ledbetter, Charles 22 Nov '06

22 Nov '06

Julio pretty well summed things up. Think of iSCSI as iSAN. It's still a SAN setup and the blocks move over the very same SCSI protocol. Whether it's SCSI over FCP or SCSI over IP makes no real difference. With that in mind, you should treat your iSAN setups with the same consideration as SAN. Dedicated links, VLANs, etc. A TOE card should only be deployed once you know you need one. Start with a standard host NIC and watch your host's CPU. If it's being driven to capacity, a TOE or iSCSI HBA will help. If it's not, you're really just wasting money. -Charles -----Original Message----- From: Julio Calderon [mailto:jcalderon@agami.com] Sent: Tuesday, November 21, 2006 9:21 PM To: Raj Patel; toasters(a)mathworks.com Subject: RE: iSCSI SAN Queries Greetings, I've had first hand experience with Netapp 270s / iSCSI on couple of windows host. (past life) Nutshell: SLOW, don't try exchange on it, it kept timing out on me. I've also seen a large client have issues with Lotus notes when using Gfilers 980c and HDS as the backend storage. (this was a poor configuration issue, too many LUNs over a 1 gig connection) I've also tried it for another purposes and I lost my LUN a few times. The only way of getting it back then was by restarting the client! not a fun thing to do when you are running multiple apps on the same server. Suggestion would be, directly attach application servers via GigE across to your iSCSI device. If direct connecting your app servers is not realistic. use VLANs to segregate traffic, dedicate iSCSI traffic to a set of bonded ports. Start slow, make sure that the number of LUNs being provisioned to the app servers are provisioned in a control manner. (meaning, see how your application behaves in a controlled environment before doing any big role out) Monitor performance, use a testing tool (iozone) to see how much performance you would gain or loose from moving away from current solution. Also, consider the advantages and disadvantages an iSCSI solution would bring your environment. (replication, HA, speed, easy of management, backup, etc) TOE ? I think that the purpose of having iSCSI in your environment is to lower the investment of FC (SAN) solution by leveraging your current network infrastructure and current staff expertise so, if you start purchasing the TOE cards, you will in fact start cutting into the savings a properly architected and managed IP SAN (iscsi) solution would offer your company. I have not had the opportunity to try a TOE card nor have I needed it to yet since I am able to reach almost wire speeds on GigE at the moment. Can not comment on the HDS and EMC solutions but I can't believe they would be better than what Netapp has to offer since their main business is in the FC world. Hope this helps guide you! Best Regards, Julio Calderon West Region - Systems Engineer agami <http://www.agami.com/> 1269 Innsbruck Drive <http://maps.google.com/maps?q=1269+Innsbruck+Drive%2CSunnyvale%2CCA+940 89%2CUSA&hl=en> Sunnyvale, CA 94089 USA Work: 408.349.0414 Mobile: 408.394.5638 Fax: 408.349.0393 Email: jcalderon(a)agami.com IM: juliocus (Yahoo) Want a signature like this? <http://www.linkedin.com/e/sig/2105155/> _____ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Tuesday, November 21, 2006 11:40 AM To: toasters(a)mathworks.com Subject: iSCSI SAN Queries Hi, We're in the preliminary scoping phase for a low/mid-range iSCSI SAN solution. I was wondering if anyone had any positive or negative real-world experiences with - NetApp 270c HDS AMS200 EMC CX320 At the moment the NetApp is the most familiar to me as I have used one of their NAS boxes before and the simulator provides a pretty good indication of how it works. However the iSCSI seems a bit of a 'bolt-on' and its not clear if it will handle tiered storage as well as the other vendors (then again does it matter?). The HDS & EMC are unknown quantities (other than what I can glean from the web). Any feedback concerning ease of use, expansion, licensing, snapshot mechanism (the EMC seems clunky from their literature but I don't know if that's the case in operation). Also does anyone have any iSCSI 'gotchas' ? Is a TOE one of those 'nice to have but not really necessary' things on a modern server or should it be factored into the solution ? Cheers, Raj.

1 0

Problem mounting NFS from a Fedora Core 6 machine
by Gustavo Chaves 22 Nov '06

22 Nov '06

Hello, We have a F760 running Data Ontap 6.4.5. (Pretty old stuff, I know.) The filer has six network interfaces, each one connected to a different network. Recently we started experimenting with a Linux Fedora Core 6 machine. When we try to mount some directory at the filer from this machine, almost always the process hang. A very few times the process succeeds. Investigating the issue using tcpdump I discovered that in the process of mouting, when the filer's portmapper answers to the Linux box, sometimes its answers come from a different network interface and the Linux box doesn't like that. See below the history of an unsuccessful mount. The Linux box (10.202.40.66) tries to mount a directory from filer (10.202.48.109) with the following command and hangs until I kill it: mount filer:/vol/vol0/unix/config /mnt Note that on packet 111 the linux box sends a UDP packet to the filer's portmapper asking for the port number of the MOUNT service. The filer responds with packet 113, which comes from another of its interfaces (10.202.40.23). This response is rejected by the linux box with the ICMP packet number 114. If I try to mount specifying the interface 10.202.40.23, which is in the same network as the client, the mount succeeds. It seems that the portmapper response always tries to go back through the interface in the same network as the client. The problem is that we have some linux clients on networks on which there is no filer interface connected. In these cases it seems to be anyone's guess through which interface the portmapper response will come. We looked for differences in the TCP stack from Fedora Core 5 to 6, because until Core 5 this problem didn't arise, but we couldn't find anything relevant. Since the filer works ok with other Linux and Solaris machines, it seems that something must have been changed in the Linux kernel that made it more strict. We suspected of some firewall rules, but there is no iptables rules configured in the linux box. Has anybody seen this before? Any clue? Thank you. Gustavo. ---------------------------------------------------------------------- No. Time Source Destination Protocol Info 94 1.036676 10.202.40.66 10.202.48.109 TCP 33614 > sunrpc [SYN] Seq=0 Len=0 MSS=1460 TSV=16336999 TSER=0 WS=6 95 1.038301 10.202.48.109 10.202.40.66 TCP sunrpc > 33614 [SYN, ACK] Seq=0 Ack=1 Win=8760 Len=0 MSS=1460 WS=0 TSV=242744087 TSER=16336999 96 1.038319 10.202.40.66 10.202.48.109 TCP 33614 > sunrpc [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=16336999 TSER=242744087 97 1.038385 10.202.40.66 10.202.48.109 Portmap V2 GETPORT Call (Reply In 98) NFS(100003) V:3 TCP 98 1.039907 10.202.48.109 10.202.40.66 Portmap V2 GETPORT Reply (Call In 97) Port:2049 99 1.039914 10.202.40.66 10.202.48.109 TCP 33614 > sunrpc [ACK] Seq=61 Ack=33 Win=5888 Len=0 TSV=16337000 TSER=242744087 100 1.039939 10.202.40.66 10.202.48.109 TCP 33614 > sunrpc [FIN, ACK] Seq=61 Ack=33 Win=5888 Len=0 TSV=16337000 TSER=242744087 101 1.039966 10.202.40.66 10.202.48.109 TCP 34612 > nfs [SYN] Seq=0 Len=0 MSS=1460 TSV=16337000 TSER=0 WS=6 102 1.040592 10.202.48.109 10.202.40.66 TCP sunrpc > 33614 [ACK] Seq=33 Ack=62 Win=8760 Len=0 TSV=242744087 TSER=16337000 103 1.040669 10.202.48.109 10.202.40.66 TCP nfs > 34612 [SYN, ACK] Seq=0 Ack=1 Win=26280 Len=0 MSS=1460 WS=0 TSV=242744087 TSER=16337000 104 1.040677 10.202.40.66 10.202.48.109 TCP 34612 > nfs [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=16337000 TSER=242744087 105 1.040722 10.202.40.66 10.202.48.109 TCP [TCP segment of a reassembled PDU] 106 1.040849 10.202.48.109 10.202.40.66 TCP sunrpc > 33614 [FIN, ACK] Seq=33 Ack=62 Win=8760 Len=0 TSV=242744087 TSER=16337000 107 1.040856 10.202.40.66 10.202.48.109 TCP 33614 > sunrpc [ACK] Seq=62 Ack=34 Win=5888 Len=0 TSV=16337000 TSER=242744087 108 1.041776 10.202.48.109 10.202.40.66 NFS V3 NULL Reply (Call In 105) 109 1.041782 10.202.40.66 10.202.48.109 TCP 34612 > nfs [ACK] Seq=45 Ack=33 Win=5888 Len=0 TSV=16337000 TSER=242744087 110 1.041801 10.202.40.66 10.202.48.109 TCP 34612 > nfs [FIN, ACK] Seq=45 Ack=33 Win=5888 Len=0 TSV=16337000 TSER=242744087 111 1.041847 10.202.40.66 10.202.48.109 Portmap V2 GETPORT Call (Reply In 113) MOUNT(100005) V:3 UDP 112 1.042834 10.202.48.109 10.202.40.66 TCP nfs > 34612 [ACK] Seq=33 Ack=46 Win=26280 Len=0 TSV=242744087 TSER=16337000 113 1.042842 10.202.40.23 10.202.40.66 Portmap V2 GETPORT Reply (Call In 111) Port:4046 114 1.042851 10.202.40.66 10.202.40.23 ICMP Destination unreachable (Port unreachable) 115 1.042854 10.202.48.109 10.202.40.66 TCP nfs > 34612 [FIN, ACK] Seq=33 Ack=46 Win=26280 Len=0 TSV=242744087 TSER=16337000 116 1.042857 10.202.40.66 10.202.48.109 TCP 34612 > nfs [ACK] Seq=46 Ack=34 Win=5888 Len=0 TSV=16337000 TSER=242744087

1 0

RE: iSCSI SAN Queries
by Christopher Mende 21 Nov '06

21 Nov '06

I can't comment much on the other vendors' most current offerings, but iSCSI is anything but bolt-on for the filers. NetApp has the lion's share of iSCSI deployments worldwide. FAS270 (including clusters) running iSCSI configurations supporting Exchange, SQL, and all varieties of other applications are going in all over the place. With a FAS270c, tiered storage (mixing FC & SATA) is possible, with certain conditions. All secondary shelves are either SATA OR FC. From the system configuration guide: Caution 2: While both types of disk shelves can connect to the filer, if there is an existing DiskShelf14mk2 FC connected to the FAS270/FAS270c, then any additional expansion shelf must be a DiskShelf14mk2 FC. If however, you do not have an expansion shelf connected to the FAS270/FAS270c, then you can choose to connect to a loop of DiskShelf14mk2 AT or a loop of DiskShelf14mk2 FC, but not both. The FAS270/FAS270c does not support both types of disk shelves on the same loop. ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Wednesday, 22 November 2006 8:40 a.m. To: toasters(a)mathworks.com Subject: iSCSI SAN Queries Hi, We're in the preliminary scoping phase for a low/mid-range iSCSI SAN solution. I was wondering if anyone had any positive or negative real-world experiences with - NetApp 270c HDS AMS200 EMC CX320 At the moment the NetApp is the most familiar to me as I have used one of their NAS boxes before and the simulator provides a pretty good indication of how it works. However the iSCSI seems a bit of a 'bolt-on' and its not clear if it will handle tiered storage as well as the other vendors (then again does it matter?). The HDS & EMC are unknown quantities (other than what I can glean from the web). Any feedback concerning ease of use, expansion, licensing, snapshot mechanism (the EMC seems clunky from their literature but I don't know if that's the case in operation). Also does anyone have any iSCSI 'gotchas' ? Is a TOE one of those 'nice to have but not really necessary' things on a modern server or should it be factored into the solution ? Cheers, Raj. The information contained in this email is privileged and confidential and intended for the addressee only. If you are not the intended recipient, you are asked to respect that confidentiality and not disclose, copy or make use of its contents. If received in error you are asked to destroy this email and contact the sender immediately. Your assistance is appreciated.

1 0

IBM Branded NetApps
by Raj Patel 19 Nov '06

19 Nov '06

Hi all, A few quick queries - Has anyone had any experience with the IBM Branded NetApp's ? What does the N5200 compare with in the NetApp family ? How does service/support differ from NetApp ? Finally why would anyone go with IBM for a NetApp rather than NetApp direct (is it purely a question of vendor service quality or is there a significant price difference) ? Thanks in advance, Raj.

7 8

RE: Measuring Filer Performance
by Willeke, Jochen 16 Nov '06

16 Nov '06

Hi, i agree that latency is one of the main factors. But sometimes i wonder how to measure the latency the best way. One way i know is to use "tethereal -i [interface] -z rpc,rtt,100003,3" Which gives me an overview of the rtt which is seen by the client. But i am not so sure what to use to see the output from the filer. I know that statit gives me some kind of rtt for each disk. But where do you get a good overall-response-time for your filers from? Those counters shown by Blake (...stat show...) are very good, but i think they only show what the head sees. What is a reasonable value to be added as an "head-overhead". For example: - I run tethereal from a direct-attached client and see a read-rtt of 12ms - I run stat show -i 1 nfsv3:nfs:nfsv3_read_latency and see 3ms - I run a ping from the direct-attached client and get a rtt of 2ms - So i would think that the filer-overhead is ~7ms Do you think this is correct way to go? Best Regards Jochen -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glenn Walker Sent: Thursday, November 16, 2006 4:25 AM To: Blake Golliher; George, Andrew Cc: Rohit; toasters(a)mathworks.com Subject: RE: Measuring Filer Performance I'd agree with the latency standpoint. It's a good metric to determine when the users will start to bitch. The latency might be hidden by the application, but it depends on how well it's written (single threaded vs multi threaded\low concurrency vs high concurrency). Typically, this is all a function of CPU\DISK (and sometimes memory) load. As a result, those are good to watch, and they are a direct result of the overall workload (CIFS\NFS\iSCSI IOPS). The hard part is determining what\why the workload tanks. Most times, users are only going to feel impact if there is a long running event that generates high latency - however, depending on workload, a single spike that lasts only moments (a second or two) can be devastating: this is where knowing your environment comes in handy. Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, November 15, 2006 7:08 PM To: George, Andrew Cc: Rohit; toasters(a)mathworks.com Subject: Re: Measuring Filer Performance Latency is primary for what most of what my filers are doing. Usually watched via nfs_hist (being sure to -z the stats), or using stats all wrapped up in a shell script. rsh $FILER 'priv set -q diag ; stats show -i $INTER -n $SAMPLES volume:$VOL:avg_latency volume:$VOL:total_ops volume:$VOL:read_latency volume:$VOL:read_ops volume:$VOL:write_latency volume:$VOL:write_ops volume:$VOL:other_latency volume:$VOL:other_ops volume:$VOL:nfs_read_latency volume:$VOL:nfs_read_ops volume:$VOL:nfs_write_latency volume:$VOL:nfs_write_ops volume:$VOL:nfs_other_latency volume:$VOL:nfs_other_ops' nfsv3:nfs:nfsv3_avg_op_latency nfsv3:nfs:nfsv3_op_percent nfsv3:nfs:nfsv3_write_ops nfsv3:nfs:nfsv3_write_latency nfsv3:nfs:nfsv3_read_ops nfsv3:nfs:nfsv3_read_latency nfsv3:nfs:nfsv3_ops' It's good to note that while stats show averages, nfs_hist shows a histogram, and can spot the occasional very high latency operation. Good to watch. I just poked around, there is a cifs component for stats, and it shows some interesting metrics the cifs users might want to watch. filer01*> stats show cifsdomain cifs cifs:cifs:cifs_ops:2658/s cifs:cifs:cifs_op_count: GetAttr: 1296 Read: 989 Write: 0 Lock: 1 Open/Close: 23 Directory: 0 Other: 335 cifs:cifs:cifs_op_pct: GetAttr: 48% Read: 37% Write: 0% Lock: 0% Open/Close: 0% Directory: 0% Other: 12% cifs:cifs:cifs_latency:3.27ms cifsdomain:10.0.0.3:netlogon_latency:9.50ms cifsdomain:10.0.0.3:netlogon_latency_base:6 cifsdomain:10.0.0.3:lsa_latency:0ms cifsdomain:10.0.0.3:lsa_latency_base:0 cifsdomain:10.0.0.3:samr_latency:0ms cifsdomain:10.0.0.3:samr_latency_base:0 cifsdomain:10.0.0.2:netlogon_latency:2.00ms cifsdomain:10.0.0.2:netlogon_latency_base:1 cifsdomain:10.0.0.2:lsa_latency:0ms cifsdomain:10.0.0.2:lsa_latency_base:0 cifsdomain:10.0.0.2:samr_latency:0ms cifsdomain:10.0.0.2:samr_latency_base:0 cifsdomain:10.0.0.1:netlogon_latency:0ms cifsdomain:10.0.0.1:netlogon_latency_base:0 cifsdomain:10.0.0.1:lsa_latency:0ms cifsdomain:10.0.0.1:lsa_latency_base:0 cifsdomain:10.0.0.1:samr_latency:0ms cifsdomain:10.0.0.1:samr_latency_base:0 There's also smb_hist. That's from a 960 running 6.5.6PsomethingDsomething.... -Blake On 11/15/06, George, Andrew <georgea(a)anz.com> wrote: > We only use our filers for CIFS (so this may be completely inapplicable > to you) > > 1. Cache age! Mainly because when I have seen filers that are worked to > death, that's the value that tends to react to it (though CP there are a > lot of secondary metrics to support that). > > 2. As mentioned by someone else...the magic metric is how badly the > users are bitching. That being a little "rubbery" we also check cache > age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic > to determine if anything's bottlenecking. > > -----Original Message----- > From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] > On Behalf Of Rohit > Sent: Wednesday, 15 November 2006 9:59 PM > To: toasters(a)mathworks.com > Subject: Measuring Filer Performance > > Hi Folks > > I had some open questions and i hope you'll help me answer them based on > your experience. > > 1. If you just had a single metric to determine filer performance what > would it be ? > Would it be CPU usage, throughput, latency, IOPS..or anything else ? > > 2. At your place, what metrics do you use to determine if a filer is > still good enough not to worry about ? > > thanks > rohit > > ------------------------------------------------------------------------ > -- > Refinance & Home Equity Loan Quotes > Low credit score? Old Merchants can help. Get a quote in 2 easy steps! > http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ > > > > > "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication." >

1 0

RE: Measuring Filer Performance
by Glenn Walker 16 Nov '06

16 Nov '06

I'd agree with the latency standpoint. It's a good metric to determine when the users will start to bitch. The latency might be hidden by the application, but it depends on how well it's written (single threaded vs multi threaded\low concurrency vs high concurrency). Typically, this is all a function of CPU\DISK (and sometimes memory) load. As a result, those are good to watch, and they are a direct result of the overall workload (CIFS\NFS\iSCSI IOPS). The hard part is determining what\why the workload tanks. Most times, users are only going to feel impact if there is a long running event that generates high latency - however, depending on workload, a single spike that lasts only moments (a second or two) can be devastating: this is where knowing your environment comes in handy. Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, November 15, 2006 7:08 PM To: George, Andrew Cc: Rohit; toasters(a)mathworks.com Subject: Re: Measuring Filer Performance Latency is primary for what most of what my filers are doing. Usually watched via nfs_hist (being sure to -z the stats), or using stats all wrapped up in a shell script. rsh $FILER 'priv set -q diag ; stats show -i $INTER -n $SAMPLES volume:$VOL:avg_latency volume:$VOL:total_ops volume:$VOL:read_latency volume:$VOL:read_ops volume:$VOL:write_latency volume:$VOL:write_ops volume:$VOL:other_latency volume:$VOL:other_ops volume:$VOL:nfs_read_latency volume:$VOL:nfs_read_ops volume:$VOL:nfs_write_latency volume:$VOL:nfs_write_ops volume:$VOL:nfs_other_latency volume:$VOL:nfs_other_ops' nfsv3:nfs:nfsv3_avg_op_latency nfsv3:nfs:nfsv3_op_percent nfsv3:nfs:nfsv3_write_ops nfsv3:nfs:nfsv3_write_latency nfsv3:nfs:nfsv3_read_ops nfsv3:nfs:nfsv3_read_latency nfsv3:nfs:nfsv3_ops' It's good to note that while stats show averages, nfs_hist shows a histogram, and can spot the occasional very high latency operation. Good to watch. I just poked around, there is a cifs component for stats, and it shows some interesting metrics the cifs users might want to watch. filer01*> stats show cifsdomain cifs cifs:cifs:cifs_ops:2658/s cifs:cifs:cifs_op_count: GetAttr: 1296 Read: 989 Write: 0 Lock: 1 Open/Close: 23 Directory: 0 Other: 335 cifs:cifs:cifs_op_pct: GetAttr: 48% Read: 37% Write: 0% Lock: 0% Open/Close: 0% Directory: 0% Other: 12% cifs:cifs:cifs_latency:3.27ms cifsdomain:10.0.0.3:netlogon_latency:9.50ms cifsdomain:10.0.0.3:netlogon_latency_base:6 cifsdomain:10.0.0.3:lsa_latency:0ms cifsdomain:10.0.0.3:lsa_latency_base:0 cifsdomain:10.0.0.3:samr_latency:0ms cifsdomain:10.0.0.3:samr_latency_base:0 cifsdomain:10.0.0.2:netlogon_latency:2.00ms cifsdomain:10.0.0.2:netlogon_latency_base:1 cifsdomain:10.0.0.2:lsa_latency:0ms cifsdomain:10.0.0.2:lsa_latency_base:0 cifsdomain:10.0.0.2:samr_latency:0ms cifsdomain:10.0.0.2:samr_latency_base:0 cifsdomain:10.0.0.1:netlogon_latency:0ms cifsdomain:10.0.0.1:netlogon_latency_base:0 cifsdomain:10.0.0.1:lsa_latency:0ms cifsdomain:10.0.0.1:lsa_latency_base:0 cifsdomain:10.0.0.1:samr_latency:0ms cifsdomain:10.0.0.1:samr_latency_base:0 There's also smb_hist. That's from a 960 running 6.5.6PsomethingDsomething.... -Blake On 11/15/06, George, Andrew <georgea(a)anz.com> wrote: > We only use our filers for CIFS (so this may be completely inapplicable > to you) > > 1. Cache age! Mainly because when I have seen filers that are worked to > death, that's the value that tends to react to it (though CP there are a > lot of secondary metrics to support that). > > 2. As mentioned by someone else...the magic metric is how badly the > users are bitching. That being a little "rubbery" we also check cache > age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic > to determine if anything's bottlenecking. > > -----Original Message----- > From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] > On Behalf Of Rohit > Sent: Wednesday, 15 November 2006 9:59 PM > To: toasters(a)mathworks.com > Subject: Measuring Filer Performance > > Hi Folks > > I had some open questions and i hope you'll help me answer them based on > your experience. > > 1. If you just had a single metric to determine filer performance what > would it be ? > Would it be CPU usage, throughput, latency, IOPS..or anything else ? > > 2. At your place, what metrics do you use to determine if a filer is > still good enough not to worry about ? > > thanks > rohit > > ------------------------------------------------------------------------ > -- > Refinance & Home Equity Loan Quotes > Low credit score? Old Merchants can help. Get a quote in 2 easy steps! > http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ > > > > > "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication." >

2 1

Measuring Filer Performance
by Rohit 16 Nov '06

16 Nov '06

Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

3 2

RE: Measuring Filer Performance
by George, Andrew 16 Nov '06

16 Nov '06

We only use our filers for CIFS (so this may be completely inapplicable to you) 1. Cache age! Mainly because when I have seen filers that are worked to death, that's the value that tends to react to it (though CP there are a lot of secondary metrics to support that). 2. As mentioned by someone else...the magic metric is how badly the users are bitching. That being a little "rubbery" we also check cache age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic to determine if anything's bottlenecking. -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Rohit Sent: Wednesday, 15 November 2006 9:59 PM To: toasters(a)mathworks.com Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit ------------------------------------------------------------------------ -- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

2 1

Re: Measuring Filer Performance
by Reinoud Reynders 15 Nov '06

15 Nov '06

Hi Rohit, My view is that your application performance is the most important. This is what our users see and feel. The problem is that for every application, the impact of the storage on the performance is different. For our Databases, responstime is the key. So for those applications we measure the latency. For our PACS (digital radiology) is Mb/s more important. So we try to measure the application performance and the key infrastructure elements of that application (like latency). On filer level, we monitor the CPU to give a general idee of the load of the filer. Best regards, Reinoud Reynders IT manager Infrastructure & Operations Leuven University Hospital -----Original Message----- From: owner-toasters(a)mathworks.com <owner-toasters(a)mathworks.com> To: toasters(a)mathworks.com <toasters(a)mathworks.com> Sent: Wed Nov 15 11:58:39 2006 Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

1 0

RE: IBM Branded NetApps
by Julio Calderon 15 Nov '06

15 Nov '06

I've been in the same spot having to deal with a vendorA when vendorA hd to get support from VendorB and I could never get quick and advance help from anyone! Not a fun place to be when you need help. Good luck -----Original Message----- From: Zayas, Russell [mailto:rzayas@qualcomm.com] Sent: Tue Nov 14 16:19:46 2006 To: Toasters; Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps Does that mean that support is geared to handle tough issues? Or do they pipe you into NetApp support at that point? Curious, -Russ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Toasters Sent: Tuesday, November 14, 2006 12:41 PM To: Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps We just went with a few IBM boxes here. We have 40 or so Netapp boxes. The determining factor for us was the price. They undercut by well over 100k on what we bought. The support structure is not the same as Netapp. If you have never experienced IBM support before then I suggest you do before you decided to go the way of IBM/NAS. If your like us and don't need the "what is a snapmirror" type of support and just call when you need parts or the box is panicing then you may want to consider them. E- ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Tuesday, November 14, 2006 2:03 PM To: toasters(a)mathworks.com Subject: IBM Branded NetApps Hi all, A few quick queries - Has anyone had any experience with the IBM Branded NetApp's ? What does the N5200 compare with in the NetApp family ? How does service/support differ from NetApp ? Finally why would anyone go with IBM for a NetApp rather than NetApp direct (is it purely a question of vendor service quality or is there a significant price difference) ? Thanks in advance, Raj.

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Usally less than 1 second, at least a couple of times during the snapmirror job It is a Volume snapmirror Ontap 7.1 -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 13:32 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: Macintosh users can't see network drive
by Glenn Walker 14 Nov '06

14 Nov '06

You might want to capture a quick packet trace on either the MAC or NetApp side (tcpdump on mac, I believe and 'pktt start all -d <directory>' on the Filer - be sure to perform 'pktt stop all' when done with the test). MAC has a notoriously horrible CIFS client - I assume you're using OSX? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Maglinger, Paul Sent: Monday, November 13, 2006 4:31 PM To: NetApps list server Subject: FW: Macintosh users can't see network drive Running Version 6.4.5 on a 740 filer. For some reason this morning our Macintosh users cannot see the "public" CIFS drive on our NetApps filer. Because this is a "public" drive, it should be open all of our users. Everything worked fine until this morning and now they cannot see the filer in their Server folder. We've connected to a Windows server using smb://ip_address/shared_directory, but when we try to connect to the filer in the same manner we never get a logon prompt. No pretending here, I don't know diddily-squat about Mac computers. Can someone give advice on how to even start troubleshooting this? ________________________________ Paul Maglinger, A+, CA, CCA, CET, MCSE Systems Administrator Shoe Carnival Inc. (812)867-4674 pmaglinger(a)scvl.com ________________________________

1 0

FW: Macintosh users can't see network drive
by Maglinger, Paul 13 Nov '06

13 Nov '06

Running Version 6.4.5 on a 740 filer. For some reason this morning our Macintosh users cannot see the "public" CIFS drive on our NetApps filer. Because this is a "public" drive, it should be open all of our users. Everything worked fine until this morning and now they cannot see the filer in their Server folder. We've connected to a Windows server using smb://ip_address/shared_directory, but when we try to connect to the filer in the same manner we never get a logon prompt. No pretending here, I don't know diddily-squat about Mac computers. Can someone give advice on how to even start troubleshooting this? ________________________________ Paul Maglinger, A+, CA, CCA, CET, MCSE Systems Administrator Shoe Carnival Inc. (812)867-4674 pmaglinger(a)scvl.com ________________________________

1 0

ndmpcopy, jndmpcopy and multiple level 1 transfers
by Paul Letta 13 Nov '06

13 Nov '06

I am getting ready to do a big data migration. I am planning on doing an initial level 0, and then multiple level 1's leading up to the big day. I know the that native ndmpcopy in OnTap 7.1 can only do one level 1, and one level 2 for each level 0. But I have read that if you get ndmpcopy or jnmdpcopy from ndmp.org that it supports multiple level 1 dumps based on the same level 0. But I am getting the same errors I got with the native ndmpcopy when I use ndmpcopy and jndmpcopy from a unix host. I just grabbed the latest copies for ndmp.org yesterday. > java jndmpcopy.JNdmpcopy -version java version "1.4.2" gcj (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3) So I did a level 0, and then one level 1 that both worked just fine. Then I re-ran the level 1 with the same command, and get the errors below. This is the same error I get if I try this with the native ndmpcopy command on the filer itself. I am moving about 1TB of data (about 16M files) off of 1 traditional volume onto multiple new flex vols on the same filer. If what I'm doing is flat out wrong, what would others suggest ? tar wouldn't give me the incrementals and rsync would take a long time with this number of files. Suggestions ? Thanks, Paul > java jndmpcopy.JNdmpcopy filer:/vol/vol0/source filer:/vol/destination -sa xxxxxxx -da xxxxxxx -level 1 Nov 09 03:53:25 [ndmpd:0]: Now waiting for source and destination to finish. SrcFiler: Nov 09 03:53:25 [ndmpd:0]: DUMP: creating "/vol/vol0/../snapshot_for_backup.153" snapshot. SrcFiler: Nov 09 03:53:26 [ndmpd:0]: DUMP: Using subtree dump SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of this level 1 dump: Thu Nov 9 15:53:25 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of last level 0 dump: Thu Nov 9 15:48:10 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Dumping /vol/vol0/source to NDMP connection SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: mapping (Pass I)[regular files] SrcFiler: Nov 09 03:53:43 [ndmpd:0]: DUMP: mapping (Pass II)[directories] SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: estimated 18 KB. SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: dumping (Pass III) [directories] DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Thu Nov 9 15:54:13 2006: Begin incremental restore DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This incremental dump does not follow the previous dump that you have restored. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This dump is based on Thu Nov 9 15:48:10 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: It should be based on Thu Nov 9 15:50:28 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Could not use tape. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: RESTORE IS ABORTED Nov 09 03:54:13 [ndmpd:0]: DstFiler just halted. Nov 09 03:54:13 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. SrcFiler: Nov 09 03:54:13 [ndmpd:0]: DUMP: dumping (Pass IV) [regular files] SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: Network communication error SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: DUMP IS ABORTED SrcFiler: Nov 09 03:54:18 [ndmpd:0]: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.153" snapshot. Nov 09 03:54:19 [ndmpd:0]: SrcFiler just halted. Nov 09 03:54:19 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. Nov 09 03:54:19 [ndmpd:0]: jndmpcopy finished -- Paul Letta Thomas Jefferson National Accelerator Facility Computer Center Unix System Administrator EMail: letta(a)jlab.org Phone: (757)269-5106

4 3

Re: ndmpcopy, jndmpcopy and multiple level 1 transfers
by Jerry 13 Nov '06

13 Nov '06

Never underestimate the power of rsync. Here is what I did to move about 70mil files (lots of tiny files). With lots of tiny files, NDMCOPY spends a lot of time before it really transmits anything. In some cases of 10+ mil files on a busy filer I was waiting 4+ hours for files to begin copying (actual data, not just writing out sparse files for inodes). I found that if I ran a level 0 ndmpcopy on the weekend or during your off hours you can let it beat up the filer a little bit (and it will beat up both CPU and Disk util). Then, during the week run rsync's with --delete to keep things nicely in sync. This is something ndmpcopy can't do (I think).... if you do a level 0 + level 1 you won't get deletes. Doing the "catchup rsync's" gives you a good judge of timing on how long it will take you when you schedule your real outage. I recommend using rsync 2.6.6 or higher as it has significant memory efficiency compared to older versions. I used a 4 CPU Sun V440 to move a lot of data with rsync. Running a single process with a 16 million file list consumes as much as 8GB ram. I almost never found myself CPU bound and I found that CPU on the filer was much lighter than if I was doing an ndmpcopy. All of this depends on your rate of change for files, but overall I think rsync was faster for me after that initial copy. Since rsync wasn't using all my CPU, I broke it into an rsync for each top level directory, as many as 70 rsync processes in some of my migrations at the same time. This allowed me to sync up 16 million files in < 20 minutes in some cases. I suspect the level 1 ndmpcopy would be slower than that, but I'm not 100% sure. At the time of cutover, change the export to be read-only to block people from making change. This doesn't work if you have a subdirectory that is hitting a different export line. In this case I would do a second rsync and see what files were still changing and hunt down apps writing to that volume still. I'd be curious to know which method you end up going with and how it wo Original Message ---- From: Darren Dunham <ddunham(a)taos.com> To: toasters(a)mathworks.com Sent: Friday, November 10, 2006 12:37:19 PM Subject: Re: ndmpcopy, jndmpcopy and multiple level 1 transfers > > I am getting ready to do a big data migration. I am planning on doing > > an initial level 0, and then multiple level 1's leading up to the big day. > > > > It doesn't make sense to me why you would do multiple level 1 copies > because a level 1 does not copy files that have changed since the > previous level 1. To do this you must use a level 2 copy. I agree with the spirt, but I would rephrase it. :-) A level 1 *does* copy all the files that have changed since the previous level 1. It's just that they are a subset of files modified since the level 0 which are copied as well. So running a level 1 on the last day should take about the same amount of time whether any other level 1 copies had been run previously or not. Agreed on the rest about 0/1/2 ndmpcopy, but for flexibility I do like rsync as an alternative. It's just how much power you can toss at it and how long you can wait for the file list when you've got millions of files. Remember you don't have to move everything in one rsync command. Espcially if you have multiple clients, it may may sense to split it up into a few separate subtrees. -- Darren Dunham ddunham(a)taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. > ____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index

3 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters