toasters November 2006

toasters@lists.teaparty.net

53 participants
50 discussions

RE: Measuring Filer Performance
by Glenn Walker 16 Nov '06

16 Nov '06

I'd agree with the latency standpoint. It's a good metric to determine when the users will start to bitch. The latency might be hidden by the application, but it depends on how well it's written (single threaded vs multi threaded\low concurrency vs high concurrency). Typically, this is all a function of CPU\DISK (and sometimes memory) load. As a result, those are good to watch, and they are a direct result of the overall workload (CIFS\NFS\iSCSI IOPS). The hard part is determining what\why the workload tanks. Most times, users are only going to feel impact if there is a long running event that generates high latency - however, depending on workload, a single spike that lasts only moments (a second or two) can be devastating: this is where knowing your environment comes in handy. Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, November 15, 2006 7:08 PM To: George, Andrew Cc: Rohit; toasters(a)mathworks.com Subject: Re: Measuring Filer Performance Latency is primary for what most of what my filers are doing. Usually watched via nfs_hist (being sure to -z the stats), or using stats all wrapped up in a shell script. rsh $FILER 'priv set -q diag ; stats show -i $INTER -n $SAMPLES volume:$VOL:avg_latency volume:$VOL:total_ops volume:$VOL:read_latency volume:$VOL:read_ops volume:$VOL:write_latency volume:$VOL:write_ops volume:$VOL:other_latency volume:$VOL:other_ops volume:$VOL:nfs_read_latency volume:$VOL:nfs_read_ops volume:$VOL:nfs_write_latency volume:$VOL:nfs_write_ops volume:$VOL:nfs_other_latency volume:$VOL:nfs_other_ops' nfsv3:nfs:nfsv3_avg_op_latency nfsv3:nfs:nfsv3_op_percent nfsv3:nfs:nfsv3_write_ops nfsv3:nfs:nfsv3_write_latency nfsv3:nfs:nfsv3_read_ops nfsv3:nfs:nfsv3_read_latency nfsv3:nfs:nfsv3_ops' It's good to note that while stats show averages, nfs_hist shows a histogram, and can spot the occasional very high latency operation. Good to watch. I just poked around, there is a cifs component for stats, and it shows some interesting metrics the cifs users might want to watch. filer01*> stats show cifsdomain cifs cifs:cifs:cifs_ops:2658/s cifs:cifs:cifs_op_count: GetAttr: 1296 Read: 989 Write: 0 Lock: 1 Open/Close: 23 Directory: 0 Other: 335 cifs:cifs:cifs_op_pct: GetAttr: 48% Read: 37% Write: 0% Lock: 0% Open/Close: 0% Directory: 0% Other: 12% cifs:cifs:cifs_latency:3.27ms cifsdomain:10.0.0.3:netlogon_latency:9.50ms cifsdomain:10.0.0.3:netlogon_latency_base:6 cifsdomain:10.0.0.3:lsa_latency:0ms cifsdomain:10.0.0.3:lsa_latency_base:0 cifsdomain:10.0.0.3:samr_latency:0ms cifsdomain:10.0.0.3:samr_latency_base:0 cifsdomain:10.0.0.2:netlogon_latency:2.00ms cifsdomain:10.0.0.2:netlogon_latency_base:1 cifsdomain:10.0.0.2:lsa_latency:0ms cifsdomain:10.0.0.2:lsa_latency_base:0 cifsdomain:10.0.0.2:samr_latency:0ms cifsdomain:10.0.0.2:samr_latency_base:0 cifsdomain:10.0.0.1:netlogon_latency:0ms cifsdomain:10.0.0.1:netlogon_latency_base:0 cifsdomain:10.0.0.1:lsa_latency:0ms cifsdomain:10.0.0.1:lsa_latency_base:0 cifsdomain:10.0.0.1:samr_latency:0ms cifsdomain:10.0.0.1:samr_latency_base:0 There's also smb_hist. That's from a 960 running 6.5.6PsomethingDsomething.... -Blake On 11/15/06, George, Andrew <georgea(a)anz.com> wrote: > We only use our filers for CIFS (so this may be completely inapplicable > to you) > > 1. Cache age! Mainly because when I have seen filers that are worked to > death, that's the value that tends to react to it (though CP there are a > lot of secondary metrics to support that). > > 2. As mentioned by someone else...the magic metric is how badly the > users are bitching. That being a little "rubbery" we also check cache > age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic > to determine if anything's bottlenecking. > > -----Original Message----- > From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] > On Behalf Of Rohit > Sent: Wednesday, 15 November 2006 9:59 PM > To: toasters(a)mathworks.com > Subject: Measuring Filer Performance > > Hi Folks > > I had some open questions and i hope you'll help me answer them based on > your experience. > > 1. If you just had a single metric to determine filer performance what > would it be ? > Would it be CPU usage, throughput, latency, IOPS..or anything else ? > > 2. At your place, what metrics do you use to determine if a filer is > still good enough not to worry about ? > > thanks > rohit > > ------------------------------------------------------------------------ > -- > Refinance & Home Equity Loan Quotes > Low credit score? Old Merchants can help. Get a quote in 2 easy steps! > http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ > > > > > "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication." >

2 1

Measuring Filer Performance
by Rohit 16 Nov '06

16 Nov '06

Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

3 2

RE: Measuring Filer Performance
by George, Andrew 16 Nov '06

16 Nov '06

We only use our filers for CIFS (so this may be completely inapplicable to you) 1. Cache age! Mainly because when I have seen filers that are worked to death, that's the value that tends to react to it (though CP there are a lot of secondary metrics to support that). 2. As mentioned by someone else...the magic metric is how badly the users are bitching. That being a little "rubbery" we also check cache age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic to determine if anything's bottlenecking. -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Rohit Sent: Wednesday, 15 November 2006 9:59 PM To: toasters(a)mathworks.com Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit ------------------------------------------------------------------------ -- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

2 1

Re: Measuring Filer Performance
by Reinoud Reynders 15 Nov '06

15 Nov '06

Hi Rohit, My view is that your application performance is the most important. This is what our users see and feel. The problem is that for every application, the impact of the storage on the performance is different. For our Databases, responstime is the key. So for those applications we measure the latency. For our PACS (digital radiology) is Mb/s more important. So we try to measure the application performance and the key infrastructure elements of that application (like latency). On filer level, we monitor the CPU to give a general idee of the load of the filer. Best regards, Reinoud Reynders IT manager Infrastructure & Operations Leuven University Hospital -----Original Message----- From: owner-toasters(a)mathworks.com <owner-toasters(a)mathworks.com> To: toasters(a)mathworks.com <toasters(a)mathworks.com> Sent: Wed Nov 15 11:58:39 2006 Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

1 0

RE: IBM Branded NetApps
by Julio Calderon 15 Nov '06

15 Nov '06

I've been in the same spot having to deal with a vendorA when vendorA hd to get support from VendorB and I could never get quick and advance help from anyone! Not a fun place to be when you need help. Good luck -----Original Message----- From: Zayas, Russell [mailto:rzayas@qualcomm.com] Sent: Tue Nov 14 16:19:46 2006 To: Toasters; Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps Does that mean that support is geared to handle tough issues? Or do they pipe you into NetApp support at that point? Curious, -Russ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Toasters Sent: Tuesday, November 14, 2006 12:41 PM To: Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps We just went with a few IBM boxes here. We have 40 or so Netapp boxes. The determining factor for us was the price. They undercut by well over 100k on what we bought. The support structure is not the same as Netapp. If you have never experienced IBM support before then I suggest you do before you decided to go the way of IBM/NAS. If your like us and don't need the "what is a snapmirror" type of support and just call when you need parts or the box is panicing then you may want to consider them. E- ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Tuesday, November 14, 2006 2:03 PM To: toasters(a)mathworks.com Subject: IBM Branded NetApps Hi all, A few quick queries - Has anyone had any experience with the IBM Branded NetApp's ? What does the N5200 compare with in the NetApp family ? How does service/support differ from NetApp ? Finally why would anyone go with IBM for a NetApp rather than NetApp direct (is it purely a question of vendor service quality or is there a significant price difference) ? Thanks in advance, Raj.

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Usally less than 1 second, at least a couple of times during the snapmirror job It is a Volume snapmirror Ontap 7.1 -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 13:32 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2006