I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?

----- Original Message ----- From: "Andrey Borzenkov" andrey.borzenkov@ts.fujitsu.com To: "Randy Rue" rrue@fhcrc.org, toasters@teaparty.net Sent: Wednesday, February 1, 2012 7:33:02 PM Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] On Behalf Of Randy Rue [rrue@fhcrc.org] Sent: Thursday, February 02, 2012 02:33 To: toasters@teaparty.net Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not specifically a NetApp question but if nothing else, the tool I'm using appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and uploading the results to a timestamped log file served on a web server so I can reach them. I have a script that parses those pages, takes the last timestamped entry from each file (one file for each client), and tallies up a) the average RTT time for all mounts to the vFiler and b) the worst single instance.

Now that I have it all working, the numbers don't believe like I think they should. The clients are adding an entry to each file every minute. I can see that from the timestamps before each entry. But the numbers don't change. Or they change by a few hundredths of a ms. My gather script returns an average that varies only by .01ms or so, and my "worst" figure has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Good question. I'm trying to resolve the concerns of a customer that is having critical performance issues they believe to be the fault of the filer. The filer is indeed struggling but from the perspective of the volumes on the vFiler they own, latencies are generally pretty low. They've agreed to also let us run some monitoring on a few of their impacted systems so we can have some indication of when they're struggling.

----- Original Message -----

From: "Jeff Mohler" speedtoys.racing@gmail.com To: "Randy Rue" rrue@fhcrc.org Cc: "Andrey Borzenkov" andrey.borzenkov@ts.fujitsu.com, toasters@teaparty.net Sent: Wednesday, February 1, 2012 8:51:21 PM Subject: Re: nfsiostat and client NFS connections

Hmm...what reason would you expect the line latency to be _different_ from host to host, to the same filer?

On Wed, Feb 1, 2012 at 11:30 PM, Randy Rue < rrue@fhcrc.org > wrote:

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?

----- Original Message ----- From: "Andrey Borzenkov" < andrey.borzenkov@ts.fujitsu.com > To: "Randy Rue" < rrue@fhcrc.org >, toasters@teaparty.net Sent: Wednesday, February 1, 2012 7:33:02 PM Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________ From: toasters-bounces@teaparty.net [ toasters-bounces@teaparty.net ] On Behalf Of Randy Rue [ rrue@fhcrc.org ] Sent: Thursday, February 02, 2012 02:33 To: toasters@teaparty.net Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not specifically a NetApp question but if nothing else, the tool I'm using appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and uploading the results to a timestamped log file served on a web server so I can reach them. I have a script that parses those pages, takes the last timestamped entry from each file (one file for each client), and tallies up a) the average RTT time for all mounts to the vFiler and b) the worst single instance.

Now that I have it all working, the numbers don't believe like I think they should. The clients are adding an entry to each file every minute. I can see that from the timestamps before each entry. But the numbers don't change. Or they change by a few hundredths of a ms. My gather script returns an average that varies only by .01ms or so, and my "worst" figure has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

I started with nfs-iostat.py, Realized that nfsiostat that ships with CentOS seems to be the same python script. Output is identical and if you diff the two scripts they're almost identical.

Under the impression that RTT (round trip time) is the same as latency.

But in either case the RTT figures given are averaged since the last reboot if you run it with no arguments, and if you run it with arguments of "1 1" it's not clear what you're actually getting.

I'm looking for something I can run once a minute that will tell me the latency of the NFS mounts in real time.

----- Original Message ----- From: "Andrey Borzenkov" andrey.borzenkov@ts.fujitsu.com To: "Randy Rue" rrue@fhcrc.org Cc: toasters@teaparty.net Sent: Wednesday, February 1, 2012 10:08:14 PM Subject: RE: nfsiostat and client NFS connections

Hmm ... but nfsiostat does not really display latency, unless you have some customized version. You want nfs-iostat.py for it: http://communities.netapp.com/thread/13445

--- With best regards

Andrey Borzenkov Senior system engineer Service operations

-----Original Message----- From: Randy Rue [mailto:rrue@fhcrc.org] Sent: Thursday, February 02, 2012 8:31 AM To: Borzenkov, Andrey Cc: toasters@teaparty.net Subject: Re: nfsiostat and client NFS connections

I appreciate your feedback.

I'm attempting to get some real-time view of current latency for NFS mounts, but have figured out that running nfsiostat returns the average since boot. Running it with multiple iterations in one call seems to return transient data, but running a single 1 second iteration doesn't.

Our ultimate goal is to capture instantaneous latency figures from several clients, post the average of those latencies and the single worst figure into an RRD database, and give a graph showing average and worst case. Can you suggest a way to do this with nfsiostat, or a better way with some other tool?

----- Original Message ----- From: "Andrey Borzenkov" andrey.borzenkov@ts.fujitsu.com To: "Randy Rue" rrue@fhcrc.org, toasters@teaparty.net Sent: Wednesday, February 1, 2012 7:33:02 PM Subject: RE: nfsiostat and client NFS connections

You do not say how you process results nor how you run nfsiostat.

nfsiostat (just like iostat, vmstat, ...) reports delta for measurement interval. The very first line is always totals since kernel boot. Usually nfsiostat is run with interval argument ("nfsiostat 60" will give you stats every minute). If you invoke it every minute you have to compute delta yourself.

There are other ways to collect information in form easier to postprocess than nfsiostat free form text output. Sadc + sadf to produce CSV, XML or other formats (which are part of the same sysstat package), collectl (can maintain RRD database) etc.

________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] On Behalf Of Randy Rue [rrue@fhcrc.org] Sent: Thursday, February 02, 2012 02:33 To: toasters@teaparty.net Subject: nfsiostat and client NFS connections

Hello All,

Trying to troubleshoot performance problems on our v3170. This is not specifically a NetApp question but if nothing else, the tool I'm using appears to have been written by a NetApp engineer, does that count?

Running nfsiostat on a handful of impacted clients on another network and uploading the results to a timestamped log file served on a web server so I can reach them. I have a script that parses those pages, takes the last timestamped entry from each file (one file for each client), and tallies up a) the average RTT time for all mounts to the vFiler and b) the worst single instance.

Now that I have it all working, the numbers don't believe like I think they should. The clients are adding an entry to each file every minute. I can see that from the timestamps before each entry. But the numbers don't change. Or they change by a few hundredths of a ms. My gather script returns an average that varies only by .01ms or so, and my "worst" figure has been steady for an hour now.

We're running nfsiostat with no arguments. Am I getting real time readings or some average? If average, since when? Is there a way to get real-time?

Hope to hear from you,

Randy Rue

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

On 2012-2-2 8:28 , Borzenkov, Andrey wrote:

I am not aware offhand of any other tool that would show NFS latency, unfortunately.

There's "stats show volume:*:avg_latency" on the filer itself, for example (and several other latency counters, both under 'volume' and under 'nfsv3', see "stats list counters nfsv3" and "stats list counters volume").

Of course, that ignores any network latency to the clients, but it might still be a useful thing to measure. The same values are also available via the API, but not via SNMP.

Been away from this issue for a little while.

Took a look at "stats list counters nfsv3" and "stats show nfsv3."

Looks like exactly what I'm looking for but it's showing for either the root filer or the total physical filer. Any way to show it for a specific vFiler?

Randy

-----Original Message----- From: Borzenkov, Andrey [mailto:andrey.borzenkov@ts.fujitsu.com] Sent: Thursday, February 02, 2012 2:39 AM To: Jan-Pieter Cornet Cc: Randy Rue; toasters@teaparty.net Subject: RE: nfsiostat and client NFS connections

I could not find information what exactly is measured on filer. I.e. does it show total time from receiving request on LAN port until response is sent out of LAN port or there may be extra delay e.g. due to packet queuing in transport layer after response has been submitted.

So it is definitely useful for trend analysis, but as I understand here the problem is to find out exact place that incurs delays.

--- With best regards

Andrey Borzenkov Senior system engineer Service operations

-----Original Message----- From: Jan-Pieter Cornet [mailto:johnpc@xs4all.nl] Sent: Thursday, February 02, 2012 1:01 PM To: Borzenkov, Andrey Cc: Randy Rue; toasters@teaparty.net Subject: Re: nfsiostat and client NFS connections

On 2012-2-2 8:28 , Borzenkov, Andrey wrote:

I am not aware offhand of any other tool that would show NFS latency,

unfortunately.

There's "stats show volume:*:avg_latency" on the filer itself, for example (and several other latency counters, both under 'volume' and under 'nfsv3', see "stats list counters nfsv3" and "stats list counters volume").

Of course, that ignores any network latency to the clients, but it might still be a useful thing to measure. The same values are also available via the API, but not via SNMP.

-- Jan-Pieter Cornet SSL is only keeping your connection safe from hackers, crooks and three letter agencies by the least secured, least likely to refuse money from strangers, and least bullying-proof of several hundred companies worldwide.

You need to get into advanced mode to run statit.

ssh HOST "priv set advanced; statit -b"

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeremy Page Sent: Thursday, February 02, 2012 5:35 AM To: toasters@teaparty.net Subject: redirecting statit output via ssh/rsh

Does anyone know if there is a way to get statit to run from ssh/rsh & have the output redirected? For instance if I do ssh <host> vol status I get the results displayed to stdout but if I do ssh <host> statit -b it says command not found.

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.

Are you prefacing it with priv set advanced. Statit isn't a standard command - you have to be at a higher priv level.

for our env using rsh ... sudo rsh filername "priv set advanced;statit -b"

Hope that helps. ChrisB

Jeremy Page wrote:

Does anyone know if there is a way to get statit to run from ssh/rsh & have the output redirected? For instance if I do /ssh <host> vol status /I get the results displayed to stdout but if I do/ ssh <host> statit -b/ it says command not found.

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.

Thanks all. Now I am a little embarrassed, it never occurred to me that it was a separate environment than the interactive shell.

Thanks!

On 02/02/2012 09:01 AM, Vervloesem Wouter wrote:

This is normal because the statit command is a 'priv set diag' command. you should do something like "ssh<host> priv set advanced; statit -b".

Mvg, Wouter Vervloesem

Neoria - Uptime Group Veldkant 35D B-2550 Kontich

Tel: +32 (0)3 451 23 82 Mailto: wouter.vervloesem@neoria.be Web: http://www.neoria.be

Op 2-feb.-2012, om 14:35 heeft Jeremy Page het volgende geschreven:

...

Does anyone know if there is a way to get statit to run from ssh/rsh& have the output redirected? For instance if I do ssh<host> vol status I get the results displayed to stdout but if I do ssh<host> statit -b it says command not found.

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment. _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.