You can also try enabling Jumbo frames. You can get more utilization out of your NFS access. I am sure you will get a gain in I/O, maybe some CPU cycles back.; how much will become a religious discussion hopefully we don't hash out here... Beware of jumbo frame requirements..

Regards

Steve Klise

Storage Engineer, NCDA

__________________________

PALO ALTO MEDICAL FOUNDATION

SUTTER HEALTH INFORMATION SERVICES

PENINSULA COASTAL REGION

Office: 408-523-3163 · Fax: 408-328-1406

e-mail: klises@sutterhealth.orgmailto:abouelj@sutterhealth.org

WARNING: "All e-mail sent to or from this address will be received or otherwise recorded by the Palo Alto Medical Foundation e-mail system and is subject to archival, monitoring or review by, and/or disclosure to, someone other than the recipient."

________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] On Behalf Of Jordan Slingerland [Jordan.Slingerland@independenthealth.com] Sent: Wednesday, September 25, 2013 6:35 PM To: Jeff Mohler; Andreev, Nikita Cc: toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

Dudup / sis running?

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Wednesday, September 25, 2013 9:33 PM To: Andreev, Nikita Cc: toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Ever check vm alignment?

Sent from my iPhone

On Sep 25, 2013, at 8:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.

Here is an excerpt from sysstat –x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat –m is not supported on FAS2050, because it’s single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au W: www.visionstream.com.auhttp://www.visionstream.com.au/

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM To: Andreev, Nikita Cc: toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem. Some try to talk it away as minor, it's not. Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn't observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn't impact production during business hours. We don't use SnapVault.

Here is an excerpt from sysstat -x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat -m is not supported on FAS2050, because it's single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don't think that the amount of misaligned operations is more than 10-20%. I'll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au W: www.visionstream.com.auhttp://www.visionstream.com.au/

From: Colin Bieberstein [mailto:colin@bieberstein.camailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We're using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I've seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it's a Celeron CPU. I just want to double check with you guys, that it's something you'd expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though.

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+)

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. nfsstat -d Will give you the names of the top files (vmdks).

What do you see there?

Colin Bieberstein

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" Nikita.Andreev@visionstream.com.au wrote:

How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem.

Some try to talk it away as minor, it's not.

Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita Nikita.Andreev@visionstream.com.au wrote: Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.

Here is an excerpt from sysstat –x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat –m is not supported on FAS2050, because it’s single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.au W: www.visionstream.com.au

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" Nikita.Andreev@visionstream.com.au wrote:

Hi All,

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

--

Gustatus Similis Pullus

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

Not going to argue that, and love the analogy. :)

nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain.

Colin Bieberstein

On Sep 26, 2013, at 12:46 AM, Jeff Mohler speedtoys.racing@gmail.com wrote:

With what we know, you can -so- say definitely.

Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios.

My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?

No, its misalignment. :)

That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.

Virtual case of beer, its all misalignment.

I'm a Guinness kinda guy.

...

On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein colin@bieberstein.ca wrote: In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though.

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+)

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. nfsstat -d Will give you the names of the top files (vmdks).

What do you see there?

Colin Bieberstein

...

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" Nikita.Andreev@visionstream.com.au wrote:

How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem.

Some try to talk it away as minor, it's not.

Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita Nikita.Andreev@visionstream.com.au wrote:

Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.

Here is an excerpt from sysstat –x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s

                              in   out   read  write  read write   age   hit time  ty util                 in   out    in   out

97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0

94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0

98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0

99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0

97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0

99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0

100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0

95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0

99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0

96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat –m is not supported on FAS2050, because it’s single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract)

Visionstream IT Infrastructure Team

236 East Boundary Road, 2 North Drive

Virginia Park, Bentleigh East VIC 3165

E: Nikita.Andreev@visionstream.com.au

W: www.visionstream.com.au

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" Nikita.Andreev@visionstream.com.au wrote:

Hi All,

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards,

Nikita

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

--

Gustatus Similis Pullus

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

--

Gustatus Similis Pullus

"nfsstat -d does still hand you a list of VM.."

It will hand you a list of objects that appear to have misaligned IO, but can be very wrong at the same time.

Do not use this as a final determination..IE, DB log files are -a- way that this stat can be fooled.

On Thu, Sep 26, 2013 at 4:30 PM, Andreev, Nikita < Nikita.Andreev@visionstream.com.au> wrote:

I’m gonna run sysstat –x 1 for a considerable amount of time and then sum up the figures. I’ll gather nfsstat statistics too and write back.****

---

*From:* Colin Bieberstein [mailto:colin@bieberstein.ca] *Sent:* Thursday, 26 September 2013 5:50 PM *To:* Jeff Mohler *Cc:* Andreev, Nikita; toasters@teaparty.net

*Subject:* Re: FAS2050 high CPU usage****

---

Not going to argue that, and love the analogy. :)****

---

nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain. ****

---

---

Colin Bieberstein ****

---

On Sep 26, 2013, at 12:46 AM, Jeff Mohler speedtoys.racing@gmail.com wrote:****

With what we know, you can -so- say definitely.****

Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios.****

My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?

No, its misalignment. :)

---

That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.****

---

Virtual case of beer, its all misalignment.****

I'm a Guinness kinda guy.****

---

On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein colin@bieberstein.ca wrote:****

In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though. ****

---

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+) ****

---

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. ****

nfsstat -d Will give you the names of the top files (vmdks). ****

---

What do you see there?****

---

---

Colin Bieberstein ****

---

---

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" < Nikita.Andreev@visionstream.com.au> wrote:****

How can you tell this from sysstat output?****

---

*From:* Jeff Mohler [mailto:speedtoys.racing@gmail.comspeedtoys.racing@gmail.com]

*Sent:* Thursday, 26 September 2013 3:39 PM *To:* Andreev, Nikita *Cc:* Colin Bieberstein; toasters@teaparty.net *Subject:* Re: FAS2050 high CPU usage****

---

Yup, misalignment is your problem.****

Some try to talk it away as minor, it's not.****

Tons of overhead in that output...all misaligned IO.****

---

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita < Nikita.Andreev@visionstream.com.au> wrote:****

Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

---

---

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.****

---

Compression is not used.****

---

SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.****

---

Here is an excerpt from sysstat –x 5 output:****

---

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s****

                              in   out   read  write  read write

age hit time ty util in out in out****

97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0****

94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0****

98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0****

99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0****

97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0****

99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0****

100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0****

95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0****

99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0****

96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0****

---

Sysstat –m is not supported on FAS2050, because it’s single core.****

---

Aggregates are more than 70% free.****

---

We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.****

---

Regards,****

---

*Nikita Andreev | Systems Engineer (Contract)*****

*Visionstream IT Infrastructure Team*****

236 East Boundary Road, 2 North Drive ****

Virginia Park, Bentleigh East VIC 3165 ****

E: Nikita.Andreev@visionstream.com.au****

W: www.visionstream.com.au****

---

*From:* Colin Bieberstein [mailto:colin@bieberstein.ca] *Sent:* Thursday, 26 September 2013 12:09 PM****

*To:* Andreev, Nikita *Cc:* toasters@teaparty.net *Subject:* Re: FAS2050 high CPU usage****

---

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.****

---

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O. ****

---

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1? ****

---

Have you filled the aggregates and or volumes up past 90%? ****

---

Do you have misaligned VMs?****

---

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves. ****

---

Colin Bieberstein ****

---

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" < Nikita.Andreev@visionstream.com.au> wrote:****

Hi All,****

---

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.****

---

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.****

---

Regards,****

Nikita ****

---

---

---

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****

****

---

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****

--

Gustatus Similis Pullus ****

---

---

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****

--

Gustatus Similis Pullus ****

---

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

---

This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

We actually have NetApp Virtual Storage Console installed. Which will show these suckers.

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.com] Sent: Friday, 27 September 2013 9:37 AM To: Andreev, Nikita; Colin Bieberstein; Jeff Mohler Cc: toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

Bind the mbralign cli tool and install it on a linux or esxi host that has access to your vmdk files and use it to scan them, align as necessary.

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Andreev, Nikita Sent: Thursday, September 26, 2013 7:31 PM To: Colin Bieberstein; Jeff Mohler Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

I’m gonna run sysstat –x 1 for a considerable amount of time and then sum up the figures. I’ll gather nfsstat statistics too and write back.

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 5:50 PM To: Jeff Mohler Cc: Andreev, Nikita; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Not going to argue that, and love the analogy. :)

nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain.

Colin Bieberstein

On Sep 26, 2013, at 12:46 AM, Jeff Mohler <speedtoys.racing@gmail.commailto:speedtoys.racing@gmail.com> wrote: With what we know, you can -so- say definitely. Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios. My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?

No, its misalignment. :) That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.

Virtual case of beer, its all misalignment. I'm a Guinness kinda guy.

On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein <colin@bieberstein.camailto:colin@bieberstein.ca> wrote: In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though.

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+)

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. nfsstat -d Will give you the names of the top files (vmdks).

What do you see there?

Colin Bieberstein

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem. Some try to talk it away as minor, it's not. Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.

Here is an excerpt from sysstat –x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat –m is not supported on FAS2050, because it’s single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au W: www.visionstream.com.auhttp://www.visionstream.com.au/

From: Colin Bieberstein [mailto:colin@bieberstein.camailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

Because it means an outage to production VMs. If it doesn't have significant impact on performance, I prefer not to touch it. Actually, I was confused by nfsstats. It didn't show too much of misaligned IOs.

I've been gathering stats for half an hour . Here're the results. Nfsstats again are not too bad. But the net in/out and disk read/write look really disturbing. You've opened my eyes, Jeff. :)

Aligned reads 148742 (BIN0) Aligned writes 244237 (BIN0) Mis Reads 4915 (BIN1-7) Mis Writes 40731 (BIN1-7) Mis Reads % 3.30 Mis Writes % 16.67

Net In 7679986 KB Net Out 3208097 KB Disk Read 9499391 KB Disk Write 10593904 KB More Read x2.96 times More Write x1.37 times

Now I'll move the misaligned VMs and see if it solves the problem.

Thanks for helping with that, guys.

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Friday, 27 September 2013 9:42 AM To: Andreev, Nikita Cc: Jordan Slingerland; Colin Bieberstein; toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Then..why do you still -have- them misaligned?

You have a tool begging for your attention on this matter...

On Thu, Sep 26, 2013 at 4:38 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: We actually have NetApp Virtual Storage Console installed. Which will show these suckers.

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.commailto:Jordan.Slingerland@independenthealth.com] Sent: Friday, 27 September 2013 9:37 AM To: Andreev, Nikita; Colin Bieberstein; Jeff Mohler

Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

Bind the mbralign cli tool and install it on a linux or esxi host that has access to your vmdk files and use it to scan them, align as necessary.

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Andreev, Nikita Sent: Thursday, September 26, 2013 7:31 PM To: Colin Bieberstein; Jeff Mohler Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

I'm gonna run sysstat -x 1 for a considerable amount of time and then sum up the figures. I'll gather nfsstat statistics too and write back.

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 5:50 PM To: Jeff Mohler Cc: Andreev, Nikita; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Not going to argue that, and love the analogy. :)

nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain.

Colin Bieberstein

On Sep 26, 2013, at 12:46 AM, Jeff Mohler <speedtoys.racing@gmail.commailto:speedtoys.racing@gmail.com> wrote: With what we know, you can -so- say definitely. Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios. My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?

No, its misalignment. :) That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.

Virtual case of beer, its all misalignment. I'm a Guinness kinda guy.

On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein <colin@bieberstein.camailto:colin@bieberstein.ca> wrote: In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though.

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+)

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. nfsstat -d Will give you the names of the top files (vmdks).

What do you see there?

Colin Bieberstein

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem. Some try to talk it away as minor, it's not. Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn't observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn't impact production during business hours. We don't use SnapVault.

Here is an excerpt from sysstat -x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat -m is not supported on FAS2050, because it's single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don't think that the amount of misaligned operations is more than 10-20%. I'll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au W: www.visionstream.com.auhttp://www.visionstream.com.au/

From: Colin Bieberstein [mailto:colin@bieberstein.camailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We're using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I've seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it's a Celeron CPU. I just want to double check with you guys, that it's something you'd expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

I'll make them functionally aligned by creating misaligned VMFS datastore.

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Friday, 27 September 2013 1:51 PM To: Andreev, Nikita Cc: Jordan Slingerland; Colin Bieberstein; toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Move them where?

Just curious. It'll cost you sooner or later. ;)

It greatly depends where the misaligned IO is in the stream. Smaller & more random writes are by far the most painful misaligned IO to experience..as that becomes more sequential and/or larger IOs, less impact.

Read misalignment is the same way..smaller more random costs more, but unlike writes, barely costs you any additional CPU..just leading and trialing IO block overhead per IO. You can have 300 misaligned VMs creating a total misaligned workload causing your pain, or -1- misaligned VM with tons of the more painful IO doing it to you. Its rarely ever just the # of VMs misaligned. (for anyone wondering).

On Thu, Sep 26, 2013 at 8:34 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Because it means an outage to production VMs. If it doesn't have significant impact on performance, I prefer not to touch it. Actually, I was confused by nfsstats. It didn't show too much of misaligned IOs.

I've been gathering stats for half an hour . Here're the results. Nfsstats again are not too bad. But the net in/out and disk read/write look really disturbing. You've opened my eyes, Jeff. :)

Aligned reads 148742 (BIN0) Aligned writes 244237 (BIN0) Mis Reads 4915 (BIN1-7) Mis Writes 40731 (BIN1-7) Mis Reads % 3.30 Mis Writes % 16.67

Net In 7679986 KB Net Out 3208097 KB Disk Read 9499391 KB Disk Write 10593904 KB More Read x2.96 times More Write x1.37 times

Now I'll move the misaligned VMs and see if it solves the problem.

Thanks for helping with that, guys.

From: Jeff Mohler [mailto:speedtoys.racing@gmail.commailto:speedtoys.racing@gmail.com] Sent: Friday, 27 September 2013 9:42 AM To: Andreev, Nikita Cc: Jordan Slingerland; Colin Bieberstein; toasters@teaparty.netmailto:toasters@teaparty.net

Subject: Re: FAS2050 high CPU usage

Then..why do you still -have- them misaligned?

You have a tool begging for your attention on this matter...

On Thu, Sep 26, 2013 at 4:38 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: We actually have NetApp Virtual Storage Console installed. Which will show these suckers.

From: Jordan Slingerland [mailto:Jordan.Slingerland@independenthealth.commailto:Jordan.Slingerland@independenthealth.com] Sent: Friday, 27 September 2013 9:37 AM To: Andreev, Nikita; Colin Bieberstein; Jeff Mohler

Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

Bind the mbralign cli tool and install it on a linux or esxi host that has access to your vmdk files and use it to scan them, align as necessary.

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Andreev, Nikita Sent: Thursday, September 26, 2013 7:31 PM To: Colin Bieberstein; Jeff Mohler Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: RE: FAS2050 high CPU usage

I'm gonna run sysstat -x 1 for a considerable amount of time and then sum up the figures. I'll gather nfsstat statistics too and write back.

From: Colin Bieberstein [mailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 5:50 PM To: Jeff Mohler Cc: Andreev, Nikita; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Not going to argue that, and love the analogy. :)

nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain.

Colin Bieberstein

On Sep 26, 2013, at 12:46 AM, Jeff Mohler <speedtoys.racing@gmail.commailto:speedtoys.racing@gmail.com> wrote: With what we know, you can -so- say definitely. Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios. My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?

No, its misalignment. :) That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.

Virtual case of beer, its all misalignment. I'm a Guinness kinda guy.

On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein <colin@bieberstein.camailto:colin@bieberstein.ca> wrote: In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though.

Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+)

nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. nfsstat -d Will give you the names of the top files (vmdks).

What do you see there?

Colin Bieberstein

On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: How can you tell this from sysstat output?

From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Thursday, 26 September 2013 3:39 PM To: Andreev, Nikita Cc: Colin Bieberstein; toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

Yup, misalignment is your problem. Some try to talk it away as minor, it's not. Tons of overhead in that output...all misaligned IO.

On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.

We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn't observe any significant difference in CPU usage.

Compression is not used.

SnapMirror is done overnight and doesn't impact production during business hours. We don't use SnapVault.

Here is an excerpt from sysstat -x 5 output:

CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0 94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0 98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0 99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0 97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0 99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0 100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0 95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0 99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0 96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0

Sysstat -m is not supported on FAS2050, because it's single core.

Aggregates are more than 70% free.

We do have misaligned VMs. But I don't think that the amount of misaligned operations is more than 10-20%. I'll collect detailed statistics tomorrow and report back.

Regards,

Nikita Andreev | Systems Engineer (Contract) Visionstream IT Infrastructure Team 236 East Boundary Road, 2 North Drive Virginia Park, Bentleigh East VIC 3165 E: Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au W: www.visionstream.com.auhttp://www.visionstream.com.au/

From: Colin Bieberstein [mailto:colin@bieberstein.camailto:colin@bieberstein.ca] Sent: Thursday, 26 September 2013 12:09 PM

To: Andreev, Nikita Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: FAS2050 high CPU usage

What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.

Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O.

If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1?

Have you filled the aggregates and or volumes up past 90%?

Do you have misaligned VMs?

That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves.

Colin Bieberstein

On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" <Nikita.Andreev@visionstream.com.aumailto:Nikita.Andreev@visionstream.com.au> wrote: Hi All,

We're using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I've seen was 70MB/s). Which results in ridiculous latencies.

I do realise that it's a Celeron CPU. I just want to double check with you guys, that it's something you'd expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.

Regards, Nikita

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________

-- --- Gustatus Similis Pullus

________________________________

This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.

______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________