"nfsstat -d does still hand you a list of VM.."
It will hand you a list of objects that appear to have misaligned IO, but can be very wrong at the same time.
Do not use this as a final determination..IE, DB log files are -a- way that this stat can be fooled.
On Thu, Sep 26, 2013 at 4:30 PM, Andreev, Nikita < Nikita.Andreev@visionstream.com.au> wrote:
I’m gonna run sysstat –x 1 for a considerable amount of time and then sum up the figures. I’ll gather nfsstat statistics too and write back.****
*From:* Colin Bieberstein [mailto:colin@bieberstein.ca] *Sent:* Thursday, 26 September 2013 5:50 PM *To:* Jeff Mohler *Cc:* Andreev, Nikita; toasters@teaparty.net
*Subject:* Re: FAS2050 high CPU usage****
Not going to argue that, and love the analogy. :)****
nfsstat -d does still hand you a list of VMs to fix, so don't write it off as the next step in the chain. ****
Colin Bieberstein ****
On Sep 26, 2013, at 12:46 AM, Jeff Mohler speedtoys.racing@gmail.com wrote:****
With what we know, you can -so- say definitely.****
Hi, I've got 30% of my VM's misaligned, my CPU impact sucks, and the numbers are sickeningly off base for my IO ratios.****
My left arm is numb, I eat three squares at McD's every day, and my chest hurts. Maybe it's an ingrown toenail?
No, its misalignment. :)
That counter for WAFL can also be misleading, as you can be 100% aligned, and still have bad numbers there due to DB (MSSQL, etc) log file IO that on the wire..appears misaligned, yet is not. Log data IO would also not create imbalanced Net to disk IO ratios.****
Virtual case of beer, its all misalignment.****
I'm a Guinness kinda guy.****
On Wed, Sep 25, 2013 at 11:35 PM, Colin Bieberstein colin@bieberstein.ca wrote:****
In sysstat you can't say definitively, but you can see evidence of read / write amplification by comparing network bytes in / out to disk write / reads. There are other reasons for these not to match though. ****
Instead go to the misaligned read stats. You could look at specific counters, but to be honest it's a pain, and NetApp nicely added it to nfsstat for us. (Presuming 7.3.5+ or 8.0.2+) ****
nfsstat Anything not in bin0 on the misaligned read / write stats is misaligned. ****
nfsstat -d Will give you the names of the top files (vmdks). ****
What do you see there?****
Colin Bieberstein ****
On Sep 25, 2013, at 11:47 PM, "Andreev, Nikita" < Nikita.Andreev@visionstream.com.au> wrote:****
How can you tell this from sysstat output?****
*From:* Jeff Mohler [mailto:speedtoys.racing@gmail.comspeedtoys.racing@gmail.com]
*Sent:* Thursday, 26 September 2013 3:39 PM *To:* Andreev, Nikita *Cc:* Colin Bieberstein; toasters@teaparty.net *Subject:* Re: FAS2050 high CPU usage****
Yup, misalignment is your problem.****
Some try to talk it away as minor, it's not.****
Tons of overhead in that output...all misaligned IO.****
On Wed, Sep 25, 2013 at 10:16 PM, Andreev, Nikita < Nikita.Andreev@visionstream.com.au> wrote:****
Each controller has one aggregate of FC 15k drives. Controller1 is using 20 internal drives and Controller2 has a shelf of 24 drives connected to it. But the problem is not with disks. Because disk busy is between 30%-50%.
We do use deduplication on all of the VMware volumes. That was my first idea. But I tried to disable all deduplication at once and didn’t observe any significant difference in CPU usage.****
Compression is not used.****
SnapMirror is done overnight and doesn’t impact production during business hours. We don’t use SnapVault.****
Here is an excerpt from sysstat –x 5 output:****
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s****
in out read write read writeage hit time ty util in out in out****
97% 1392 0 0 1392 22597 6044 18172 32302 0 0 5s 97% 50% Fs 23% 0 0 0 0 0 0****
94% 2024 0 0 2024 14507 8039 20251 18952 0 0 4s 95% 61% Ff 13% 0 0 0 0 0 0****
98% 2125 0 0 2125 20336 10821 18652 31241 0 0 5s 98% 85% Ff 19% 0 0 0 0 0 0****
99% 1801 0 0 1801 31365 14119 20186 31352 0 0 5s 96% 60% Ff 13% 0 0 0 0 0 0****
97% 1396 0 0 1396 27938 14741 24378 29044 0 0 3s 97% 70% F 20% 0 0 0 0 0 0****
99% 1644 0 0 1644 24698 20634 26610 29229 0 0 1 95% 70% Fn 17% 0 0 0 0 0 0****
100% 1337 0 0 1337 28669 19982 20661 42895 0 0 3s 96% 91% Ff 21% 0 0 0 0 0 0****
95% 1082 0 0 1082 21795 11085 25374 34816 0 0 2s 95% 77% Ff 17% 0 0 0 0 0 0****
99% 982 0 0 982 31760 15161 25116 42265 0 0 2s 97% 73% Ff 19% 0 0 0 0 0 0****
96% 1096 0 0 1096 21200 5391 15923 28497 0 0 2s 98% 58% Ff 27% 0 0 0 0 0 0****
Sysstat –m is not supported on FAS2050, because it’s single core.****
Aggregates are more than 70% free.****
We do have misaligned VMs. But I don’t think that the amount of misaligned operations is more than 10-20%. I’ll collect detailed statistics tomorrow and report back.****
Regards,****
*Nikita Andreev | Systems Engineer (Contract)*****
*Visionstream IT Infrastructure Team*****
236 East Boundary Road, 2 North Drive ****
Virginia Park, Bentleigh East VIC 3165 ****
E: Nikita.Andreev@visionstream.com.au****
W: www.visionstream.com.au****
*From:* Colin Bieberstein [mailto:colin@bieberstein.ca] *Sent:* Thursday, 26 September 2013 12:09 PM****
*To:* Andreev, Nikita *Cc:* toasters@teaparty.net *Subject:* Re: FAS2050 high CPU usage****
What sort of aggregates do you have behind your datastores? (aggr status -r or sysconfig -v) Is it, for instance, using only the internal sata disk? A minimum config was 10 drives iirc. That would easily limit your performance down where you describe.****
Otherwise the common CPU culprits are: deduplication jobs, compression jobs, and inline compression. Then snap vault and snap mirror activity. Given that it's a branch office... How many replication jobs have you got running? These won't show up on your protocol ops / second however they will absolutely drag performance down with their I/O. ****
If not, what are you seeing with a sysstat -x 1 and a sysstat -m 1? ****
Have you filled the aggregates and or volumes up past 90%? ****
Do you have misaligned VMs?****
That's where I'd start looking... Without specifics it's hard to point you at a cause, but your 2050 can deliver a lot more than the 1000 IOPS you see with a couple shelves... I believe that you max out with 1 loop of six shelves on that controller but it might have been 4 shelves. ****
Colin Bieberstein ****
On Sep 25, 2013, at 6:42 PM, "Andreev, Nikita" < Nikita.Andreev@visionstream.com.au> wrote:****
Hi All,****
We’re using FAS2050 in one of our branch offices to run VMware cluster over NFS. It turns out that this box runs on 100% CPU usage even when servicing 40MBs/1000IOPS (the max I’ve seen was 70MB/s). Which results in ridiculous latencies.****
I do realise that it’s a Celeron CPU. I just want to double check with you guys, that it’s something you’d expect from this box. Because these days 40MB/s seems to be too little even as a CIFS file server for a small team in a branch office.****
Regards,****
Nikita ****
This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.
This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
****
This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.
This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus ****
This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.
This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________****
--
Gustatus Similis Pullus ****
This email is intended for the named recipient only. The information contained in this message may be confidential, or commercially sensitive. If you are not the intended recipient you must not reproduce or distribute any part of the email, disclose its contents to any other party, or take any action in reliance on it. If you have received this email in error, please contact the sender immediately and please delete this message completely from any systems. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you.
This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________