Hi Brian, Toasters,
A week ago we were hit with the “ :s “ CP type again. One vm, on a rather idle aggregate of 15 SATA disks, was storage vmotioned from one vol on the aggregate to another vol on the same aggr. This took about 25 minutes and put the aggr disks to 100%. Near the completion of the VM move, the nasty “ :s “ CP types began and the controller started refusing all protocols while this CP type ran for up to 30 seconds.
Looking at a graph, one sees a “dead cat bounce” as all traffic and activity on the filer goes to zero, then starts again, then goes to zero, etc., three times total, as the controller attempts to catch up with queued protocol requests, but also trying to handle the long running “ :s “. Attempting to attach screen shot of block/s throughput on the aggr.
Brian, I have netapp’s attention on this right now, so if you still have a case open, shoot me the case number and I can let them know someone else has seen the same thing.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Brian Beaulieu
Sent: Thursday, January 17, 2013 8:21 PM
To: Scott Eno
I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug.
We'll see where it goes.
Brian
On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <s.eno@icloud.com> wrote:
We're on 8.1.1P1 across the environment.
7-mode, of course.
On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <brian.beaulieu@gmail.com> wrote:
Scott, what version of OnTAP are you on?
We're on 8.1.1P1
I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.
Brian
On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing@gmail.com> wrote:
:s is wafl updating special files in the CP process.
Going on _that_ long....??? A few seconds of special file updates in a CP sure, but that much?
I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket. That's not normal IO activity in a healthy system.
On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno@me.com> wrote:
Yes, this is what I see. ":s" and all the other protocols go to "0".
There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter). But it happens other times too.
On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu@gmail.com> wrote:
3rd time is the charm.
I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well?
During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs. But, AIX handles that just fine and at least has an alternate path through the other filer. NFS isn't so lucky.
I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.
While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.
I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.
I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
Brian
<sysstat - Copy.txt>_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
--
---
Gustatus Similis Pullus
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
