Classification: UNCLASSIFIED Caveats: NONE

Is this really so unusual with an aggregate such as the one described? Another contributor would be if the aggregate is attached to a filer with either a relatively-small amount of NVRAM/NVMEM, or if there's a lot of competing write traffic for any amount of NVRAM/NVMEM. Also, streaming writes have never been a strength of NetApp FAS units. -- Jim Blackburn US Army james.m.blackburn33.civ@mail.mil Note: I am not authorized to obligate, commit, or execute government funds. No part of this message should be construed as a request or directive to obligate, commit, or execute government funds.

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Christopher S Eno Sent: Friday, March 01, 2013 9:35 AM To: 'Brian Beaulieu' Cc: toasters@teaparty.net Subject: RE: wafl_cp_slovol_warning_1 with big latency spikes

Hi Brian, Toasters,

A week ago we were hit with the " :s " CP type again. One vm, on a rather idle aggregate of 15 SATA disks, was storage vmotioned from one vol on the aggregate to another vol on the same aggr. This took about 25 minutes and put the aggr disks to 100%. Near the completion of the VM move, the nasty " :s " CP types began and the controller started refusing all protocols while this CP type ran for up to 30 seconds.

Looking at a graph, one sees a "dead cat bounce" as all traffic and activity on the filer goes to zero, then starts again, then goes to zero, etc., three times total, as the controller attempts to catch up with queued protocol requests, but also trying to handle the long running " :s ". Attempting to attach screen shot of block/s throughput on the aggr.

Brian, I have netapp's attention on this right now, so if you still have a case open, shoot me the case number and I can let them know someone else has seen the same thing.

a5421a7013fbb1ea772a9a98f901b934.png

403c6ca88576065c6a644acdbd12269c.png

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Brian Beaulieu Sent: Thursday, January 17, 2013 8:21 PM To: Scott Eno Cc: toasters@teaparty.net Lists Subject: Re: wafl_cp_slovol_warning_1 with big latency spikes

I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug.

We'll see where it goes.

Brian

On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno s.eno@icloud.com wrote:

We're on 8.1.1P1 across the environment.

7-mode, of course.

On Jan 17, 2013, at 3:35 PM, Brian Beaulieu brian.beaulieu@gmail.com wrote:

Scott, what version of OnTAP are you on?

We're on 8.1.1P1

I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.

Brian

On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler speedtoys.racing@gmail.com wrote:

:s is wafl updating special files in the CP process.

Going on _that_ long....??? A few seconds of special file updates in a CP sure, but that much?

I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket. That's not normal IO activity in a healthy system.

On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno s.eno@me.com wrote:

Yes, this is what I see. ":s" and all the other protocols go to "0".

There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter). But it happens other times too.

On Jan 17, 2013, at 1:47 PM, Brian Beaulieu brian.beaulieu@gmail.com wrote:

3rd time is the charm.

I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well?

During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs. But, AIX handles that just fine and at least has an alternate path through the other filer. NFS isn't so lucky.

I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.

While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.

I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.

I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.

I definitely wouldn't bet on SATA+PAM == FC/SAS performance.

Brian

<sysstat - Copy.txt>_______________________________________________

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

-- --- Gustatus Similis Pullus

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

Classification: UNCLASSIFIED Caveats: NONE