really? i am running a mixed mode (2 nodes at 8.1P1 and 2 nodes at 8.1.1RC1) I do not see anything wrong.

What did you see specifically?

--tmac Tim McCarthy Principal Consultant

RedHat Certified Engineer 804006984323821 (RHEL4) 805007643429572 (RHEL5)

On Fri, Jul 27, 2012 at 6:00 AM, Darren Sykes Darren.Sykes@csr.com wrote:

As a word of warning, we attempted to upgrade our C mode cluster from 8.1 to 8.1.1RC1 but ended up rolling back after the storage simply didn’t behave correctly for NFS clients. ****

---

---

---

*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *tmac *Sent:* 26 July 2012 22:00 *To:* alonz *Cc:* toasters@teaparty.net *Subject:* Re: Ontap 8.1 performance issue****

---

Looks like 8.1P2 was just released in the last 24 hours..****

On Jul 26, 2012 4:47 PM, "Alon Zeltser" alonz@emet.co.il wrote:****

-------- Original message -------- Subject:RE: Ontap 8.1 performance issue From:Alon Zeltser alonz@emet.co.il To:"Klise, Steve" klises@sutterhealth.org Cc:

Yes this include p1 according to netapp support this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" klises@sutterhealth.org wrote:

Was this noticed also on 8.1p1? Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Thursday, July 26, 2012 11:01 AM To: toasters@teaparty.net Subject: Ontap 8.1 performance issue

Hi all As a netapp ASP I spent quite alot of my time installing netapp's at costumers site and resolving their issues and I wanted to share my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release. but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt that they are justified something is off in system I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy. After opening support cases at netapp one of the engineers told me that they also gets alot of performance cases with this release an even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput Today on new system I install I completely skip th GA version an go straight to 8.1.1rc1 So my advice if you on this release and suffering from performance and don't know if you have enough disks or the right controller upgrade first then check

Sent from Samsung Mobile

---

Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****

---

To report this email as spam click herehttps://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg== .****

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

How long do the spikes last?

I think with dfm you can set an alert based on latency and email an operator to kick off a perfstat.

Jack Sent from my Verizon Wireless BlackBerry

-----Original Message----- From: Nilsson Marcus marcus.nilsson@atea.se Sender: toasters-bounces@teaparty.net Date: Sun, 29 Jul 2012 07:48:25 To: tmactmacmd@gmail.com; alonzalonz@emet.co.il Cc: toasters@teaparty.nettoasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

Hi, I have a 3210 single controller system with intermittent latency spikes. The system has 48 x 3 TB disks. We went with ONTAP 8.1 to utilize big aggregates. Two aggregates has been created, one with 16 disk and another with 31 disk. The latency spikes affect volumes from both aggregates (at the same time). It's a CIFS only system running SMB2.

Upgrading to ONTAP 8.1P2 has not resolved the latency spikes on this system, see screenshot from DFM: http://i.imgur.com/TAXvl.png

I have not tried 8.1.1RC1. Disk utilization is not peaking.

I have an open support ticket with NetApp, they want me to capture a perfstat log during the latency spikes. This has proven difficult since the spikes are so intermittent.

Any suggestions on how to track this down are welcome.

Thanks, Marcus

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: den 26 juli 2012 23:00 To: alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

Looks like 8.1P2 was just released in the last 24 hours.. On Jul 26, 2012 4:47 PM, "Alon Zeltser" alonz@emet.co.il wrote:

-------- Original message -------- Subject:RE: Ontap 8.1 performance issue From:Alon Zeltser alonz@emet.co.il To:"Klise, Steve" klises@sutterhealth.org Cc:

Yes this  include  p1 according to netapp support  this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" klises@sutterhealth.org wrote:

Was this noticed also on 8.1p1?  Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Thursday, July 26, 2012 11:01 AM To: toasters@teaparty.net Subject: Ontap 8.1 performance issue

Hi all As a netapp ASP I spent quite alot of my time installing netapp's at costumers  site and resolving their issues and I wanted to share  my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release.  but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt  that they  are justified something is off in system  I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy. After opening support cases at netapp one of the engineers told me that they also gets alot of performance  cases with this release an  even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade  I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s  and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput  Today on new system I install I completely skip th  GA version an  go straight to 8.1.1rc1 So my advice if you on this release  and suffering from performance and don't know if you have enough disks or the right controller upgrade  first then check

Sent from Samsung Mobile

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

We've had issues with anti-virus scanners as well. If your running trend, I can share what happened with us. Maybe running CIFS TOP when the latency goes up may show something as well. Another issue we had back in the day was the combintation of DFM, CIFS, 3070's caused the filer to run low on memory, and we actually started to loose packets.. Best of luck.

Regards Steve Klise Storage Engineer, NCDA __________________________ PALO ALTO MEDICAL FOUNDATION SUTTER HEALTH INFORMATION SERVICES PENINSULA COASTAL REGION

Office: 408-523-3163 · Fax: 408-328-1406 e-mail: klises@sutterhealth.org

WARNING: "All e-mail sent to or from this address will be received or otherwise recorded by the Palo Alto Medical Foundation e-mail system and is subject to archival, monitoring or review by, and/or disclosure to, someone other than the recipient." ________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] On Behalf Of Nilsson Marcus [marcus.nilsson@atea.se] Sent: Sunday, July 29, 2012 9:11 AM To: jack1729@gmail.com; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

It only lasts for a couple of minutes. I've set up an alert in DFM to kick of a script to start perfstat when latency exceeds 20ms. Hopefully that will do the trick.

Thanks, Marcus

-----Original Message----- From: Jack Lyons [mailto:jack1729@gmail.com] Sent: den 29 juli 2012 12:42 To: Nilsson Marcus; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

How long do the spikes last?

I think with dfm you can set an alert based on latency and email an operator to kick off a perfstat.

Jack Sent from my Verizon Wireless BlackBerry

-----Original Message----- From: Nilsson Marcus marcus.nilsson@atea.se Sender: toasters-bounces@teaparty.net Date: Sun, 29 Jul 2012 07:48:25 To: tmactmacmd@gmail.com; alonzalonz@emet.co.il Cc: toasters@teaparty.nettoasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

Hi, I have a 3210 single controller system with intermittent latency spikes. The system has 48 x 3 TB disks. We went with ONTAP 8.1 to utilize big aggregates. Two aggregates has been created, one with 16 disk and another with 31 disk. The latency spikes affect volumes from both aggregates (at the same time). It's a CIFS only system running SMB2.

Upgrading to ONTAP 8.1P2 has not resolved the latency spikes on this system, see screenshot from DFM: http://i.imgur.com/TAXvl.png

I have not tried 8.1.1RC1. Disk utilization is not peaking.

I have an open support ticket with NetApp, they want me to capture a perfstat log during the latency spikes. This has proven difficult since the spikes are so intermittent.

Any suggestions on how to track this down are welcome.

Thanks, Marcus

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: den 26 juli 2012 23:00 To: alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

Looks like 8.1P2 was just released in the last 24 hours.. On Jul 26, 2012 4:47 PM, "Alon Zeltser" alonz@emet.co.il wrote:

-------- Original message -------- Subject:RE: Ontap 8.1 performance issue From:Alon Zeltser alonz@emet.co.il To:"Klise, Steve" klises@sutterhealth.org Cc:

Yes this include p1 according to netapp support this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" klises@sutterhealth.org wrote:

Was this noticed also on 8.1p1? Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Thursday, July 26, 2012 11:01 AM To: toasters@teaparty.net Subject: Ontap 8.1 performance issue

Hi all As a netapp ASP I spent quite alot of my time installing netapp's at costumers site and resolving their issues and I wanted to share my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release. but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt that they are justified something is off in system I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy. After opening support cases at netapp one of the engineers told me that they also gets alot of performance cases with this release an even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput Today on new system I install I completely skip th GA version an go straight to 8.1.1rc1 So my advice if you on this release and suffering from performance and don't know if you have enough disks or the right controller upgrade first then check

Sent from Samsung Mobile

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

No AV scanning activated.

/ Marcus

-----Original Message----- From: Klise, Steve [mailto:klises@sutterhealth.org] Sent: den 29 juli 2012 18:22 To: Nilsson Marcus; jack1729@gmail.com; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

We've had issues with anti-virus scanners as well. If your running trend, I can share what happened with us. Maybe running CIFS TOP when the latency goes up may show something as well. Another issue we had back in the day was the combintation of DFM, CIFS, 3070's caused the filer to run low on memory, and we actually started to loose packets.. Best of luck.

Regards Steve Klise Storage Engineer, NCDA __________________________ PALO ALTO MEDICAL FOUNDATION SUTTER HEALTH INFORMATION SERVICES PENINSULA COASTAL REGION

Office: 408-523-3163 * Fax: 408-328-1406 e-mail: klises@sutterhealth.org

WARNING: "All e-mail sent to or from this address will be received or otherwise recorded by the Palo Alto Medical Foundation e-mail system and is subject to archival, monitoring or review by, and/or disclosure to, someone other than the recipient." ________________________________________ From: toasters-bounces@teaparty.net [toasters-bounces@teaparty.net] On Behalf Of Nilsson Marcus [marcus.nilsson@atea.se] Sent: Sunday, July 29, 2012 9:11 AM To: jack1729@gmail.com; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

It only lasts for a couple of minutes. I've set up an alert in DFM to kick of a script to start perfstat when latency exceeds 20ms. Hopefully that will do the trick.

Thanks, Marcus

-----Original Message----- From: Jack Lyons [mailto:jack1729@gmail.com] Sent: den 29 juli 2012 12:42 To: Nilsson Marcus; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

How long do the spikes last?

I think with dfm you can set an alert based on latency and email an operator to kick off a perfstat.

Jack Sent from my Verizon Wireless BlackBerry

-----Original Message----- From: Nilsson Marcus marcus.nilsson@atea.se Sender: toasters-bounces@teaparty.net Date: Sun, 29 Jul 2012 07:48:25 To: tmactmacmd@gmail.com; alonzalonz@emet.co.il Cc: toasters@teaparty.nettoasters@teaparty.net Subject: RE: Ontap 8.1 performance issue

Hi, I have a 3210 single controller system with intermittent latency spikes. The system has 48 x 3 TB disks. We went with ONTAP 8.1 to utilize big aggregates. Two aggregates has been created, one with 16 disk and another with 31 disk. The latency spikes affect volumes from both aggregates (at the same time). It's a CIFS only system running SMB2.

Upgrading to ONTAP 8.1P2 has not resolved the latency spikes on this system, see screenshot from DFM: http://i.imgur.com/TAXvl.png

I have not tried 8.1.1RC1. Disk utilization is not peaking.

I have an open support ticket with NetApp, they want me to capture a perfstat log during the latency spikes. This has proven difficult since the spikes are so intermittent.

Any suggestions on how to track this down are welcome.

Thanks, Marcus

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: den 26 juli 2012 23:00 To: alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

Looks like 8.1P2 was just released in the last 24 hours.. On Jul 26, 2012 4:47 PM, "Alon Zeltser" alonz@emet.co.il wrote:

-------- Original message -------- Subject:RE: Ontap 8.1 performance issue From:Alon Zeltser alonz@emet.co.il To:"Klise, Steve" klises@sutterhealth.org Cc:

Yes this include p1 according to netapp support this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" klises@sutterhealth.org wrote:

Was this noticed also on 8.1p1? Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Thursday, July 26, 2012 11:01 AM To: toasters@teaparty.net Subject: Ontap 8.1 performance issue

Hi all As a netapp ASP I spent quite alot of my time installing netapp's at costumers site and resolving their issues and I wanted to share my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release. but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt that they are justified something is off in system I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy. After opening support cases at netapp one of the engineers told me that they also gets alot of performance cases with this release an even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput Today on new system I install I completely skip th GA version an go straight to 8.1.1rc1 So my advice if you on this release and suffering from performance and don't know if you have enough disks or the right controller upgrade first then check

Sent from Samsung Mobile

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

No weird messages in the syslog.

cifs stat

reject 0 0%

mkdir 0 0%

rmdir 0 0%

open 0 0%

create 0 0%

close 107 0%

X&close 0 0%

flush 0 0%

X&flush 0 0%

delete 0 0%

rename 0 0%

NTRename 0 0%

getatr 0 0%

setatr 0 0%

read 0 0%

X&read 0 0%

write 0 0%

X&write 0 0%

lock 0 0%

unlock 0 0%

mknew 0 0%

chkpth 0 0%

exit 0 0%

lseek 0 0%

lockread 0 0%

X&lockread 0 0%

writeunlock 0 0%

readbraw 0 0%

writebraw 0 0%

writec 0 0%

gettattre 0 0%

settattre 0 0%

lockingX 0 0%

IPC 12 0%

open2 0 0%

find_first2 41 0%

find_next2 0 0%

query_fs_info 5 0%

query_path_info 30 0%

set_path_info 0 0%

query_file_info 24 0%

set_file_info 8 0%

create_dir2 0 0%

Dfs_referral 9 0%

Dfs_report 0 0%

echo 47 0%

writeclose 0 0%

openX 0 0%

readX 89 0%

writeX 1800 0%

findclose 0 0%

tcon 0 0%

tdis 42 0%

negprot 9 0%

login 34 0%

logout 6 0%

tconX 9 0%

dskattr 0 0%

search 0 0%

fclose 0 0%

NTCreateX 429 0%

NTTransCreate 0 0%

NTTransIoctl 0 0%

NTTransNotify 9 0%

NTTransSetSec 0 0%

NTTransQuerySec 0 0%

NTNamedPipeMulti 0 0%

NTCancel CN 3 0%

NTCancel Other 0 0%

SMB2Echo 0 0%

SMB2Negprot 63 0%

SMB2TreeConnnect 896 0%

SMB2TreeDisconnect 34 0%

SMB2Login 126 0%

SMB2Create 1767 0%

SMB2Read 268979 27%

SMB2Write 709851 71%

SMB2Lock 0 0%

SMB2Unlock 0 0%

SMB2OplkBrkAck 0 0%

SMB2ChgNfy 7 0%

SMB2CLose 951 0%

SMB2Flush 855 0%

SMB2Logout 0 0%

SMB2Cancel 5 0%

SMB2IPCCreate 0 0%

SMB2IPCRead 0 0%

SMB2IPCWrite 0 0%

SMB2QueryDir 4738 0%

SMB2QueryFileBasicInfo 0 0%

SMB2QueryFileStndInfo 0 0%

SMB2QueryFileIntInfo 0 0%

SMB2QueryFileEAInfo 0 0%

SMB2QueryFileFEAInfo 0 0%

SMB2QueryFileModeInfo 0 0%

SMB2QueryAltNameInfo 0 0%

SMB2QueryFileStreamInfo 0 0%

SMB2QueryNetOpenInfo 0 0%

SMB2QueryAttrTagInfo 0 0%

SMB2QueryAccessInfo 0 0%

SMB2QueryFileUnsupported 0 0%

SMB2QueryFileInvalid 0 0%

SMB2QueryFSVolInfo 174 0%

SMB2QueryFSSizeInfo 0 0%

SMB2QueryFSDevInfo 0 0%

SMB2QueryFSAttrInfo 174 0%

SMB2QueryFSFullSzInfo 4819 0%

SMB2QueryFSObjIdInfo 0 0%

SMB2QueryFSInvalid 0 0%

SMB2QuerySecurityInfo 0 0%

SMB2SetBasicInfo 0 0%

SMB2SetRenameInfo 0 0%

SMB2SetFileLinkInfo 0 0%

SMB2SetFileDispInfo 85 0%

SMB2SetFullEAInfo 0 0%

SMB2SetModeInfo 0 0%

SMB2SetAllocInfo 0 0%

SMB2SetEOFInfo 0 0%

SMB2SetUnsupported 0 0%

SMB2SetInfoInvalid 0 0%

SMB2SetSecurityInfo 0 0%

SMB2FsctlPipeTransceive 0 0%

SMB2FsctlPipePeek 0 0%

SMB2FsctlEnumSnapshots 0 0%

SMB2FsctlDfsReferrals 327 0%

SMB2FsctlSetSparse 0 0%

SMB2FsctlSecureShare 0 0%

SMB2FsctlFileUnsupported 0 0%

SMB2FsctlIpcUnsupported 0 0%

cancel lock 0

wait lock 0

copy to align 0

alignedSmall 5915

alignedLarge 1523

alignedSmallRel 0

alignedLargeRel 0

FidHashAllocs 0

TidHashAllocs 0

UidHashAllocs 0

mbufWait 0

nbtWait 0

pBlkWait 0

BackToBackCPWait 0

cwaWait 0

short msg prevent 31

multipleVCs 2

SMB signing 0

mapped null user 0

PDCupcalls 0

nosupport 0

read pipe busy 0

write pipe busy 0

trans pipe busy 0

read pipe broken 0

write pipe broken 0

trans pipe broken 0

queued writeraw 0

nbt disconnect 71

smb disconnect 0

dup disconnect 62

OpLkBkXorBatchToL2 0

OpLkBkXorBatchToNone 0

OpLkBkL2ToNone 0

OpLkBkNoBreakAck 0

OpLkBkNoBreakAck95 0

OpLkBkNoBreakAckNT 0

OpLkBkIgnoredAck 0

OpLkBkWaiterTimedOut 0

OpLkBkDelayedBreak 0

SharingErrorRetries 0

FoldAttempts 0

FoldRenames 0

FoldRenameFailures 0

FoldOverflows 0

FoldDuplicates 0

FoldWAFLTooBusy 0

NoAllocCredStat 0

RetryRPCcollision 0

TconCloseTID 0

GetNTAPExtAttrs 0

SetNTAPExtAttrs 0

SearchBusy 0

ChgNfyNoMemory 0

ChgNfyNewWatch 4

ChgNfyLastWatch 3

UsedMIDTblCreated 24

UnusedMIDTblCreated 15

InvalidMIDRejects 0

SMB2InvalidSignature 0

SMB2DurableCreateReceived 1460

SMB2DurableCreateSucceeded 546

SMB2DurableReclaimReceived 307

SMB2DurableReclaimSucceeded 18

SMB2DurableHandlePreserved 186

SMB2DurableHandlePurged 8

SMB2DurableHandleExpired 164

SMB2FileDirInfo 0

SMB2FileFullDirInfo 0

SMB2FileIdFullDirInfo 0

SMB2FileBothDirInfo 3629

SMB2FileIdBothDirInfo 803

SMB2FileNamesInfo 0

SMB2FileDirUnsupported 0

SMB2QueryInfo 5167

SMB2SetInfo 85

SMB2Ioctl 327

SMB2RelatedCompRequest 780

SMB2UnRelatedCompRequest 0

SMB2FileRequest 992418

SMB2PipeRequest 343

SMB2nosupport 0

Max Multiplex = 49, Max pBlk Exhaust = 0, Max pBlk Reserve Exhaust = 0

Max FIDs = 204, Max FIDs on one tree = 17

Max Searches on one tree = 1, Max Core Searches on one tree = 0

Max sessions = 27

Max trees = 103

Max shares = 9

Max session UIDs = 2, Max session TIDs = 5

Max locks = 126

Max credentials = 26

Max group SIDs per credential = 4

Max pBlks = 1024 Current pBlks = 999 Num Logons = 0

Max reserved pBlks = 32 Current reserved pBlks = 32

Max gAuthQueue depth = 3

Max gSMBBlockingQueue depth = 4

Max gSMBTimerQueue depth = 3

Max gSMBAlfQueue depth = 1

Max gSMBRPCWorkerQueue depth = 1

Max gOffloadQueue depth = 2

Local groups: builtins = 6, user-defined = 1, SIDs = 7

RPC group count = 10, RPC group active count = 0

Max Watched Directories = 6, Current Watched Directories = 5

Max Pending ChangeNotify Requests = 4, Current Pending ChangeNotify Requests = 2

Max Pending DeleteOnClose Requests = 3072, Current Pending DeleteOnClose Requests = 24

/ Marcus

-----Original Message----- From: Jack Lyons [mailto:jack1729@gmail.com] Sent: den 29 juli 2012 19:10 To: Nilsson Marcus; toasters-bounces@teaparty.net; tmac; alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

Are you seeing pBLK exhaustion messages in the logs?

Sent from my Verizon Wireless BlackBerry

-----Original Message-----

From: Nilsson Marcus <marcus.nilsson@atea.semailto:marcus.nilsson@atea.se>

Date: Sun, 29 Jul 2012 18:11:08

To: jack1729@gmail.com<jack1729@gmail.commailto:jack1729@gmail.com%3cjack1729@gmail.com>; toasters-bounces@teaparty.net<toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net%3ctoasters-bounces@teaparty.net>; tmac<tmacmd@gmail.commailto:tmacmd@gmail.com>; alonz<alonz@emet.co.ilmailto:alonz@emet.co.il>

Cc: toasters@teaparty.net<toasters@teaparty.netmailto:toasters@teaparty.net%3ctoasters@teaparty.net>

Subject: RE: Ontap 8.1 performance issue

It only lasts for a couple of minutes. I've set up an alert in DFM to kick of a script to start perfstat when latency exceeds 20ms. Hopefully that will do the trick.

Thanks,

Marcus

-----Original Message-----

From: Jack Lyons [mailto:jack1729@gmail.com]mailto:[mailto:jack1729@gmail.com]

Sent: den 29 juli 2012 12:42

To: Nilsson Marcus; toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net; tmac; alonz

Cc: toasters@teaparty.netmailto:toasters@teaparty.net

Subject: Re: Ontap 8.1 performance issue

How long do the spikes last?

I think with dfm you can set an alert based on latency and email an operator to kick off a perfstat.

Jack

Sent from my Verizon Wireless BlackBerry

-----Original Message-----

From: Nilsson Marcus <marcus.nilsson@atea.semailto:marcus.nilsson@atea.se>

Sender: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net

Date: Sun, 29 Jul 2012 07:48:25

To: tmac<tmacmd@gmail.commailto:tmacmd@gmail.com>; alonz<alonz@emet.co.ilmailto:alonz@emet.co.il>

Cc: toasters@teaparty.net<toasters@teaparty.netmailto:toasters@teaparty.net%3ctoasters@teaparty.net>

Subject: RE: Ontap 8.1 performance issue

Hi,

I have a 3210 single controller system with intermittent latency spikes. The system has 48 x 3 TB disks. We went with ONTAP 8.1 to utilize big aggregates. Two aggregates has been created, one with 16 disk and another with 31 disk. The latency spikes affect volumes from both aggregates (at the same time). It's a CIFS only system running SMB2.

Upgrading to ONTAP 8.1P2 has not resolved the latency spikes on this system, see screenshot from DFM: http://i.imgur.com/TAXvl.png

I have not tried 8.1.1RC1. Disk utilization is not peaking.

I have an open support ticket with NetApp, they want me to capture a perfstat log during the latency spikes. This has proven difficult since the spikes are so intermittent.

Any suggestions on how to track this down are welcome.

Thanks, Marcus

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net]mailto:[mailto:toasters-bounces@teaparty.net] On Behalf Of tmac

Sent: den 26 juli 2012 23:00

To: alonz

Cc: toasters@teaparty.netmailto:toasters@teaparty.net

Subject: Re: Ontap 8.1 performance issue

Looks like 8.1P2 was just released in the last 24 hours..

On Jul 26, 2012 4:47 PM, "Alon Zeltser" <alonz@emet.co.ilmailto:alonz@emet.co.il> wrote:

-------- Original message --------

Subject:RE: Ontap 8.1 performance issue

From:Alon Zeltser <alonz@emet.co.ilmailto:alonz@emet.co.il>

To:"Klise, Steve" <klises@sutterhealth.orgmailto:klises@sutterhealth.org>

Cc:

Yes this include p1 according to netapp support this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" <klises@sutterhealth.orgmailto:klises@sutterhealth.org> wrote:

Was this noticed also on 8.1p1? Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net]mailto:[mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser

Sent: Thursday, July 26, 2012 11:01 AM

To: toasters@teaparty.netmailto:toasters@teaparty.net

Subject: Ontap 8.1 performance issue

Hi all

As a netapp ASP I spent quite alot of my time installing netapp's at costumers site and resolving their issues and I wanted to share my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release.

but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt that they are justified something is off in system

I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy.

After opening support cases at netapp one of the engineers told me that they also gets alot of performance cases with this release an even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput

Today on new system I install I completely skip th GA version an go straight to 8.1.1rc1 So my advice if you on this release and suffering from performance and don't know if you have enough disks or the right controller upgrade first then check

Sent from Samsung Mobile

_______________________________________________

Toasters mailing list

Toasters@teaparty.netmailto:Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________

Toasters mailing list

Toasters@teaparty.netmailto:Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters

The wierd thing is that the latancy spikes does not correlate with IOPS load. For example the system can have a low disk util % and IOPS load and still show high CIFS latency (in excess of 500ms).

/ Marcus

From: Fletcher Cocquyt [mailto:fcocquyt@stanford.edu] Sent: den 29 juli 2012 19:43 To: Nilsson Marcus Cc: toasters@teaparty.net Lists Subject: Re: Ontap 8.1 performance issue

Marcus - have you

1 - done the math to work out the IOPs profile of your aggr and  2 - compared it to your IOPs workload (including internal ONTAP IOPs like snap mirror)?- pls see http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html

For instance: 1 - since these are 3Tb disks, we'll assume (plugin a new spec if not) these are 7200RPM SATA drives which can deliver ~40 IOPs at 20ms latency 48 x 40 = 1920 IOPS delivered at 20ms (Note 15K disks can deliver > 5x this  IOPs at 20ms) 2 - check out DFM disk busy history: View->Physical->selectYourAggr->TopDisksByDisk_Busy->Actions->ShowAsLineChart->Range->1week - this will quickly show you a - how close you are to the IO capacity of your aggr b - time of day patterns which may indicate where the IO is coming from (snap mirror/virus scans etc)

Also check out: View->Physical->selectYourHead->TopAggregates->TotalTransfers->Actions->ShowAsLineChart->Range->1week compare the IOPs in the chart with the 1920 number we calculated - do you have IO spikes crossing the 1920 threshold? - do these correlate to the times of your latency spikes?

If so, you may have an IOPs issue - (further deconstruction of your IOPs may reveal a solution - eg in my case it was rescheduling snap mirror operations)

Happy hunting,

Fletcher

On Jul 28, 2012, at 10:48 PM, Nilsson Marcus wrote:

Hi, I have a 3210 single controller system with intermittent latency spikes. The system has 48 x 3 TB disks. We went with ONTAP 8.1 to utilize big aggregates. Two aggregates has been created, one with 16 disk and another with 31 disk. The latency spikes affect volumes from both aggregates (at the same time). It's a CIFS only system running SMB2.

Upgrading to ONTAP 8.1P2 has not resolved the latency spikes on this system, see screenshot from DFM: http://i.imgur.com/TAXvl.png

I have not tried 8.1.1RC1. Disk utilization is not peaking.

I have an open support ticket with NetApp, they want me to capture a perfstat log during the latency spikes. This has proven difficult since the spikes are so intermittent.

Any suggestions on how to track this down are welcome.

Thanks, Marcus

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: den 26 juli 2012 23:00 To: alonz Cc: toasters@teaparty.net Subject: Re: Ontap 8.1 performance issue

Looks like 8.1P2 was just released in the last 24 hours.. On Jul 26, 2012 4:47 PM, "Alon Zeltser" alonz@emet.co.il wrote:

-------- Original message -------- Subject:RE: Ontap 8.1 performance issue From:Alon Zeltser alonz@emet.co.il To:"Klise, Steve" klises@sutterhealth.org Cc:

Yes this  include  p1 according to netapp support  this should be fixed at p2 schedule to be released at end of the month

Sent from Samsung Mobile"Klise, Steve" klises@sutterhealth.org wrote:

Was this noticed also on 8.1p1?  Looks like there were some good fixes in the p release.

Thanks in advance.

From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Alon Zeltser Sent: Thursday, July 26, 2012 11:01 AM To: toasters@teaparty.net Subject: Ontap 8.1 performance issue

Hi all As a netapp ASP I spent quite alot of my time installing netapp's at costumers  site and resolving their issues and I wanted to share  my recent experience with 8.1 Usually I try to avoid RC release but until recently the fas22xx didn't gave us much choice Once the 8.1 GA released I felt more free to install this release.  but after I started to get performance complains from my costumers I wasn't so sure After investigating some of the complains I felt  that they  are justified something is off in system  I can't put my finger on it but its looks the system don't perform as expected even though the system don't look busy. After opening support cases at netapp one of the engineers told me that they also gets alot of performance  cases with this release an  even though he can't find a bug that feets with our logs and perfstat he recommend upgrade to 8.1.1rc1 Following his advise after upgrade  I saw major performance improvment I mean throughput went up from 70-80 MB/s to 250-300 MB/s  and latency also went down The same result occurred on more then one 8.1 system after upgrade to 8.1.1rc1 I saw at least 50% improvment in throughput  Today on new system I install I completely skip th  GA version an  go straight to 8.1.1rc1 So my advice if you on this release  and suffering from performance and don't know if you have enough disks or the right controller upgrade  first then check

Sent from Samsung Mobile

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters