toasters March 2013

toasters@lists.teaparty.net

26 participants
15 discussions

Aggregate Disk Busy 100% with volume IOPS low
by Fletcher Cocquyt 01 Apr '13

01 Apr '13

3270 cluster, OnTAP 8.1-7mode We are investigating a SATA aggregate showing repeated 5am disk 100% busy spikes without its volumes showing any corresponding IOPS spike as reported by Netapp Management Console (NMC). The 5am disk busy spikes correlate with very high latency on volumes on a different SAS aggregate. These volumes host VMs which then timeout, some needing reboots. Today when I heard from Netapp support after reviewing my perfstat the engineer reported this is expected since NVRAM buffers are shared btw aggregates. But when I dig further into the NMC stats I see the SATA aggregate disk busy actually corresponds to a DROP in IOPS on the 3 volumes hosted on the SATA aggregate - almost like some internal aggregate operations are starving out the external volume ops. I checked the snapshots (vol and aggr), snap mirror, dedup and none of the usual suspects were running. When I look at the NMC throughput graphs and switch on the legend - it shows a 5am READ blocks/sec spike corresponding perfectly to the disk busy. Where are these AGGR level READ operations coming from that are missing from the constituent volume IOPS, and in fact seem to be starving out volume level IO? I don't see much in the messages log, but will check the rest of the logs for internal type OPS thanks for any insight

13 24

iSCSI best practices
by Kalra, Manish 25 Mar '13

25 Mar '13

Hey guys, Are there any iscsi best practices with Netapp? I am going to setup Oracle DB on LUNs which will use iscsi. I am mostly concerned about redundancy and how iscsi behaves during filer failovers. Manish

2 1

Separating management and data traffic (e0M vs replication interfaces)
by Chris Picton 25 Mar '13

25 Mar '13

Hi all I have two pairs of 3210s. The default gateway of the systems are via the management interface IP range, so that they are reachable remotely on the e0M interfaces. However, this is causing snapmirror replication to use those interfaces as well, which is undesirable from a speed/data path persepctive. I have considered putting e0M into its own ipspace, but then how would I manipulate its routing table as it would not be in a vfiler. Any other ideas about having e0M reachable from anywhere, but still use a different vif as the default gateway for generic traffic on the system? Chris

6 7

RE: Separating management and data traffic (e0M vs. replication interfaces)
by Clark, Andre 22 Mar '13

22 Mar '13

What you can do is modify your rc file so that the other networks that are non-production would be defined first and set appropriate routes. Then leave defining e0M for after the others are defined. Also, depending on your ONTAP version, there are the options to limit/prevent protocol or snapmirror traffic over certain interfaces "options interface.xxxx.blocked". Regards, André M. Clark | Sr. Consulting Engineer, Team Lead | Insight Integrated Systems | 917.388.8236 Tell me I will forget... Show me I may remember... Involve me I WILL UNDERSTAND!!! Start by doing what's necessary, then what's possible, and suddenly you are doing the impossible!!! -----Original Message----- From: toasters-bounces(a)teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Chris Picton Sent: Thursday, March 21, 2013 11:31 To: toasters(a)teaparty.net Subject: Separating management and data traffic (e0M vs replication interfaces) Hi all I have two pairs of 3210s. The default gateway of the systems are via the management interface IP range, so that they are reachable remotely on the e0M interfaces. However, this is causing snapmirror replication to use those interfaces as well, which is undesirable from a speed/data path persepctive. I have considered putting e0M into its own ipspace, but then how would I manipulate its routing table as it would not be in a vfiler. Any other ideas about having e0M reachable from anywhere, but still use a different vif as the default gateway for generic traffic on the system? Chris _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

2 1

FCP FAN IN QUESTION
by Milazzo Giacomo 21 Mar '13

21 Mar '13

Hi all, we're setting up a very large FCP environment on a 6240HA with 4 target HBA each dedicated to that SAN. Hosts are IBM AIX LPAR each with 4 NPIV HBAs coming from two different VIOS. We will have more or less 300 LPARs. Both NetApp controllers and hosts are simmetrically zoned and divided on two FC switches, as usual. Well, the FC & iSCSI Configuration Guide report the limits for each family of controller and for 6200 one the FC port fan-in is 64. Is it really the maximum number of hosts they mean? Not the inititators number? I ask it because if you look at the current fcp stat you can see we already have roughly 40 initiators connected to each target and we'll quickly reach that limit. If that is the host number we can easily add more target ports to balance workloads...but if that number is "initiators" we're in trouble. Thanks in advance, Regards V6240A*> fcp stats Statistics for adapter 0c Read Ops: 142646 Write Ops: 5387410 Other Ops: 5501 KBytes In: 606619208 KBytes Out: 18650223 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 2 LIP Resets: 7 SCSI Requests Dropped: 0 Interrupts: 20804631 Spurious Interrupts: 0 Total Logins: 415 Total Logouts: 415 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 1 OLS Received: 8 Queue Depth: 0 Statistics for SFP on 0c Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53SSX Date Code: 12:42:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 442.5 (uWatts) SFP TX Optical Power: 460.1 (uWatts) Statistics for adapter 0d Read Ops: 0 Write Ops: 0 Other Ops: 0 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 0 LIP Resets: 0 SCSI Requests Dropped: 0 Interrupts: 4 Spurious Interrupts: 0 Total Logins: 0 Total Logouts: 0 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 0 Queue Depth: 0 Statistics for SFP on 0d Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM537SY Date Code: 12:24:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 0.0 (uWatts) (out of range) SFP TX Optical Power: 0.0 (uWatts) (out of range) Statistics for adapter 1a Read Ops: 56009868 Write Ops: 91968480 Other Ops: 7178759 KBytes In: 9625731996 KBytes Out: 1622330068 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 41 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 411105094 Spurious Interrupts: 0 Total Logins: 2143 Total Logouts: 2102 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 1a Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM5379D Date Code: 12:04:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 486.4 (uWatts) SFP TX Optical Power: 463.5 (uWatts) Statistics for adapter 1b Read Ops: 53939443 Write Ops: 48576966 Other Ops: 6672592 KBytes In: 1819149800 KBytes Out: 1220340034 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 47 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 242434524 Spurious Interrupts: 0 Total Logins: 2572 Total Logouts: 2525 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 1b Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53PGM Date Code: 12:17:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 619.2 (uWatts) SFP TX Optical Power: 463.3 (uWatts) Statistics for adapter 1c Read Ops: 0 Write Ops: 0 Other Ops: 5 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 587980 Spurious Interrupts: 0 Total Logins: 5 Total Logouts: 5 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 1c Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53V1V Date Code: 12:39:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 521.8 (uWatts) SFP TX Optical Power: 472.0 (uWatts) Statistics for adapter 1d Read Ops: 0 Write Ops: 0 Other Ops: 5 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 586249 Spurious Interrupts: 0 Total Logins: 5 Total Logouts: 5 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 1d Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM536P9 Date Code: 12:36:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 661.6 (uWatts) SFP TX Optical Power: 461.9 (uWatts) Statistics for adapter 0a Read Ops: 329944 Write Ops: 2818801 Other Ops: 949 KBytes In: 124038934 KBytes Out: 14749111 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 4 LIP Resets: 9 SCSI Requests Dropped: 0 Interrupts: 9006114 Spurious Interrupts: 0 Total Logins: 57 Total Logouts: 57 CRC Errors: 1 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 10 Queue Depth: 0 Statistics for SFP on 0a Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53NRB Date Code: 12:42:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 464.7 (uWatts) SFP TX Optical Power: 461.8 (uWatts) Statistics for adapter 0b Read Ops: 0 Write Ops: 0 Other Ops: 0 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 0 LIP Resets: 0 SCSI Requests Dropped: 0 Interrupts: 4 Spurious Interrupts: 0 Total Logins: 0 Total Logouts: 0 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 0 Queue Depth: 0 Statistics for SFP on 0b Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM537T0 Date Code: 12:28:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 0.0 (uWatts) (out of range) SFP TX Optical Power: 0.0 (uWatts) (out of range) Statistics for adapter 11a Read Ops: 56021935 Write Ops: 94502028 Other Ops: 7138201 KBytes In: 9659958208 KBytes Out: 1621021005 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 41 Link Breaks: 2 LIP Resets: 7 SCSI Requests Dropped: 0 Interrupts: 417616681 Spurious Interrupts: 0 Total Logins: 1925 Total Logouts: 1884 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 8 Queue Depth: 0 Statistics for SFP on 11a Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53SA4 Date Code: 12:18:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 440.0 (uWatts) SFP TX Optical Power: 463.0 (uWatts) Statistics for adapter 11b Read Ops: 53734018 Write Ops: 46687962 Other Ops: 6700517 KBytes In: 1799406809 KBytes Out: 1216393589 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 47 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 237637872 Spurious Interrupts: 0 Total Logins: 2443 Total Logouts: 2396 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 11b Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53V1Y Date Code: 12:16:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 639.0 (uWatts) SFP TX Optical Power: 470.1 (uWatts) Statistics for adapter 11c Read Ops: 0 Write Ops: 0 Other Ops: 6 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 2 LIP Resets: 7 SCSI Requests Dropped: 0 Interrupts: 587951 Spurious Interrupts: 0 Total Logins: 6 Total Logouts: 6 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 8 Queue Depth: 0 Statistics for SFP on 11c Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM538VV Date Code: 12:27:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 491.8 (uWatts) SFP TX Optical Power: 460.7 (uWatts) Statistics for adapter 11d Read Ops: 0 Write Ops: 0 Other Ops: 5 KBytes In: 0 KBytes Out: 0 Adapter Resets: 0 Frame Overruns: 0 Frame Underruns: 0 Initiators Connected: 0 Link Breaks: 1 LIP Resets: 6 SCSI Requests Dropped: 0 Interrupts: 586210 Spurious Interrupts: 0 Total Logins: 5 Total Logouts: 5 CRC Errors: 0 Adapter Qfulls: 0 Protocol Errors: 0 Invalid Transmit Words: 0 LR Sent: 0 LR Received: 0 Discarded Frames: 0 NOS Received: 0 OLS Received: 7 Queue Depth: 0 Statistics for SFP on 11d Vendor Name: FINISAR CORP. Vendor OUI: 0:144:101 Vendor PN: FTLF8528P2BNV Vendor Rev: A Serial No: PM53SS4 Date Code: 12:31:0 Media Form: SFP Connector: LC Wavelength: 850 (nm) Encoding: 8B10B FC Speed Capabilities: 2, 4, 8 Gbit/Sec SFP RX Optical Power: 607.2 (uWatts) SFP TX Optical Power: 465.4 (uWatts) Statistics for adapter vtic Read Ops: 246105 Write Ops: 2907529 Other Ops: 168582 KBytes In: 539680064 KBytes Out: 4695478 out_of_vtic_cmdblks: 0 out_of_vtic_msgs: 0 out_of_vtic_resp_msgs: 0 out_of_bulk_msgs: 0 out_of_bulk_buffers: 0 out_of_r2t_buffers: 0 Message: sent completed received VTIC_MESSAGE_L2P_NEW: 0 0 3322216 VTIC_MESSAGE_L2P_DATA_IN: 0 0 61561 VTIC_MESSAGE_L2P_DATA_IN_RESP: 0 0 271383 VTIC_MESSAGE_L2P_RESP: 0 0 3050832 VTIC_MESSAGE_L2P_DATA_OUT: 0 0 9196613 VTIC_MESSAGE_L2P_ABORT: 0 0 1 VTIC_MESSAGE_L2P_TASK_MGT: 112 112 1763 VTIC_MESSAGE_P2L_DATA_IN: 61561 61561 0 VTIC_MESSAGE_P2L_DATA_IN_RESP: 271383 271383 0 VTIC_MESSAGE_P2L_RESP: 3050833 3050833 0 VTIC_MESSAGE_P2L_R2T: 9196613 9196613 0 VTIC_MESSAGE_P2L_ABORT: 0 0 0 VTIC_MESSAGE_FCP_CONFIG_INITIALIZE: 3 3 0 VTIC_MESSAGE_FCP_CONFIG_UPDATE: 0 0 4 VTIC_MESSAGE_L2P_ADD_INITIATOR: 9576 9576 9056 VTIC_MESSAGE_L2P_REMOVE_INITIATOR: 9400 9400 8880 VTIC_MESSAGE_L2P_GET_LUN_LIST: 2139 2139 25 VTIC_MESSAGE_P2L_RETURN_LUN_LIST: 25 25 2139 VTIC_MESSAGE_DUMMY_UPDATE: 2213 2213 106 Dott. Giacomo Milazzo [cid:image001.jpg@01C75422.0D316270] Technical Account Manager Sinergy SpA Filiale di Roma * 00198. viale Regina Margherita, 269 ' (+39) 3406001045 0644245272 7 (+39) 0226922048 * Giacomo.Milazzo(a)Sinergy.it

1 0

Fingerprint is being recovered from volume (0% complete)
by Fletcher Cocquyt 21 Mar '13

21 Mar '13

So we migrated this vfiler to a 8.1.2 head in another datacenter yesterday na01> sis status Path State Status Progress /vol/vm64 Enabled Active Fingerprint is being recovered from volume (0% complete) na01> df -sh Filesystem used saved %saved /vol/vm64net2/ 3829GB 3051GB 44% Disk space is sawtoothing since the dedup job kicked off: I started growing the volume a coupe hours ago to keep it under 70% full. What is going on? thanks

2 1

Fingerprint is being recovered from volume (0% complete)
by Fletcher Cocquyt 21 Mar '13

21 Mar '13

446157 So we migrated this vfiler to a 8.1.2 head in another datacenter yesterday na01> sis status Path State Status Progress /vol/vm64 Enabled Active Fingerprint is being recovered from volume (0% complete) na01> df -sh Filesystem used saved %saved /vol/vm64net2/ 3829GB 3051GB 44% Disk space is sawtoothing since the dedup job kicked off: I've almost doubled the volume size. Experienced this with a smaller (700g) volume as well What is going on? Is there a way to avoid this ? thanks

2 2

Upgrading 32->64 bit aggregates/vols without adding disk
by Fletcher Cocquyt 21 Mar '13

21 Mar '13

Again in the position where my source aggregate is 32 bit and destination is 64 bit and this prevents vfiler online migration. Last year this was a priv set diag command aggr 64bit-upgrade start aggr1 -mode grow-all http://www.gossamer-threads.com/lists/netapp/toasters/11820 In test mode, this completed the 64 bit upgrade quickly and without error. Anyone done this in production? (Ontap 8.1) thanks

8 10

vfiler: useradmin user list root ... like to see the output
by Peter Kain 21 Mar '13

21 Mar '13

Can someone post from a vfiler the output of the command below, I'd like to see what you have for Allowed Capabilities and if it's a "*": > useradmin user list root On my vfiler I'm seeing nothing for "Allowed Capabilities" and a script I was using has stopped working. I can fix it by adding the group Administrators to root but I'd like to know what happen. blackhaw@FAS3240B> useradmin user list root Name: root Info: Default vfiler root Rid: 131073 Groups: Full Name: Allowed Capabilities: Password min/max age in days: 0/4294967295 Status: enabled NetApp Release 8.0.2P6 7-Mode: Fri Jan 27 14:48:08 PST 2012 Thanks, -- Pete Kain 310-506-4475

2 1

Changing qtree security type on snapvault destination.
by Jeff Cleverley 20 Mar '13

20 Mar '13

Greetings, How do you change the security style of a snapvault destination from ntfs to unix? The qtree is currently ntfs as is the primary qtree. When I try to change the security style it just says it change change it. How can I change it so I can see the qtree via nfs from our admin server? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters March 2013