toasters

toasters@lists.teaparty.net

13531 discussions

6290 10G ports vs PCI 10G cards
by Jeff Cleverley 19 Jun '13

19 Jun '13

Greetings, I haven't been able to find out anything about the bus speeds for the ports on a 6290. The hardware universe shows an order to fill slots for things like 10G or SAS controllers. This makes me believe all the ports are not the same. I was thinking of using the on-board 10G ports instead of a dual port PCI card that would be in one of the top 5 priority slots. We do have the IOX module on the arrays. Does anyone know where I can find this information? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

1 0

roles an capabilities in ONTAP: the missing link in security-*
by Michael Bergman 10 Jun '13

10 Jun '13

Greeting, does anyone know what the secret (undocumented) security-* capability which allows you to elevate to diag level and see those CLI commands, i.e. running 'priv set diag', is called? The man pages for 8.1.x tells us this: - - - The security-* type currently only has a few elements (5): security-passwd-change-others which is used specifically to control if a user can change another user's password without knowing their previous password. By default, only root and members of the Administrators group have this capability. security-priv-advanced which is necessary to run advanced commands that are not used for normal administration. Please talk to a NetApp Inc representative before using advanced commands. By default, only root and members of the Administrators group have this capability. security-api-vfiler Normally a client will send ONTAP APIs directly to a vfiler if it wishes the API to be executed on the vfiler. The security-apivfiler capability is necessary to send ONTAP APIs to the physical node which are to be forwarded to a vfiler for execution. By default, only root and members of the Administrators group have this capability. security-load-lclgroups which is necessary to run the useradmin domainuser load command. This command changes all group membership. By default, only root and members of the Administrators group have this capability. security-complete-user-control which is used to allow an admin to add, modify, and delete users, groups and roles with more capabilities than himself. These users typically only have access to the cli-useradmin* and associated commands, though they can give themselves greater permissions. By default, only root and members of the Administrators group have this capability. - - - So that's fine and setting security-priv-advanced gives the results I expected. But it will not allow you to execute 'priv set diag'. However, setting up a role with capability security-* will, so there's at least one more in there with a name unknown to me. I tried a few educated guesses like security-priv-diag security-priv-diagnostics security-diag security-diagnostics security-priv-* security-priv-advdiag security-priv-advanced-diag plus a some more along those lines, to no avail. I'm assuming there is something in there that has a specific name -- this assumption could be wrong of course /M -- Michael Bergman Sr Systems Analyst / Storage Architect michael.bergman a ericsson d com Engineering Hub Stockholm Phone +46 10 715xxxx Service Delivery, Engineering EMEA N SMS/MMS +46 70 548xxxx Ericsson Torshamnsg 33, 16480 Sthlm, Sweden

1 1

clustered ontap monitoring snapmirror lag?
by Blake Golliher 10 Jun '13

10 Jun '13

Hi! It looks as if the concept of a 'snapmirror lag' has been scrubbed from Clustered OnTap, at least on 8.1.2P3. 'snapmirror show' will show the state, and transferring, but there's no lag, or a start time that I can extrapolate. There's a progress, which reports what's been transferred, but no lag. The api documentation doesn't have any mention of snapmirror lag in snapmirror-info attributes-list. There's no support of snapmirror in snmp (at least so far). So does DFM (or whatever they call it now) monitor this? And how would it? Thanks for any and all suggestions. If I find where this data is, I'll put the script on github. -Blake

3 7

Fixing LUN Misalignment on Physical Machines
by Philbert Rupkins 08 Jun '13

08 Jun '13

Hello Toasters, I found a ton of information regarding fixing misalignment issues in virtual environments. However, I am interested in how to resolve misalignment on a physical machine. The only way I am aware of is to create a new LUN and perform a host based copy of the data to the new LUN. Is this my only option or are there any other methods that require less overhead/downtime? Here is the situation: Server OS = Windows 2008 Server LUN Type = Windows 2008 Partition Type = MBR (W2K3) How did we get a W2K3 MBR partition out of a Windows 2008 server? We didnt. The LUN was originally created by a W2K3 machine while using an EMC storage array. The W2K3 server was eventually retired and the LUN was presented to the W2K8 machine. The LUN was then migrated to a NetApp array using a block for block mirror of the data from the EMC LUN to the NetApp LUN. Unfortunately, the NetApp LUN was created as a windows_2008 lun type because it was going to be used by a Windows 2008 server - this is where the mistake was made. So now, we have a Windows Server 2008 box with a disk using a 2003-MBR partition type on windows_2008 lun type. Again, I'd like to fix the misalignment issue in the easiest way possible. Creating a new lun and performing a host based copy of the data is doable but requires a fair amount of overhead. Just wondering if there is a more efficient way to resolve lun alignment issues with physical machines. Thanks! -Phil

4 4

SAS port speeds
by Jeff Cleverley 07 Jun '13

07 Jun '13

Greetings, We have a stack of DS4243 shelves for a NearStore running SATA/BSAS drives. We will be getting some DS4246 with the 4TB FSAS drives. I know it is supported to put them in the same stack, my understanding is everything in the stack will drop to 3 GB/s speed. Since the HBAs run 2 ports on one chip, does that one chip control the speed for both ports A/B, or are they separate? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

2 2

growing a vol on vFiler that is a source in Vfiler DR relationship
by Iluhes 05 Jun '13

05 Jun '13

Hello Toasters, What is the best procedure to grow a volume on the vFiler that is a source in VFiler DR setup. Is there a need to break / queues replication somehow

6 7

Performance Advisor Reporting on V-Series Initiator Ports?
by Philbert Rupkins 24 May '13

24 May '13

Hello, Anybody know if there is any way to report on a V-Series initiator ports from Performance Advisor? Looks like Performance Advisor only captures metrics for target ports on the NetApp. The best I can find is enabling the "hostadapter" metrics in a custom view. Would be nice if PA included an object for V-Series initiators out of the box along with some expanded counters. Hopefully I am just missing something. Thanks, Phil

1 0

Boot version info.
by Jeff Cleverley 24 May '13

24 May '13

Greetings, I'm replacing a 3140 with a 3270 stand-alone NearStore replacement. The current 3140 OS is 8.1.2P3 and the new head shipped with 8.1.2. Is that going to be close enough for the system to boot from the old system root disk, or does it have to be an exact match? If there is a way to modify the boot config from the command line instead of pulling the compact flash, I'd be OK with that also. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

3 8

VM Virtual Disk Latency Versus NFS Latency on Filer
by Philbert Rupkins 22 May '13

22 May '13

Hello, This may be more of a VMWare question. From within Data Fabric Manager, I am seeing a spike in latency on a NetApp volume used as an NFS datastore. DFM shows a spike in NFS latency of 50 ms on the volume. However, a virtual machine's performance counters in vSphere show a spike in Virtual Disk latency upwards of 700ms for the same "latency event". Any idea what might account for this difference in latency reporting between the NetApp (DFM) and ESXi? I expected there to be some difference for obvious reasons but I didnt expect to see a difference of this magnitude. Thanks, Phil

3 5

Hot adding DS14MK4 trays to existing loop
by Fletcher Cocquyt 22 May '13

22 May '13

This is something I've never had reason to attempt until now. We have a 3270 cluster (8.1.2) Head 2 aggr1 consists of these two multipathed DS14MK4 trays (IDs 1,2): http://i.imgur.com/821iN35.jpg We want to hot add another DS14MK4 trays racked right above (IDs 3,4): http://imgur.com/FlMxpyz I've read the 2011 DS14MK4 guide and verified the loop speeds are all set consistently and IDs are all unique. (note: trays IDs are 1-4 from bottom to top) Can I simply: 1 - remove the top two pairs of fiber and transceivers (these go to head 1 (partner)) 2 - daisy chain up from those vacated ports to new trays (3,4) 3 - plug the fiber and tranceivers back in on the top tray? We would then grow (effectively double) this aggr1's capacity What are the risks ? (2011 guide mentions panic's are possible with loop misconfig) (we have 60 VMs running on head 2 on a separate loop and aggregate) Any other tips appreciated thanks

5 4

Oracle RAC on Netapp unstable under load
by Fletcher Cocquyt 20 May '13

20 May '13

Recently we have seen our Oracle RAC VMs reboot during netowkr and Netapp (Ontap 8.1.2 NFS) load correlated to our Friday 8pm full backup window start time. Initial strategies to prevent this include distributing RAC voting disks on different Netapp volumes, but I'd like to hear if others who have seen this have better solutions in practice? thanks, Fletcher

2 1

Delaying 64 bit aggregate upgrades
by Jeff Cleverley 17 May '13

17 May '13

Greetings, Is there a way to go from 7.3.5 to 8.1.2 without having all the aggregates start doing automatic conversions to 64 bit? Our current filers are getting hit pretty hard. I think the OS upgrade will help, but I'm afraid the aggregate upgrades will generate too much disk activity and slow things down for a day or two. We have ~9 aggregates per head that are 10T or larger. Most of these only have one volume in them. The shared ones have a few volumes with 40M+ inodes in use. If I can upgrade the aggregates selectively it will make it less noticeable to the users. Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

4 5

ONTAP 8.1.2P4 / Spike in Latency After Storage vMotion
by Philbert Rupkins 16 May '13

16 May '13

Hello, We were experiencing issues with large file deletes and spikes in latency from our Netapp filers. We run v3240's and FAS2240's. All are running 8.1.2GA or 8.1.1GA. The problem was only observed on those filers running 8.1.2GA. One operation that was observed to causing the problem consistently was Storage vMotions from an NFS datastore. In particular, the tail end of the Storage vMotion when the guest machine's files were deleted from the datastore seemed to trigger the problem. It looks like other people were experiencing the same issue. NetApp confirmed that a bug existed related to the deleting of large files. That issue was addressed in 8.1.2P3. We decided to deploy 8.1.2P4 as it included additional fixes related to deleting large files. Has anybody else deployed 8.1.2P4 yet? We just completed our deploy of 8.1.2P4 and it seems to have resolved the issue with large file deletes. We've tested a few Storage vMotions and removals of VM's with no issues thus far. If you are using 8.1.2P4, have you observed other issues? Thanks, Phil

3 4

FAS backup question
by Mike Gossett 15 May '13

15 May '13

Greetings: I have a general question in regards to backup solutions for NetApp filers. We have a client that is in the process of purchasing a FAS-2240-2 HA, and will be purchasing a FAS2220 for a DR site later this year. We are getting the complete license bundle to have SnapManager backups for VMWare, Exchange, etc to local disk, and licensing SnapMirror on the FAS2220 for replication of current data. It is our intent to implement some slower SATA disks to maintain longer retention windows. However, this environment is very interested in additional backup measures beyond that. Historically we have been using BackupExec, using agents on physical servers and VMs, to backup to both a NAS device and LTO autoloader. I am a little confused on how we can effectively accomplish an efficient backup of these filers outside of the netapp ecosystem. I have seen evidence of other users with older or larger models using direct-attach SCSI tape autoloaders (or FC autoloaders) with NDMP. We can't implement the SCSI cards in the 2240, and we don't wish to have the budgetary hit of implementing FC solely for tape connections. We were prefer to avoid tape altogether, if at all possible, as these filers will be in collocation facilities that we don't otherwise physically visit. Some other background: most\all of the data on these 2240s will be VMWare VMs stored on NFS (thin) datastores. There may be one or two iSCSI connections from VMWare for SQL and Exchange for SME reasons. The environment is a MS shop, with Sharepoint\SQL being the critical business application. Does anyone have any recommendations about other ways to backup our data? We hope to rely on the NetApp replication and SnapManager\SnapRestore\SnapVault for most of our normal recovery needs, however require the safety net of a weekly or monthly backup to some other media. We also hope to avoid something that is too difficult to manage, and were hoping for a commercial solution. Backup Exec with appropriate agents and a local NAS device or something along those lines is what I was thinking. Does anyone have any experience with tape devices connected to a media server backing up NetApp filers? Can NDMP stream the content to the NDMP backup server with a locally attached autoloader? How about some other sort of disk target (e.g. cheap NAS appliance connected to the backup server via cifs \ iSCSI) I've reached out to my SE, who indicated that most organizations are either using FC\SAS\SCSI autoloaders with NDMP or are relying on the built-in NetApp SnapVault solution. I would welcome any guidance or experience that anyone can share. Thanks, Mike Gossett

5 7

Re: RAID not reconstructing on FAS3050c
by Chris Daniel 10 May '13

10 May '13

Doug, Thank you so much for your detailed reply! I have deleted the core files, removed the bad disks (now dblade 'aggr status -f' is empty), and cleared the shelf fault. But nothing else. Did some more poking around, and yes I agree, the available spares seem like they should get used ... the list of spares says "Spare disks for block or zoned checksum traditional volumes or aggregates" and my degraded aggregate says it is "block checksum". All the disks are the same block sizes, etc. Now, I couldn't figure out how to turn off the "ignore inconsistent" flag. Here's what I tried: in the cluster shell (what you get when you log in as admin(a)ip.ad.dr.ess), I ran 'set -privilege advanced' (as that seems to be what your prompt indicated), and tried 'aggr modify ...' but 'aggr' was not available. Then I tried 'storage aggregate modify ...' but there was no -ignore-inconsistent switch available (just a few: -state, -raidtype, -maxraidsize). Last, I tried running 'aggr modify ...' under the node shell (what you get when you run 'dblade' from the cluster shell), and there was no 'modify' subcommand available -- just the following: media_scrub, read_fsid, show_space, wafliron, raidlabel, scrub, status. Also, one last funny thing: looking at 'aggr status -r' in dblade, the RAID group with the failed disk seems to be stuck at "recomputing parity 0% completed". If you have any other input you are willing to provide free, I would appreciate it loads! If you'd like to take this off list let me know, more than willing to pay your fee. Thank you! On Thu, May 9, 2013 at 6:48 PM, Doug Siggins <DSiggins(a)ma.maileig.com>wrote: > 1. I hope you have a backup of this data! please make sure you at least > have a mirror (on a different aggregate)! > > Before anything else get those failed disks out of there and save them for > a short bit, and yes you will need to run a WAFL check, although it should > still rebuild while inconsistent. > Delete some cores off of the head: > ngsh*> coredump delete ? this might take a few tries, as it looks like its > trying to save the core files :) > > > or you could always rm them from /mroot/kcores > I find that much quicker. > > Clear the shelf fault: > May 5 07:00:00 [statd]: monitor.shelf.fault: Fault reported on disk > storage shelf attached to channel 0b. Please check fans, power, and > temperature. > > > > Broken disks > > RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) > Phys (MB/blks) > --------- ------ ------------- ---- ---- ---- ----- -------------- > -------------- > failed 0a.23 0a 1 7 FC:A - ATA 7200 > 423111/866531584 423889/868126304 > failed 0b.41 0b 2 9 FC:A - ATA 7200 > 423111/866531584 423889/868126304 > failed 0b.44 0b 2 12 FC:A - ATA 7200 > 423111/866531584 423889/868126304 > failed 0c.53 0c 3 5 FC:B - ATA 7200 > 423111/866531584 423889/868126304 > failed 0c.57 0c 3 9 FC:B - ATA 7200 > 423111/866531584 423889/868126304 > failed 0c.60 0c 3 12 FC:B - ATA 7200 > 423111/866531584 423889/868126304 > > you can light up the failed disks like this: > storage disk setled -disk bosback01:0a.98 -action on > > -action off to turn them off :) > > > The message below does explain the issue with it not rebuilding the 520 > bps thing, though that is odd. they look iike the same darned disks. > Whomever administered this machine over rode the inconsistent flag (which > is a horrible idea). So essentially you have a volume online while the fs > is inconsistent, and can run into issues where it panics. Turn that flag > off. > > bosback01::*> aggr modify -aggregate bosback01_aggr1 -ignore-inconsistent > off > > I think you can then offline the aggregate and run the wafl iron > (dbladecli aggr wafliron) while the system is online. Of course its always > safer to boot into maint mode (ctrl c while booting) and then run the wafl > check. It's been so long since I've done that and I cannot login to maint > mode to try it :) > > https://kb.netapp.com/support/index?page=content&id=3013616 > > I'd need to research the 520 bps thing further to figure this out, though > try the options raid.disktype raid.whatever under the dblade. > > Of course if you run into deeper issues and you need some help I can be > had for a somewhat cheap consulting fee :) > > You have lots of issues here, and are a little precarious with data > integrity. Again I hope you have a backup of the data. Have fun. > > > > ------------------------------ > *From:* toasters-bounces(a)teaparty.net [toasters-bounces(a)teaparty.net] on > behalf of Chris Daniel [cjdaniel(a)gmail.com] > *Sent:* Thursday, May 09, 2013 8:44 PM > *To:* toasters(a)teaparty.net > > *Subject:* Re: RAID not reconstructing on FAS3050c > > Thanks all! So far I have been able to get into the node shell and > determine that the aggregate is marked as "wafl inconsistent". > > The disks are all the same size and model (SATA connected via FC, NetApp > X267_MGRIZ500SSX), except for one, which is model X267_HKURO500SSX -- it > appears to be of the same size as the rest, however, and is currently a > good data disk. > > Around the time of the failure, this message appears in the messages.ems > log: > [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start > in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size > available in spare pool > > That might explain why it's not reconstructing. The full messages.ems > log is here: http://pastebin.com/b9Xgj628 Also here is the output of > 'aggr status -r': http://pastebin.com/MmHFNbiz > > I'm guessing that what I need to do is run WAFL_check? Anything else I > should look at before I run off and do this? > > > > On Thu, May 9, 2013 at 2:03 PM, Doug Siggins <DSiggins(a)ma.maileig.com>wrote: > >> This is GX, he doesn't have the option to view the messages file >> without logging into the CLI: >> >> root@mgmt1 ip, cd /mroot/log >> there you can cat messages.log, and to view the messages.ems use >> ems_logviewer command: >> >> Last login: Wed May 8 20:01:17 2013 from 10.1.91.10 >> bosback01# cd /mroot/log >> bosback01# ems_logviewer messages.ems >> >> That will give some more information. Although I have never had this >> issue I have had GX from 10.0 to 10.0.4 In fact if you are still on GX, I >> have no idea why you would stick with 10.0.1, its horribly unstable! >> >> you can also get more information via storage disk show >> toast1a:0d.(number) >> >> One thing, those look like 450/500G disks. >> >> Are you using the same type of disk in the aggregate? (SATA/FC?) >> >> You may need to login to the dbladecli and allow the aggregate to use a >> different sized disk to rebuild onto (options disk or raid or something) >> >> Or the other thing I can think of is that you put an unsupported disk >> into that shelf. I know when I put in the 450G 15k FC I had to load up a >> new "unsupported" disk_qual package to see all of the space and use them. >> Which given Netapps congenial attitude towards screwing us GX users with >> FAS3050s didn't bother me the most to hack to get working :) >> >> >> I have been through a lot with GX/C-mode, and know quite a bit. I am >> looking for others who know things like zsmcli, and vldbtest :) >> >> >> >> ------------------------------ >> *From:* toasters-bounces(a)teaparty.net [toasters-bounces(a)teaparty.net] on >> behalf of tmac [tmacmd(a)gmail.com] >> *Sent:* Thursday, May 09, 2013 4:45 PM >> *To:* Chris Daniel >> *Cc:* Toasters >> *Subject:* Re: RAID not reconstructing on FAS3050c >> >> Try looking at it from the node perspective: >> (i forget the syntax offhand as it is slightly different that Ontap 8.1+ >> Cluster Mode) >> >> After you get into the node shell, do a disk show -n (make sure disks >> are properly assigned) >> >> try a aggr status -r >> >> Maybe the spares are zeroing before they are being used...that may be >> what pending is. >> >> Have you checked the event log? what about the messages file? >> >> --tmac >> >> *Tim McCarthy* >> *Principal Consultant* >> >> >> >> >> Clustered ONTAP >> Clustered ONTAP >> NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: >> C14QPHE21FR4YWD4 >> Expires: 08 November 2014 Expires w/release of RHEL7 >> Expires: 08 November 2014 >> >> >> >> On Thu, May 9, 2013 at 4:32 PM, Chris Daniel <cjdaniel(a)gmail.com> wrote: >> >>> Hello, >>> >>> We've got a couple of FAS3050c filers running 4 DS14mk2 shelves full of >>> disks (each filer is connected to two disk shelves). They've mostly been >>> trouble-free, but this week it seems a disk failed, and our main storage >>> aggregate went into "degraded" mode. For some reason, despite spare disks >>> being available, it's not reconstructing as I would think it should. >>> >>> The software running on these filers is Data ONTAP GX 10.0.1P2 -- from >>> previous discussions with the community, I've learned that GX has a whole >>> different set of commands, so many of the Google-able resources I've found >>> aren't relevant. Adding to that difficulty, we don't have a support >>> contract on these filers (but they are properly licensed and whathaveyou). >>> >>> Here is the output of 'storage aggregate show -aggregate engdata1' >>> (that is the degraded aggregate): >>> >>> toast1a::> storage aggregate show -aggregate engdata1 >>> >>> Aggregate: engdata1 >>> Size (MB): 0 >>> Used Size (MB): 0 >>> Used Percentage: - >>> Available Size (MB): 0 >>> State: restricted >>> Nodes: toast1a >>> Number Of Disks: 37 >>> Disks: toast1a:0a.16, toast1a:0b.32, toast1a:0c.48, >>> toast1a:0a.17, toast1a:0b.33, toast1a:0c.49, >>> toast1a:0a.18, toast1a:0b.34, toast1a:0c.50, >>> toast1a:0a.19, toast1a:0b.35, toast1a:0c.51, >>> toast1a:0a.20, toast1a:0d.64, toast1a:0b.37, >>> toast1a:0a.21, toast1a:0c.52, toast1a:0b.38, >>> toast1a:0a.22, toast1a:0c.61, toast1a:0b.39, >>> toast1a:0d.69, toast1a:0c.54, toast1a:0b.40, >>> toast1a:0a.24, toast1a:0c.55, toast1a:0d.65, >>> toast1a:0a.25, toast1a:0a.26, toast1a:0b.42, >>> toast1a:0c.59, toast1a:0a.27, toast1a:0b.43, >>> toast1a:0a.28, toast1a:0b.45, toast1a:0d.68, >>> toast1a:0d.71 >>> Number Of Volumes: 0 >>> Plexes: /engdata1/plex0(online) >>> RAID Groups: /engdata1/plex0/rg0, /engdata1/plex0/rg1, >>> /engdata1/plex0/rg2 >>> Raid Type: raid_dp >>> Max RAID Size: 14 >>> RAID Status: raid_dp,degraded >>> Checksum Enabled: true >>> Checksum Status: active >>> Checksum Style: block >>> Inconsistent: true >>> Volume Types: flex >>> >>> There are spare disks available now, but there were not when the >>> failure occurred. I moved two spare disks to the right filer after the >>> failure, thinking that would cause the aggregate to start reconstructing. >>> Here is the output of 'storage disk show -state spare': >>> >>> toast1a::> storage disk show -state spare >>> Disk UsedSize(MB) Shelf Bay State RAID Type Aggregate >>> Owner >>> ---------------- ------------ ----- --- --------- ---------- --------- >>> -------- >>> toast1a:0d.72 423090 4 8 spare pending - >>> toast1a >>> toast1a:0d.73 423090 4 9 spare pending - >>> toast1a >>> toast1b:0d.74 423090 4 10 spare pending - >>> toast1b >>> toast1b:0d.75 423090 4 11 spare pending - >>> toast1b >>> toast1b:0d.76 423090 4 12 spare pending - >>> toast1b >>> toast1b:0d.77 423090 4 13 spare pending - >>> toast1b >>> 6 entries were displayed. >>> >>> Can anyone provide insight on this problem? Why is the aggregate not >>> reconstructing when there are spares available? NetApp stuff is not my >>> specialty, but I'm the one who gets to deal with it, and I am pretty >>> stumped. Thank you in advance! >>> >>> -- >>> Chris Daniel >>> >>> _______________________________________________ >>> Toasters mailing list >>> Toasters(a)teaparty.net >>> http://www.teaparty.net/mailman/listinfo/toasters >>> >>> >> > > > -- > Chris Daniel > -- Chris Daniel

1 0

Re: RAID not reconstructing on FAS3050c
by Chris Daniel 10 May '13

10 May '13

Thanks all! So far I have been able to get into the node shell and determine that the aggregate is marked as "wafl inconsistent". The disks are all the same size and model (SATA connected via FC, NetApp X267_MGRIZ500SSX), except for one, which is model X267_HKURO500SSX -- it appears to be of the same size as the rest, however, and is currently a good data disk. Around the time of the failure, this message appears in the messages.ems log: [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size available in spare pool That might explain why it's not reconstructing. The full messages.ems log is here: http://pastebin.com/b9Xgj628 Also here is the output of 'aggr status -r': http://pastebin.com/MmHFNbiz I'm guessing that what I need to do is run WAFL_check? Anything else I should look at before I run off and do this? On Thu, May 9, 2013 at 2:03 PM, Doug Siggins <DSiggins(a)ma.maileig.com>wrote: > This is GX, he doesn't have the option to view the messages file without > logging into the CLI: > > root@mgmt1 ip, cd /mroot/log > there you can cat messages.log, and to view the messages.ems use > ems_logviewer command: > > Last login: Wed May 8 20:01:17 2013 from 10.1.91.10 > bosback01# cd /mroot/log > bosback01# ems_logviewer messages.ems > > That will give some more information. Although I have never had this issue > I have had GX from 10.0 to 10.0.4 In fact if you are still on GX, I have no > idea why you would stick with 10.0.1, its horribly unstable! > > you can also get more information via storage disk show toast1a:0d.(number) > > One thing, those look like 450/500G disks. > > Are you using the same type of disk in the aggregate? (SATA/FC?) > > You may need to login to the dbladecli and allow the aggregate to use a > different sized disk to rebuild onto (options disk or raid or something) > > Or the other thing I can think of is that you put an unsupported disk into > that shelf. I know when I put in the 450G 15k FC I had to load up a new > "unsupported" disk_qual package to see all of the space and use them. Which > given Netapps congenial attitude towards screwing us GX users with FAS3050s > didn't bother me the most to hack to get working :) > > > I have been through a lot with GX/C-mode, and know quite a bit. I am > looking for others who know things like zsmcli, and vldbtest :) > > > > ------------------------------ > *From:* toasters-bounces(a)teaparty.net [toasters-bounces(a)teaparty.net] on > behalf of tmac [tmacmd(a)gmail.com] > *Sent:* Thursday, May 09, 2013 4:45 PM > *To:* Chris Daniel > *Cc:* Toasters > *Subject:* Re: RAID not reconstructing on FAS3050c > > Try looking at it from the node perspective: > (i forget the syntax offhand as it is slightly different that Ontap 8.1+ > Cluster Mode) > > After you get into the node shell, do a disk show -n (make sure disks > are properly assigned) > > try a aggr status -r > > Maybe the spares are zeroing before they are being used...that may be > what pending is. > > Have you checked the event log? what about the messages file? > > --tmac > > *Tim McCarthy* > *Principal Consultant* > > > > > Clustered ONTAP > Clustered ONTAP > NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: > C14QPHE21FR4YWD4 > Expires: 08 November 2014 Expires w/release of RHEL7 > Expires: 08 November 2014 > > > > On Thu, May 9, 2013 at 4:32 PM, Chris Daniel <cjdaniel(a)gmail.com> wrote: > >> Hello, >> >> We've got a couple of FAS3050c filers running 4 DS14mk2 shelves full of >> disks (each filer is connected to two disk shelves). They've mostly been >> trouble-free, but this week it seems a disk failed, and our main storage >> aggregate went into "degraded" mode. For some reason, despite spare disks >> being available, it's not reconstructing as I would think it should. >> >> The software running on these filers is Data ONTAP GX 10.0.1P2 -- from >> previous discussions with the community, I've learned that GX has a whole >> different set of commands, so many of the Google-able resources I've found >> aren't relevant. Adding to that difficulty, we don't have a support >> contract on these filers (but they are properly licensed and whathaveyou). >> >> Here is the output of 'storage aggregate show -aggregate engdata1' >> (that is the degraded aggregate): >> >> toast1a::> storage aggregate show -aggregate engdata1 >> >> Aggregate: engdata1 >> Size (MB): 0 >> Used Size (MB): 0 >> Used Percentage: - >> Available Size (MB): 0 >> State: restricted >> Nodes: toast1a >> Number Of Disks: 37 >> Disks: toast1a:0a.16, toast1a:0b.32, toast1a:0c.48, >> toast1a:0a.17, toast1a:0b.33, toast1a:0c.49, >> toast1a:0a.18, toast1a:0b.34, toast1a:0c.50, >> toast1a:0a.19, toast1a:0b.35, toast1a:0c.51, >> toast1a:0a.20, toast1a:0d.64, toast1a:0b.37, >> toast1a:0a.21, toast1a:0c.52, toast1a:0b.38, >> toast1a:0a.22, toast1a:0c.61, toast1a:0b.39, >> toast1a:0d.69, toast1a:0c.54, toast1a:0b.40, >> toast1a:0a.24, toast1a:0c.55, toast1a:0d.65, >> toast1a:0a.25, toast1a:0a.26, toast1a:0b.42, >> toast1a:0c.59, toast1a:0a.27, toast1a:0b.43, >> toast1a:0a.28, toast1a:0b.45, toast1a:0d.68, >> toast1a:0d.71 >> Number Of Volumes: 0 >> Plexes: /engdata1/plex0(online) >> RAID Groups: /engdata1/plex0/rg0, /engdata1/plex0/rg1, >> /engdata1/plex0/rg2 >> Raid Type: raid_dp >> Max RAID Size: 14 >> RAID Status: raid_dp,degraded >> Checksum Enabled: true >> Checksum Status: active >> Checksum Style: block >> Inconsistent: true >> Volume Types: flex >> >> There are spare disks available now, but there were not when the >> failure occurred. I moved two spare disks to the right filer after the >> failure, thinking that would cause the aggregate to start reconstructing. >> Here is the output of 'storage disk show -state spare': >> >> toast1a::> storage disk show -state spare >> Disk UsedSize(MB) Shelf Bay State RAID Type Aggregate >> Owner >> ---------------- ------------ ----- --- --------- ---------- --------- >> -------- >> toast1a:0d.72 423090 4 8 spare pending - >> toast1a >> toast1a:0d.73 423090 4 9 spare pending - >> toast1a >> toast1b:0d.74 423090 4 10 spare pending - >> toast1b >> toast1b:0d.75 423090 4 11 spare pending - >> toast1b >> toast1b:0d.76 423090 4 12 spare pending - >> toast1b >> toast1b:0d.77 423090 4 13 spare pending - >> toast1b >> 6 entries were displayed. >> >> Can anyone provide insight on this problem? Why is the aggregate not >> reconstructing when there are spares available? NetApp stuff is not my >> specialty, but I'm the one who gets to deal with it, and I am pretty >> stumped. Thank you in advance! >> >> -- >> Chris Daniel >> >> _______________________________________________ >> Toasters mailing list >> Toasters(a)teaparty.net >> http://www.teaparty.net/mailman/listinfo/toasters >> >> > -- Chris Daniel

2 1

RAID not reconstructing on FAS3050c
by Chris Daniel 09 May '13

09 May '13

Hello, We've got a couple of FAS3050c filers running 4 DS14mk2 shelves full of disks (each filer is connected to two disk shelves). They've mostly been trouble-free, but this week it seems a disk failed, and our main storage aggregate went into "degraded" mode. For some reason, despite spare disks being available, it's not reconstructing as I would think it should. The software running on these filers is Data ONTAP GX 10.0.1P2 -- from previous discussions with the community, I've learned that GX has a whole different set of commands, so many of the Google-able resources I've found aren't relevant. Adding to that difficulty, we don't have a support contract on these filers (but they are properly licensed and whathaveyou). Here is the output of 'storage aggregate show -aggregate engdata1' (that is the degraded aggregate): toast1a::> storage aggregate show -aggregate engdata1 Aggregate: engdata1 Size (MB): 0 Used Size (MB): 0 Used Percentage: - Available Size (MB): 0 State: restricted Nodes: toast1a Number Of Disks: 37 Disks: toast1a:0a.16, toast1a:0b.32, toast1a:0c.48, toast1a:0a.17, toast1a:0b.33, toast1a:0c.49, toast1a:0a.18, toast1a:0b.34, toast1a:0c.50, toast1a:0a.19, toast1a:0b.35, toast1a:0c.51, toast1a:0a.20, toast1a:0d.64, toast1a:0b.37, toast1a:0a.21, toast1a:0c.52, toast1a:0b.38, toast1a:0a.22, toast1a:0c.61, toast1a:0b.39, toast1a:0d.69, toast1a:0c.54, toast1a:0b.40, toast1a:0a.24, toast1a:0c.55, toast1a:0d.65, toast1a:0a.25, toast1a:0a.26, toast1a:0b.42, toast1a:0c.59, toast1a:0a.27, toast1a:0b.43, toast1a:0a.28, toast1a:0b.45, toast1a:0d.68, toast1a:0d.71 Number Of Volumes: 0 Plexes: /engdata1/plex0(online) RAID Groups: /engdata1/plex0/rg0, /engdata1/plex0/rg1, /engdata1/plex0/rg2 Raid Type: raid_dp Max RAID Size: 14 RAID Status: raid_dp,degraded Checksum Enabled: true Checksum Status: active Checksum Style: block Inconsistent: true Volume Types: flex There are spare disks available now, but there were not when the failure occurred. I moved two spare disks to the right filer after the failure, thinking that would cause the aggregate to start reconstructing. Here is the output of 'storage disk show -state spare': toast1a::> storage disk show -state spare Disk UsedSize(MB) Shelf Bay State RAID Type Aggregate Owner ---------------- ------------ ----- --- --------- ---------- --------- -------- toast1a:0d.72 423090 4 8 spare pending - toast1a toast1a:0d.73 423090 4 9 spare pending - toast1a toast1b:0d.74 423090 4 10 spare pending - toast1b toast1b:0d.75 423090 4 11 spare pending - toast1b toast1b:0d.76 423090 4 12 spare pending - toast1b toast1b:0d.77 423090 4 13 spare pending - toast1b 6 entries were displayed. Can anyone provide insight on this problem? Why is the aggregate not reconstructing when there are spares available? NetApp stuff is not my specialty, but I'm the one who gets to deal with it, and I am pretty stumped. Thank you in advance! -- Chris Daniel

3 2

DataFabric Manager Question
by Ray Van Dolson 09 May '13

09 May '13

We have IBM-branded DataFabric Manager v5.0 installed. I *believe* this was replaced by OnCommand Core. Is that correct? Want to make sure I'm upgrading the right product and there isn't a newer version of DFM I should be using instead (haven't found one). Thanks, Ray

5 4

thin volume with autodelete config and snapmirror
by Iluhes 07 May '13

07 May '13

Hi Folks, I am trying to setup a pair on thin volumes that will be used in snapmirror relationship I was going to use these netapp recommend volumes Autodelete Configuration guarantee = volume LUN reservation = enabled fractional_reserve = 0% snap_reserve = 0% autodelete = volume / oldest_first autogrow = off try_first = snap_delete So I did the following vol create vol1-s none aggr0 550g vol options vol1 fractional_reserve 0 snap reserve -V vol1 0 vol autosize vol1 off snap autodelete vol1 on vol options vol1 try_first snap_delete Does it play nice with snapmirror? Does snap autodelete can potentially mess up snapmirror? Thank you!

1 0

DFM user password reset.
by Jeff Cleverley 06 May '13

06 May '13

Greetings, We have a DFM server running 4.0.2. I have a remote user who can't log into the web interface. He is listed as a user and has global view permissions. He shows up in the dfm user list output. It appears he has forgotten what his password is. I've been looking around for an hour and can't figure out how to change his password so he can get into the web page. Where is it doing the authentication from? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

3 3

AW: Enabling create_ucode on volumes afterwards
by Alexander Griesser 03 May '13

03 May '13

Hi Matt, thanks for your fast reply - in the manpages it says, that create_ucode needs to be on for FCP/iSCSI volumes - is that a wrong information? Bye, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-463-208501-320 Telefax: +43-463-208501-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Matt Hallmark [mailto:matt@cosmictoaster.com] Gesendet: Freitag, 03. Mai 2013 17:40 An: Alexander Griesser Betreff: Re: Enabling create_ucode on volumes afterwards You really only want it on for CIFS accessed volumes. One potential issue is with reading SnapMirror destinations without create_ucode being on, on both the source and destination volumes. You will get weird errors if not, and not be able to browse files at all on the destination. Matt On May 3, 2013, at 8:35 AM, Alexander Griesser <ag(a)anexia.at<mailto:ag@anexia.at>> wrote: Hey there, Today I was trying to find out, why with newer OnTap versions, new volumes created do not have the create_ucode = on setting anymore and I found out that it might have something to do what and when licenses are installed on the system. If no block level license (iscsi or fcp) is installed during setup, the root volume gets created with the create_ucode setting set to off and since all new volumes seem to inherit their settings from the root volume, they also will have create_ucode set to off. I did now set create_ucode = on on the root volume to make sure all new volumes get created with the option set to on also, that works, but I'm wondering if I need to be careful about setting create_ucode = on on already existing NFS, CIFS and iSCSI/FCP volumes. Does anyone know if that could cause troubles? Bye, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-463-208501-320 Telefax: +43-463-208501-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

1 0

Enabling create_ucode on volumes afterwards
by Alexander Griesser 03 May '13

03 May '13

Hey there, Today I was trying to find out, why with newer OnTap versions, new volumes created do not have the create_ucode = on setting anymore and I found out that it might have something to do what and when licenses are installed on the system. If no block level license (iscsi or fcp) is installed during setup, the root volume gets created with the create_ucode setting set to off and since all new volumes seem to inherit their settings from the root volume, they also will have create_ucode set to off. I did now set create_ucode = on on the root volume to make sure all new volumes get created with the option set to on also, that works, but I'm wondering if I need to be careful about setting create_ucode = on on already existing NFS, CIFS and iSCSI/FCP volumes. Does anyone know if that could cause troubles? Bye, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-463-208501-320 Telefax: +43-463-208501-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

1 0

Linux Host Utilities 4.1
by Peta Thames 03 May '13

03 May '13

Hi, Anyone have a dusty old copy of Linux Host Utilities 4.1 lying around? I need to build a test cluster with RH 4.6, and I know the real fix is to upgrade the OS, but there's a world of dependencies on the app running on these servers so we need a test cluster to be able to prepare for that. Thanks, Peta

2 2

Configuration Error Messages on FAS2220-Single Controller
by Alexander Griesser 02 May '13

02 May '13

Hi again, we are running a FAS2220 Single Controller at one of our sites which acts as a snapvault secondary only (so just for backup purposes, no high availability required). Since the system has been installed, I'm getting messages like this in /etc/messages: Wed May 1 23:00:00 CEST [netapp03:monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0a. Please check fans, power supplies, disks, and temperature sensors. Wed May 1 23:00:00 CEST [netapp03:monitor.shelf.configError:CRITICAL]: Enclosure services has detected an error in access to shelves or shelf configuration 0a. Config Advisor tells me, that there is just a single path to the disks which is correct and maybe the reason for these messages. The FAS2220 does not have any external disk shelves, just 12 Disks in the head. Since there is no second controller, I didn't do any external SAS cabling (like you do for the 2240 HA systems when there are no disk shelves attached), so what can I do to make this filer happy again? Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-463-208501-320 Telefax: +43-463-208501-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

2 3

Running on non-redundant power
by John Stoffel 02 May '13

02 May '13

Guys, We're in the process of starting to migrate from some 3140s running 7.3.6 to a 3220 cluster running 8.1.2, or whatever the latest release is. Going from 3.5 racks down to just 1 rack, woohoo! But I'm short of 30amp power plugs in my data center. So my plan is to drop one of my 3140s down to just two 30A circuits so I can bring up the 3220 cluster with fully redundant power. I know I need to turn off autosupport while this is happening, though I'd like to just turn off warning about power problems and keep auto support going for disk failures. Any suggestions or comments? Thanks, John

7 8

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters