toasters

toasters@lists.teaparty.net

1 participants
13515 discussions

test sending
by Luca Domenella 25 Aug '11

25 Aug '11

test Luca Domenella T: +39 0698962316 E: <mailto:luca.domenella@bwinparty.com> luca.domenella(a)bwinparty.com bwin Italia Via Adolfo Ravà, 124 00142 Roma (RM) <http://www.bwinparty.com/> www.bwinparty.com This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.

1 0

crashes due to corrupt volume (now resolved by upgrade)
by William Yardley 25 Aug '11

25 Aug '11

Along the lines of posting stuff we tried to post to the list in the past, here's some information about a situation we had recently, where our NetApp was crashing over and over, resulting in 4 or so consecutive sleepless nights for me. The situation was resolved in the short term by upgrading Data OnTAP from 7.1.1 (built 6/2006) to 7.3.3. Sadly, this was also the impetus to speed up migrating the data to our new EMC based infrastructure (not my decision). Posting here just in case it's helpful to anyone. Interestingly, the errors seemed to continue after the upgrade, however something dumped core, but the system didn't crash; note also that the debugging information is more specific about which volume the problem is on than in the log messages from before the upgrade: Wed Jul 27 19:34:51 PDT [raid.data.ws.blkErr:error]: WAFL sanity check detected bad data on volume acs, Disk /aggr0/plex0/rg1/0a.41 Shelf 2 Bay 9 [NETAPP X274_S10K7146F10 NA07] S/N [3KS0R58B000075418D6B], block 3765221, inode number -45, snapid 0, file block 690030, level 1. Wed Jul 27 19:34:51 PDT [raid.data.lw.blkErr:CRITICAL]: Bad data detected on Disk /aggr0/plex0/rg1/0a.41 Shelf 2 Bay 9 [NETAPP X274_S10K7146F10 NA07] S/N [3KS0R58B000075418D6B], block #3765221 Wed Jul 27 19:34:51 PDT [wafl.inconsistent.indirect:error]: Bad indirect block in vol 'acs', snap 13, fbn 690030, level 1, fileid -1. Wed Jul 27 19:34:51 PDT [wafl.inconsistent.vol:error]: WAFL inconsistent: volume acs is inconsistent. Note: Any new Snapshot copies might contain this inconsistency. Wed Jul 27 19:34:51 PDT [wafl.raid.incons.buf:error]: WAFL inconsistent: bad block 143002981 (vvbn:97941303 fbn:690030 level:1) in inode (fileid: 4294967251 snapid:0 file_type:1 disk_flags:0x2) in volume acs. Wed Jul 27 19:34:51 PDT [coredump.micro.completed:info]: Microcore (/etc/crash/micro-core.101186401.2011-07-28.02_34_51) generation completed Sorry about the unfinished sentence at the beginning. I was running on very little sleep at the time. ----- Forwarded message from William Yardley <toasters(a)veggiechinese.net> ----- From: William Yardley <toasters(a)veggiechinese.net> To: toasters(a)mathworks.com Subject: crashes due to corrupt volume? I have a FAS3020 which has been crashing repeatedly. It has a single aggregate (aggr0) with Doing WAFL_check fixed a few errors, but after that ran clean. wafliron from the maintenance menu also came back clean. However, we kept seeing errors like this, even after replacing the disks which were supposedly having problems (leading me to believe it's something on the volume that's corrupt rather than a physical problem). We did have a disk failure, and that disk already rebuilt to the spare. The problem Mon Jul 25 02:19:44 PDT [raid.data.ws.blkErr:error]: WAFL sanity check detected bad data on volume aggr0, Disk /aggr0/plex0/rg1/0a.41 Shelf 2 Bay 9 [NETAPP X274_SCHT6146F10 NA08] S/N [3HY8FV2S00007418EU6V], block 3765221, inode number -45, snapid 0, file block 690030, level 1. Mon Jul 25 02:40:10 PDT [raid.data.lw.blkErr:CRITICAL]: Bad data detected on Disk /aggr0/plex0/rg1/0a.41 Shelf 2 Bay 9 [NETAPP X274_SCHT6146F10 NA08] S/N [3HY8FV2S00007418EU6V], block #3765221 Mon Jul 25 02:40:10 PDT [raid.rg.readerr.repair.data:notice]: Fixing bad data on Disk /aggr0/plex0/rg1/0a.41 Shelf 2 Bay 9 [NETAPP X274_SCHT6146F10 NA08] S/N [3HY8FV2S00007418EU6V], block #3765221 [and later, note a few different SNs after we replaced the two drives which the errors were showing up on -- we did this by doing a fail -i on the drives, and then reconstructing one at a time.] Mon Jul 25 21:44:45 PDT [raid.data.ws.blkErr:error]: WAFL sanity check detected bad data on volume aggr0, Disk /aggr0/plex0/rg1/0a.40 Shelf 2 Bay 8 [NETAPP X274_SCHT6146F10 NA08] S/N [3HY8FRM000007451DZZ2], block 3765221, inode number -45, snapid 0, file block 690030, level 1. Tue Jul 26 04:15:29 PDT [raid.data.lw.blkErr:CRITICAL]: Bad data detected on Disk /aggr0/plex0/rg1/0c.41 Shelf 2 Bay 9 [NETAPP X274_S10K7146F10 NA07] S/N [3KS0R58B000075418D6B], block #3765221 Tue Jul 26 04:15:29 PDT [raid.rg.readerr.repair.data:notice]: Fixing bad data on Disk /aggr0/plex0/rg1/0c.41 Shelf 2 Bay 9 [NETAPP X274_S10K7146F10 NA07] S/N [3KS0R58B000075418D6B], block #3765221 also a bunch of these: Tue Jul 26 04:16:30 PDT [wafl.dir.inconsistent:error]: wafl_nfs_lookup: inconsistent directory entry {x20 13 14495183 105181080 -520015383} <C7DA730C028D> in {x20 13 2781660 870430057 -520015383}. WAFL may be inconsistent. Call NetApp Support. Tue Jul 26 04:16:30 PDT [wafl.dir.inconsistent:error]: wafl_nfs_lookup: inconsistent directory entry {x20 13 14495184 105181084 -520015383} <C7DDF30B0808> in {x20 13 2781660 870430057 -520015383}. WAFL may be inconsistent. Call NetApp Support. The parity inconsistency files in etc/crash/ show something like: Checksum mismatch on disk /aggr0/plex0/rg1/0a.41 (S/N 3HY8FV2S00007418EU6V), block #3765221 (sector #0x1cb9f28). Checksum mismatch found on Mon Jul 25 02:30:00 PDT 2011 Expected block: [...] The system would often crash, and then crash again on reboot -- booting without /etc/rc and *then* enabling services seemed to help, though the system would usually crash again within hours. After looking at a (very) old post on this list, I tried disabling quotas on the active volumes; this seems to have possibly helped, but I'm nervous that the system will crash again. Short of restoring to a snapshot from before the problem started, is there anything else (online or offline) I can do to fix any problems that might be on the volume, or manually remove the problematic block(s)? Some other info, in case it helps: files*> aggr status -v Aggr State Status Options aggr0 online raid_dp, aggr root, diskroot, nosnap=off, raidtype=raid_dp, raidsize=16, snapmirrored=off, resyncsnaptime=60, fs_size_fixed=off, snapshot_autodelete=on, lost_write_protect=on Volumes: m_hosts, luns, acs, root, tmp, software, home, legacy_mail, x_tmt, backups, dropbox, netflow2, xen_backup Plex /aggr0/plex0: online, normal, active RAID group /aggr0/plex0/rg0: normal RAID group /aggr0/plex0/rg1: normal files*> aggr status -r Aggregate aggr0 (online, raid_dp) (block checksums) Plex /aggr0/plex0 (online, normal, active) RAID group /aggr0/plex0/rg0 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 0a.22 0a 1 6 FC:A - FCAL 10000 136000/278528000 137104/280790184 parity 0a.32 0a 2 0 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0a.23 0a 1 7 FC:A - FCAL 10000 136000/278528000 139072/284820800 data 0a.39 0a 2 7 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.35 0c 2 3 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0c.26 0c 1 10 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.45 0a 2 13 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.18 0c 1 2 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0c.28 0c 1 12 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0c.37 0c 2 5 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.24 0a 1 8 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0a.43 0a 2 11 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.29 0c 1 13 FC:B - FCAL 10000 136000/278528000 137485/281570072 RAID group /aggr0/plex0/rg1 (normal) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 0a.17 0a 1 1 FC:A - FCAL 10000 136000/278528000 137422/281442144 parity 0a.33 0a 2 1 FC:A - FCAL 10000 136000/278528000 139072/284820800 data 0c.41 0c 2 9 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.38 0a 2 6 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0a.36 0a 2 4 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.27 0c 1 11 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.21 0a 1 5 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0a.34 0a 2 2 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.19 0c 1 3 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.20 0a 1 4 FC:A - FCAL 10000 136000/278528000 137104/280790184 data 0c.44 0c 2 12 FC:B - FCAL 10000 136000/278528000 137422/281442144 data 0c.16 0c 1 0 FC:B - FCAL 10000 136000/278528000 137104/280790184 data 0a.42 0a 2 10 FC:A - FCAL 10000 136000/278528000 137104/280790184 Spare disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- Spare disks for block or zoned checksum traditional volumes or aggregates spare 0c.25 0c 1 9 FC:B - FCAL 10000 136000/278528000 137104/280790184 spare 0c.40 0c 2 8 FC:B - FCAL 10000 136000/278528000 137104/280790184 ----- End forwarded message -----

1 0

Netapp 3040 Ontap 7.3.3 latency spikes - crashes vmware
by Fletcher Cocquyt 25 Aug '11

25 Aug '11

Originally attempted to post June 22, 2010! - thanks for fixing the list - We have since identified the issue by deconstructing the IOPS behind the latency spikes and resolved per: http://www.vmadmin.info/2010/07/vmware-and-netapp-deconstructing.html Hope it proves useful for anyone else with similar issues On 6/22/10 10:19 AM, "Fletcher Cocquyt" <fcocquyt(a)stanford.edu> wrote: Hi, We have a 3040 cluster hosting 11 vSphere hosts with 200 VMs on NFS datastores. We see latency spikes 3-4 times a month as reported by Operations Manager. We hoped our upgrade from 7.3.1.1 last week to 7.3.3 would help, but we¹ve had many spikes up to 1 second take out a NFS mount and all several of the VMs since going to 7.3.3 We previously determined the High & medium IO VMs and either aligned them or migrated them to local disk - has NOT helped - still getting the spikes. I have another case opened with Netapp. Following the notes in this latency spike related thread, http://communities.netapp.com/message/30657 I ran the wafl_susp -w to check the pw.over_limit Turns out ours is ZERO (is it relevant to NFS?) I suspect an internal Netapp process is responsible for these (dedup?) - we had it disabled on 7.3.1.1 - 7.3.3 was supposed to fix this (we re-enabled de-dup after the upgrade) And the latency spike outages are back Will share any info from the case thanks for any tips, -- Fletcher Cocquyt Principal Engineer Information Resources and Technology (IRT) Stanford University School of Medicine http://vmadmin.info

1 0

checking if new list working
by Uddhav Regmi 25 Aug '11

25 Aug '11

6 6

Command syntax
by Jeff Cleverley 24 Aug '11

24 Aug '11

Greetings, I haven't been able to post for quite a while so I'm quite happy things look good (assuming my post goes through). I thought I'd try breaking up the discussions about the domain name and test emails with a NetApp question :-) I was reading comments about reallocate with regards to a blog and someone posted this command to run: "stats show disk:*:disk_busy" I don't recall ever seeing anything about special constructs or functionality using the colon delimited fields. If this is available I'm guessing there might be other cool tricks available also. Does anyone know where there is documentation on how to use this? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

4 4

RE: Hosting and branding of toasters list (was "Re: Google Groups?")
by Reginald Jefferson 24 Aug '11

24 Aug '11

Chris - toasters.net is a great idea that seems hard to disagree with. teaparty.net is fine with me too. Tom - THANK YOU for hosting this list for all of us. Reggie Reginald Jefferson Sr. NetApp Support Engineer Volo Labs reginald.jefferson(a)vololabs.com ________________________________ From: toasters-bounces(a)teaparty.net on behalf of Christopher B. Allison Sent: Wed 8/24/2011 8:52 AM To: toasters(a)teaparty.net Subject: Hosting and branding of toasters list (was "Re: Google Groups?") I read Brian's proposal as three separate arguments: 1) The toasters mailing list should not be hosted on a domain which many people would probably associate with a political group. ("sends a political message") 2) The toasters mailing list should not be hosted by an organization which is smaller in size than Google Inc. ("moving from one boutique mailing list hosting service to another") 3) Google Inc the best choice to host the toasters mailing list. ("moving from one boutique mailing list hosting service to another") Several people have replied to Brian's initial email with some variant of "Agreed". To those people I ask: To which parts (some? all?) of Brian's suggestion are you agreeing? I'm inclined to concur with point number one. I realize that Tom has held registration of teaparty.net since before the onset of the current political movement and that (as per the disclaimer at the top of http://teaparty.net/) he disclaims any and all affiliation with said political movement - a point which it seems may have been lost on one or two respondents - but I agree with Brian that the domain name has the potential to be misinterpreted by others who (as such) might be disinclined to consider joining the list or who might otherwise be confused as to the list's purpose or tenor. That being said, I do not agree with points two and three. I think it is lovely that toasters has moved back to its initial home on the internet (and thank you to the original founders, everyone who has provided hosting over the years, and the current host for all that you have done!), and I see no need to move the list to Google Groups or any other host. It appears that the domain name 'toasters.net' is available for registration, and I would be happy to register it on Tom's behalf, if he is so interested. Tom? Respectfully, -Chris Allison -- UNIX Systems Administrator College of Computer and Information Science Northeastern University, Boston, MA _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

thank you Tom
by Coder, James (STP) 24 Aug '11

24 Aug '11

Tom - thanks for taking on the task of moving the toasters. job well done. jc James C. Coder "The saddest words of tongue and pen are not - 'It might have been' but- I DIDN'T THINK." - Waite Phillips Information Systems Principal Global Storage Team Boston Scientific Corporation 4100 Hamline Avenue North, M/S 7-309 St. Paul, MN 55112 email: james.coder(a)bsci.com www.bostonscientific.com Normal Hours - M/F - 7:00AM-4:00PM CDT Tel: 651-582-4797 Cell: 612-747-5377 Fax 651-581-3519 "Any views expressed in this e-mail are not those of Boston Scientific Corporation."

2 1

List is sluggish and takes some time to send posts out
by Tom Yates 24 Aug '11

24 Aug '11

As some have noticed, the list engine has been backlogging posts to the list today; in some cases, email hasn't been sent out until 60-120 minutes after it was received. I'm sorry about this. I think I've identified the "feature" of mailman that was causing this, and changed it. When I tested my mailman to see if it could cope with the toasters traffic, I didn't anticipate seventy posts in a single day. The backlog is being rapidly processed, and we should soon have caught up. -- Tom Yates - http://www.teaparty.net

1 0

TOASTERS LIST IS MOVING: PLEASE READ
by Todd C. Merrill 24 Aug '11

24 Aug '11

Greetings. The toasters list is moving! As of next Wednesday August 24, the toasters list will be moving from its current address at toasters(a)mathworks.com to toasters(a)teaparty.net. (That domain name was in place long before it had the recent political meaning in the United States!) For those with knowledge of the history of the toasters list, it will be returning to its roots and will be hosted by the founding member of the toasters list. "Why is this happening?" The Mathworks is no longer supporting external facing email lists. "What do I have to do?" On the cutover date, we plan to take the existing member list and use it at the new address, so there is nothing you have to do to un/subscribe. However, after the cutover you may need to update your address books or spam filters for sending or receiving email from the new address (toasters(a)teaparty.net). If you'd like to unsubscribe and not be transferred over, simply send a text-only email (no HTML!) before Aug 24th to the old list address toasters(a)mathworks.com (a majordomo list) with a blank subject and the email body containing the words: unsubscribe toasters end If you would possibly like to subscribe later, simply send an email to the new list address toasters-subscribe(a)teaparty.net (a mailman list) after Aug 24th. If you prefer to maintain your subscription via a web interface, it can be found at http://www.teaparty.net/mailman/listinfo/toasters after Aug 24th. We'd like to point out that the NetApp website hosts a set of communities at http://www.netapp.com/us/communities/; these offer valuable information in addition to the independent toasters mailing list. Sincerely, Toasters List Administrators: Todd Merrill Kevin Davis Tom Yates

9 10

Test only, pls ignore
by IT.INF.DC.g.Eko Haryono 24 Aug '11

24 Aug '11

Test only, pls ignore Eko Haryono IT Infrastructure Division

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters