toasters February 1999

toasters@lists.teaparty.net

106 participants
95 discussions

Fiber Channel filer failures
by Graham Knight 22 Feb '99

22 Feb '99

Last week we had a fiber channel F760 completely crash and burn. Right now the system is in a state where it will serve data for 10 or 15 minutes and then it reports inconsistent parity errors and crashes. We will be sending it to Netapp for failure analysis. That was a nightmare to say the least - and, of course, it happened while i was on vacation. (My first thought was that somebody was mad at me for flaming about the disk firmware thing and decided to take it out on me personally - yikes - i swear off flaming for the rest of my life). Anyway, last night another of our filers crashed with the same type of errors. This is an F630 with FC disks. I called NetApp - just got off the phone with them - they are looking into it, but before i lose another filer i thought i'd bounce this one off of the user community. These are the types of errors it spits out: Sun Feb 21 04:22:00 MST [isp2100_main]: isp2100_error_proc: dev 8.11 (0.11) data underrun (cmd opcode 0x28) ( 0x0 0x5c28 0x8000 ) Sun Feb 21 04:22:00 MST [isp2100_main]: Disk 8.11: Data underrun (0xfffffc0000c204f0,0x28,0) a few of those then a few of these: Sun Feb 21 04:22:00 MST [raid_stripe_owner]: Inconsistent parity on volume vol0, RAID group 2, stripe #377824. Sun Feb 21 04:22:00 MST [raid_stripe_owner]: Rewriting bad parity block on volume vol0, RAID group 2, stripe #377824. Then: Sun Feb 21 04:23:22 MST [edm_admin]: No valid paths to Enclosure Services in shelf 2 on ha 8. Sun Feb 21 04:23:22 MST [raid_stripe_owner]: Inconsistent parity on volume vol0, RAID group 2, stripe #377861. Sun Feb 21 04:23:22 MST [raid_stripe_owner]: Out of messages; cannot dump contents of inconsistent stripe. Sun Feb 21 04:24:00 MST [edm_admin]: No valid paths to Enclosure Services in shelf 3 on ha 8. Sun Feb 21 11:30:36 GMT [rc]: de_main: e10 : Link up. Sun Feb 21 04:30:37 MST [rc]: saving 44M to /etc/crash/core.0.nz ("wafl_check_vbns: vbn too big") Netapp thinks the previous failure may have been due to heat in the computer room - BUT - the filer *never* spit out any thermal warnings - I'm not too sure if i believe that heat was the culprit - i can run it for hours doing disk scrubs, but as soon as i use it for NIS it crashes. This system is in a different room and has a beter ventilated cabinet - again, no thermal warnings in the messages file - but maybe the thermal warning messages don't work? Just out of curiosity - has anyone ever seen a FC system spit out a thermal warning?? (5.1.2P2) Has anyone experienced these types of problem with FC filers? I've never had a problem with SCSI filers - but these FC ones just seem flaky to me.... Thanks for any insight anyone can lend... Graham

4 3

Re: Recovered error messages no longer logged in 5.1.2R3
by sirbruce＠ix.netcom.com 20 Feb '99

20 Feb '99

On 02/18/99 17:44:54 you wrote: > >> Does anyone else have an interest in having the flexibility of including >> soft error messages in the log files or is there a general consensus >> that they are unnecessary. We're trying to get support from other users >> for making the toggle available. > >Yes, I would like to have the toggle as well. In the first place, I would >like those messages to remain in the log file (I have not upgraded to 5.1 >yet but plan to do so next month). We have prevented disk crashes by >monitoring them. The toggle *should* already be in place. The messages should be logged at the syslog DEBUG level. If you specify a higher level, you shouldn't see them. One argument against this is there's a lot of other junk that can get logged at the DEBUG level. All this means is that some priorities need to be reorganized, and an acceptable default level found. Or you could break them up through useful facility designations. And simply inform old customers who may have turned on DEBUG just for these messages what the recommendation is for the new version. You don't see me criticizing netapp often, but I will here. They sometimes seem to bow to pressure to remove informational messages and options useful to "power users" in order to reduce the load on their support staff and keep up the appearance of simplicity. This seems to be what happened here. I also think it's pretty poor of them to make the change without telling their customers (was it documented?), or without even asking if that's what their customers wanted. Bruce

2 1

F740 performance: rule of thumbs?
by Todd C. Merrill 20 Feb '99

20 Feb '99

Our new F740 has a pretty constant load (`sysstat 10`) thusly: CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 46% 1114 1131 0 512 1331 701 371 0 0 5 53% 1559 1316 0 595 1615 446 387 0 0 6 50% 1343 1275 0 596 1211 489 464 0 0 5 48% 1454 1283 0 544 1272 276 395 0 0 6 52% 1844 1275 0 651 1837 280 320 0 0 6 58% 1399 1551 0 1027 2787 1037 450 0 0 6 56% 1796 1351 0 763 1541 352 378 0 0 6 45% 1024 1349 0 620 1422 330 542 0 0 6 58% 1657 1353 0 1193 2991 620 754 0 0 6 51% 1171 1467 0 598 1424 272 453 0 0 6 with occasional (every 5-10 minutes) sustained peaks of: 77% 1242 1201 0 489 1731 5285 5049 0 0 6 81% 1497 1279 0 600 2048 4327 5026 0 0 5 78% 1495 1221 0 563 1589 4287 5488 0 0 4 76% 1229 1213 0 472 1795 4546 5262 0 0 3 81% 1467 1150 0 1345 2634 3683 5746 0 0 2 80% 969 1068 0 397 2051 3876 6298 0 0 2 91% 1049 905 0 384 1102 1230 10622 0 0 2 92% 1328 942 0 448 1801 1536 10336 0 0 2 Are there any rules of thumb about when I should start worrying? I've got only about 2/3 of the data on it and the same fraction of clients connected to it that I need. The low cache numbers already worry me, but it's got the stated max of 1/2 GB. Max CPU and nfs-ops/sec I've seen is: 86% 3148 563 0 4228 5598 4131 3845 0 0 1 93% 4425 452 0 3573 4503 3283 3977 0 0 1 92% 3320 437 0 6299 4711 3778 5573 0 0 1 95% 3827 552 0 5282 5195 4094 5663 0 0 1 at which point I imagine the robot from "Lost In Space" coming out flailing its arms, if there existed the option "danger.will_robinson on". (Basic config is two 7 drive shelves of 18 GB FC-AL disks in a single RAID group with one spare and one volume and a quad 100 Mb/s board trunked into a 400 Mb/s fd pipe to the switch. I think I overspec'ed the network connection, though I have seen Net kB in and out burst and exceed 10,000 each, almost the limit of a single 100 Mb/s pipe). Until next time... Todd C. Merrill The Mathworks, Inc. 508-647-7792 24 Prime Park Way, Natick, MA 01760-1500 508-647-7012 FAX tmerrill(a)mathworks.com http://www.mathworks.com ---

3 2

Re: F740 performance: rule of thumbs?
by sirbruce＠ix.netcom.com 20 Feb '99

20 Feb '99

On 02/19/99 19:03:37 you wrote: > >On Thu, 18 Feb 1999, Todd C. Merrill wrote: >> >> with occasional (every 5-10 minutes) sustained peaks of: >> >> 77% 1242 1201 0 489 1731 5285 5049 0 0 6 >> 81% 1497 1279 0 600 2048 4327 5026 0 0 5 >> 78% 1495 1221 0 563 1589 4287 5488 0 0 4 >> 76% 1229 1213 0 472 1795 4546 5262 0 0 3 >> 81% 1467 1150 0 1345 2634 3683 5746 0 0 2 >> 80% 969 1068 0 397 2051 3876 6298 0 0 2 >> 91% 1049 905 0 384 1102 1230 10622 0 0 2 >> 92% 1328 942 0 448 1801 1536 10336 0 0 2 > > The inbound network traffic (less than 1MB/sec) doesn't jive with >your disk writes (5-10MB/sec). Are these not consecutive lines of >output, or are you recovering from a failed drive, or what? He said he was doing systat 10 earlier, I assume this are systat 10 stats as well. Some systat 1 output of typical and peak periods would be more helpful. However, my guess is he won't have much trouble adding another 1/3 users as indicated. NVRAM might have to be updgraded if these peak writing periods become too long. However, one should also consider other load this server might have on it in full production, like how it'll perform during a backup or a disk failure. If this hasn't been considered yet and you want to leave room for some growth then I would hesitate adding 50% more users. Bruce

2 1

RE: Shared Snapshot directory
by Benn, Paul 20 Feb '99

20 Feb '99

I believe that a UNIX based Networker server can backup an NT based Networker client while preserving the NT metadata. If this is so, you might try "front ending" a filer (via a mapped drive letter) with an NT box running the Networker client service. I have this procedure explained in my backup paper which should hit the external web within a week. -----Original Message----- From: Jim Harm [mailto:harm1@llnl.gov] Sent: Friday, February 19, 1999 9:46 AM To: Benn, Paul Cc: toasters(a)mathworks.com Subject: RE: Shared Snapshot directory Could UNIX backups maintain the NT information, when you have the Legato backup agent running on the netapp box? 8><-----snip-----snip-----snip-----

1 0

Re: Shared Snapshot directory
by Daniel Quinlan 20 Feb '99

20 Feb '99

Paul Benn <paul.benn(a)netapp.com> writes: > I just wanted to clarify that if a UNIX backup server is used, it > must use NDMP. Not quite -- it must use dump. Dumping via rsh instead of NDMP will also preserve CIFS attributes and 8.3 names. You just can't preserve CIFS attributes from a Unix NFS client whereas you can preserve NFS attributes from a WinNT CIFS client. - Dan

1 0

Re: Recovered error messages no longer logged in 5.1.2R3
by Daniel Quinlan 20 Feb '99

20 Feb '99

Greg Kitch <kitch(a)seas.ucla.edu> writes: > Does anyone else have an interest in having the flexibility of > including soft error messages in the log files or is there a general > consensus that they are unnecessary. We're trying to get support > from other users for making the toggle available. Yes, I want the soft errors to be logged. - Dan

1 0

Re: Recovered error messages no longer logged in 5.1.2R3
by John Tatar 18 Feb '99

18 Feb '99

Would someone from netapp let us know if that there is a facility and a level that we can set in the etc/syslog.conf file which would produce the Read sector error messages. Also is there a way you can kill -HUP the syslogd process so it will re-read the configuration file? JT -- =================================================================== John J. Tatar Systems & Network Engineering Section Advanced Computer Applications Group Decision & Information Sciences Division Argonne National Laboratory mailto:tatar@dis.anl.gov or john(a)tatar.com Office: 630-252-3242 Fax: 630-252-5128 Pager: 630-266-1063 ===================================================================

2 1

Re: Recovered error messages no longer logged in 5.1.2R3
by George Kahler 18 Feb '99

18 Feb '99

Greetings... Elena Samsonova <E.Samsonova(a)wxs.nl> writes: > > Does anyone else have an interest in having the flexibility of including > > soft error messages in the log files or is there a general consensus > > that they are unnecessary. We're trying to get support from other users > > for making the toggle available. > > Yes, I would like to have the toggle as well. In the first place, I would > like those messages to remain in the log file (I have not upgraded to 5.1 > yet but plan to do so next month). We have prevented disk crashes by > monitoring them. Ditto! I'm still on 4.3.5D4 on both of our F520 and F330. The F330 is using the old 4 GB Seagate drives which number of them were replaced due to crashes. I find the messages very important in judging and predicting the state of the drives. George ------------------------------------------------------------------------------- George Kahler e-mail: george(a)yorku.ca UNIX Systems Administrator humans: (416) 736-5257 x.22699 Computing Services, York University machines: (416) 736-5830 Ontario, Canada, M3J-1P3

2 1

wierd NFS (Solaris) behavior
by Jay Soffian 18 Feb '99

18 Feb '99

We've been having a problem with Solaris NFS clients not seeing file updates occuring on our NetApps for a matter of minutes. This appears to be more than just an attribute caching issue. All our Suns are running 2.5.1 and a recently patched (Generic_103640-24). We are using NFS v3 over udp. This behavior is difficult to reproduce reliably. I managed to catch a particularly egregious example one time. A file had been updated from one of our clients, and another client didn't see the update. Out of curiousty, I mounted the NetApp partition containing the file at a differnt mountpoint on the same client that wasn't seeing the update. The client saw two copies of the same file. Here is the _same_ file on the NetApp from two different mount points on one client: root@presidio:~/ # ls -l {/studios,/mnt}/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html -rw-r--r-- 1 20006 accatl 722 Dec 1 14:45 /mnt/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html -rw-r--r-- 1 20006 accatl 49 Nov 12 10:22 /studios/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html root@presidio:~/ # mount | grep '/export/studio' /studios on whirlpool:/export/studio proto=udp/remote on Tue Oct 13 04:02:54 1998 /mnt on whirlpool:/export/studio proto=udp/remote on Tue Dec 1 14:48:21 1998 Note that presidio is able to concurrently access two different copies of this file. It has cached not just the file attributes, but the data as well: root@presidio:~/ # wc {/studios,/mnt}/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html 1 3 49 /studios/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html 25 72 722 /mnt/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html Gets wierder still: root@presidio:~/ # touch {/studios,/mnt}/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html root@presidio:~/ # !ls ls -l {/studios,/mnt}/atlanta/accessatlanta/docs/pacman/includes/partners/rotation/namhoward.html -rw-r--r-- 1 20006 accatl 722 Dec 1 15:00 /mnt/atlanta/accessatlanta/docs/pacman/includes/partners/rotatio n/namhoward.html -rw-r--r-- 1 20006 accatl 49 Dec 1 15:00 /studios/atlanta/accessatlanta/docs/pacman/includes/partners/rot ation/namhoward.html Times updated, sizes did not. no_atime_update is off on the filer. Couldn't find anything at Sunsolve that seemed to be related to this problem. Any ideas? j. -- Jay Soffian <jay(a)cimedia.com> UNIX Systems Administrator 404.572.1941 Cox Interactive Media

2 1

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters February 1999