Hi all,
At last, my filer is in production :) Pfff.... almost had a hearth attack!
I had to copy about 29 Gb of files in different qtree all located on the same volume. After the whole thing finished (it took 3 hours), I went to the console and saw this :
Thu May 11 03:06:14 EDT [de0]: 8 CORR SYS ERR 201 eia:f6f60058b20000 fs:274f400 eis:0 isr:80000001 Thu May 11 03:06:14 EDT [de0]: ...SYS ECC error: address=0xf6f600, bit=31, DIMM=J27 Thu May 11 03:12:29 EDT [raid_stripe_owner]: 9 CORR SYS ERR 201 eia:ccf254058b20000 fs:2c3f400 eis:0 isr:80000001 Thu May 11 03:12:29 EDT [raid_stripe_owner]: ...SYS ECC error: address=0xccf2540, bit=31, DIMM=J27 Thu May 11 03:27:18 EDT [de0]: 10 CORR SYS ERR 201 eia:94960058b20000 fs:2f1f400 eis:0 isr:80000001 Thu May 11 03:27:18 EDT [de0]: ...SYS ECC error: address=0x949600, bit=31, DIMM=J27 Thu May 11 04:00:00 EDT [statd]: 4:00am up 9 days, 4:14 21685006 NFS ops, 0 CIFS ops, 0 HTTP ops Thu May 11 04:14:33 EDT [de0]: 11 CORR SYS ERR 201 eia:46545058b20000 fs:202f400 eis:0 isr:80010001 Thu May 11 04:14:33 EDT [de0]: ...SYS ECC error: address=0x465450, bit=31, DIMM=J27 Thu May 11 04:41:14 EDT [de0]: 12 CORR SYS ERR 201 eia:d578c058b30000 fs:2d0f400 eis:f0 isr:80000001 Thu May 11 04:41:14 EDT [de0]: ...SYS ECC error: address=0xd578c0, bit=31, DIMM=J27 Thu May 11 04:41:14 EDT [de0]: 13 CORR SYS ERR 201 eia:ec93b0058b20000 fs:2d0f400 eis:f0 isr:80000001 Thu May 11 04:41:14 EDT [de0]: ...SYS ECC error: address=0xec93b00, bit=31, DIMM=J27 Thu May 11 04:43:50 EDT [wafl_hipri]: 14 CORR SYS ERR 201 eia:11b2a90058b20000 fs:2b5f400 eis:0 isr:80000001 Thu May 11 04:43:50 EDT [wafl_hipri]: ...SYS ECC error: address=0x11b2a900, bit=31, DIMM=J30 Thu May 11 05:00:00 EDT [statd]: 5:00am up 9 days, 5:14 25265758 NFS ops, 0 CIFS ops, 0 HTTP ops Thu May 11 05:07:31 EDT [raid_stripe_owner]: 15 CORR SYS ERR 201 eia:1b52ca4058b20000 fs:2a3f400 eis:0 isr:80000001 Thu May 11 05:07:31 EDT [raid_stripe_owner]: ...SYS ECC error: address=0x1b52ca40, bit=31, DIMM=J30 Thu May 11 06:00:00 EDT [statd]: 6:00am up 9 days, 6:14 28703375 NFS ops, 0 CIFS ops, 0 HTTP ops
can anyone explain me what this means? Is it serious?
Thanks. Haim.
can anyone explain me what this means? Is it serious?
This is the same problem that was recently discussed on this list; your filer's memory detected and corrected some bit errors in its RAM in DIMMs J27 and J30. This is not unusual and you will see them from time to time, particularly the more the filer is loaded. Since you were doing 29GB of writes for 3 hours, at nearly an average of 1000 ops/s, it's not surprising that the memory got utilized a lot and a few bits had to be corrected.
The messages are only serious if you get a lot of them in a row or if its consistent or related to another problem or crash. If you want to feel safe, you could take the filer down for a few minutes and reseat those DIMMs just to make sure they have good connections. (If this is a new filer in production, perhaps it's been moved around lately and thus some connections may have gotten loosened up.)
If Netapp says it's a serious problem they can send you replacement memory. However, these sorts of errors are common. Be thankful that your filer has error-detecting error-correcting memory and didn't crash on a memory parity error like the old filers used to. :)
Bruce
Hi Haim :>
I heard Haim Dimermanas (Thu, May 11, 2000 at 06:30:52AM -0400) say
At last, my filer is in production :) Pfff.... almost had a hearth attack!
I had to copy about 29 Gb of files in different qtree all located on the same volume. After the whole thing finished (it took 3 hours), I went to the console and saw this :
OK, to keep things in proportion - that's a fair amount of data over a fair amount of time, so the number of messages below, if that's all of them, doesn't represent the horrible mangling of your data. Your copy should be fine.
Thu May 11 03:06:14 EDT [de0]: 8 CORR SYS ERR 201 eia:f6f60058b20000 fs:274f400 eis:0 isr:80000001 Thu May 11 03:06:14 EDT [de0]: ...SYS ECC error: address=0xf6f600, bit=31, DIMM=J27 Thu May 11 03:12:29 EDT [raid_stripe_owner]: 9 CORR SYS ERR 201 eia:ccf254058b20000 fs:2c3f400 eis:0 isr:80000001 Thu May 11 03:12:29 EDT [raid_stripe_owner]: ...SYS ECC error: address=0xccf2540, bit=31, DIMM=J27 Thu May 11 03:27:18 EDT [de0]: 10 CORR SYS ERR 201 eia:94960058b20000 fs:2f1f400 eis:0 isr:80000001 Thu May 11 03:27:18 EDT [de0]: ...SYS ECC error: address=0x949600, bit=31, DIMM=J27 Thu May 11 04:00:00 EDT [statd]: 4:00am up 9 days, 4:14 21685006 NFS ops, 0 CIFS ops, 0 HTTP ops Thu May 11 04:14:33 EDT [de0]: 11 CORR SYS ERR 201 eia:46545058b20000 fs:202f400 eis:0 isr:80010001 Thu May 11 04:14:33 EDT [de0]: ...SYS ECC error: address=0x465450, bit=31, DIMM=J27 Thu May 11 04:41:14 EDT [de0]: 12 CORR SYS ERR 201 eia:d578c058b30000 fs:2d0f400 eis:f0 isr:80000001 Thu May 11 04:41:14 EDT [de0]: ...SYS ECC error: address=0xd578c0, bit=31, DIMM=J27 Thu May 11 04:41:14 EDT [de0]: 13 CORR SYS ERR 201 eia:ec93b0058b20000 fs:2d0f400 eis:f0 isr:80000001 Thu May 11 04:41:14 EDT [de0]: ...SYS ECC error: address=0xec93b00, bit=31, DIMM=J27 Thu May 11 04:43:50 EDT [wafl_hipri]: 14 CORR SYS ERR 201 eia:11b2a90058b20000 fs:2b5f400 eis:0 isr:80000001 Thu May 11 04:43:50 EDT [wafl_hipri]: ...SYS ECC error: address=0x11b2a900, bit=31, DIMM=J30 Thu May 11 05:00:00 EDT [statd]: 5:00am up 9 days, 5:14 25265758 NFS ops, 0 CIFS ops, 0 HTTP ops Thu May 11 05:07:31 EDT [raid_stripe_owner]: 15 CORR SYS ERR 201 eia:1b52ca4058b20000 fs:2a3f400 eis:0 isr:80000001 Thu May 11 05:07:31 EDT [raid_stripe_owner]: ...SYS ECC error: address=0x1b52ca40, bit=31, DIMM=J30 Thu May 11 06:00:00 EDT [statd]: 6:00am up 9 days, 6:14 28703375 NFS ops, 0 CIFS ops, 0 HTTP ops
A long of the messages include a complaint about DIMM J27. I would wonder if that DIMM is seated properly. There's a couple of complaints about DIMM J30. I'd check they are all seated properly actually. Netapp support can give you the word on whether or not J27 is in need of replacement, but I would be tempted to ask for a new pair of DIMMS anyway. Not to replace 27 and 30. You have to replace them in pairs, ideally, so you'd be looking to replace 27 and 29. They are labelled on the motherboard, though be in good lighting when you look, it's written RIGHT small.
Run a raid scrub if you are paranoid - it should turn up any errors in the raid stripes and fix them.
All of the errors are CORRected errors. So you should be fine. Keep an eye on it and if it happens again, I'd think about RMA'ing some RAM.
J
--------------------------------------------------------------------------- # John Denholm johnd@theplanet.net # # Webcache & Filer Administrator, Planet Online +44 113 207 6357 # ---------------------------------------------------------------------------
Error 404: There is no spoon
John Denholm wrote:
Hi Haim :>
I heard Haim Dimermanas (Thu, May 11, 2000 at 06:30:52AM -0400) say
At last, my filer is in production :) Pfff.... almost had a hearth attack!
I had to copy about 29 Gb of files in different qtree all located on the same volume. After the whole thing finished (it took 3 hours), I went to the console and saw this :
OK, to keep things in proportion - that's a fair amount of data over a fair amount of time, so the number of messages below, if that's all of them, doesn't represent the horrible mangling of your data. Your copy should be fine.
That's all I got. Thank you for your answer (A big thanks to Bruce too). I will keep a close eye on it.
Haim.
P.S. : I really like this mailing list :)
-
Bruce Sterling Woodcock -
Haim Dimermanas -
John Denholm