toasters

toasters@lists.teaparty.net

13531 discussions

RE: Measuring Filer Performance
by Glenn Walker 16 Nov '06

16 Nov '06

I'd agree with the latency standpoint. It's a good metric to determine when the users will start to bitch. The latency might be hidden by the application, but it depends on how well it's written (single threaded vs multi threaded\low concurrency vs high concurrency). Typically, this is all a function of CPU\DISK (and sometimes memory) load. As a result, those are good to watch, and they are a direct result of the overall workload (CIFS\NFS\iSCSI IOPS). The hard part is determining what\why the workload tanks. Most times, users are only going to feel impact if there is a long running event that generates high latency - however, depending on workload, a single spike that lasts only moments (a second or two) can be devastating: this is where knowing your environment comes in handy. Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Blake Golliher Sent: Wednesday, November 15, 2006 7:08 PM To: George, Andrew Cc: Rohit; toasters(a)mathworks.com Subject: Re: Measuring Filer Performance Latency is primary for what most of what my filers are doing. Usually watched via nfs_hist (being sure to -z the stats), or using stats all wrapped up in a shell script. rsh $FILER 'priv set -q diag ; stats show -i $INTER -n $SAMPLES volume:$VOL:avg_latency volume:$VOL:total_ops volume:$VOL:read_latency volume:$VOL:read_ops volume:$VOL:write_latency volume:$VOL:write_ops volume:$VOL:other_latency volume:$VOL:other_ops volume:$VOL:nfs_read_latency volume:$VOL:nfs_read_ops volume:$VOL:nfs_write_latency volume:$VOL:nfs_write_ops volume:$VOL:nfs_other_latency volume:$VOL:nfs_other_ops' nfsv3:nfs:nfsv3_avg_op_latency nfsv3:nfs:nfsv3_op_percent nfsv3:nfs:nfsv3_write_ops nfsv3:nfs:nfsv3_write_latency nfsv3:nfs:nfsv3_read_ops nfsv3:nfs:nfsv3_read_latency nfsv3:nfs:nfsv3_ops' It's good to note that while stats show averages, nfs_hist shows a histogram, and can spot the occasional very high latency operation. Good to watch. I just poked around, there is a cifs component for stats, and it shows some interesting metrics the cifs users might want to watch. filer01*> stats show cifsdomain cifs cifs:cifs:cifs_ops:2658/s cifs:cifs:cifs_op_count: GetAttr: 1296 Read: 989 Write: 0 Lock: 1 Open/Close: 23 Directory: 0 Other: 335 cifs:cifs:cifs_op_pct: GetAttr: 48% Read: 37% Write: 0% Lock: 0% Open/Close: 0% Directory: 0% Other: 12% cifs:cifs:cifs_latency:3.27ms cifsdomain:10.0.0.3:netlogon_latency:9.50ms cifsdomain:10.0.0.3:netlogon_latency_base:6 cifsdomain:10.0.0.3:lsa_latency:0ms cifsdomain:10.0.0.3:lsa_latency_base:0 cifsdomain:10.0.0.3:samr_latency:0ms cifsdomain:10.0.0.3:samr_latency_base:0 cifsdomain:10.0.0.2:netlogon_latency:2.00ms cifsdomain:10.0.0.2:netlogon_latency_base:1 cifsdomain:10.0.0.2:lsa_latency:0ms cifsdomain:10.0.0.2:lsa_latency_base:0 cifsdomain:10.0.0.2:samr_latency:0ms cifsdomain:10.0.0.2:samr_latency_base:0 cifsdomain:10.0.0.1:netlogon_latency:0ms cifsdomain:10.0.0.1:netlogon_latency_base:0 cifsdomain:10.0.0.1:lsa_latency:0ms cifsdomain:10.0.0.1:lsa_latency_base:0 cifsdomain:10.0.0.1:samr_latency:0ms cifsdomain:10.0.0.1:samr_latency_base:0 There's also smb_hist. That's from a 960 running 6.5.6PsomethingDsomething.... -Blake On 11/15/06, George, Andrew <georgea(a)anz.com> wrote: > We only use our filers for CIFS (so this may be completely inapplicable > to you) > > 1. Cache age! Mainly because when I have seen filers that are worked to > death, that's the value that tends to react to it (though CP there are a > lot of secondary metrics to support that). > > 2. As mentioned by someone else...the magic metric is how badly the > users are bitching. That being a little "rubbery" we also check cache > age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic > to determine if anything's bottlenecking. > > -----Original Message----- > From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] > On Behalf Of Rohit > Sent: Wednesday, 15 November 2006 9:59 PM > To: toasters(a)mathworks.com > Subject: Measuring Filer Performance > > Hi Folks > > I had some open questions and i hope you'll help me answer them based on > your experience. > > 1. If you just had a single metric to determine filer performance what > would it be ? > Would it be CPU usage, throughput, latency, IOPS..or anything else ? > > 2. At your place, what metrics do you use to determine if a filer is > still good enough not to worry about ? > > thanks > rohit > > ------------------------------------------------------------------------ > -- > Refinance & Home Equity Loan Quotes > Low credit score? Old Merchants can help. Get a quote in 2 easy steps! > http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ > > > > > "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication." >

2 1

Measuring Filer Performance
by Rohit 16 Nov '06

16 Nov '06

Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

3 2

RE: Measuring Filer Performance
by George, Andrew 16 Nov '06

16 Nov '06

We only use our filers for CIFS (so this may be completely inapplicable to you) 1. Cache age! Mainly because when I have seen filers that are worked to death, that's the value that tends to react to it (though CP there are a lot of secondary metrics to support that). 2. As mentioned by someone else...the magic metric is how badly the users are bitching. That being a little "rubbery" we also check cache age, cp type, disk utilisation, cpu usage, Ops/sec and network traffic to determine if anything's bottlenecking. -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Rohit Sent: Wednesday, 15 November 2006 9:59 PM To: toasters(a)mathworks.com Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit ------------------------------------------------------------------------ -- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/ "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

2 1

Re: Measuring Filer Performance
by Reinoud Reynders 15 Nov '06

15 Nov '06

Hi Rohit, My view is that your application performance is the most important. This is what our users see and feel. The problem is that for every application, the impact of the storage on the performance is different. For our Databases, responstime is the key. So for those applications we measure the latency. For our PACS (digital radiology) is Mb/s more important. So we try to measure the application performance and the key infrastructure elements of that application (like latency). On filer level, we monitor the CPU to give a general idee of the load of the filer. Best regards, Reinoud Reynders IT manager Infrastructure & Operations Leuven University Hospital -----Original Message----- From: owner-toasters(a)mathworks.com <owner-toasters(a)mathworks.com> To: toasters(a)mathworks.com <toasters(a)mathworks.com> Sent: Wed Nov 15 11:58:39 2006 Subject: Measuring Filer Performance Hi Folks I had some open questions and i hope you'll help me answer them based on your experience. 1. If you just had a single metric to determine filer performance what would it be ? Would it be CPU usage, throughput, latency, IOPS..or anything else ? 2. At your place, what metrics do you use to determine if a filer is still good enough not to worry about ? thanks rohit -------------------------------------------------------------------------- Refinance & Home Equity Loan Quotes Low credit score? Old Merchants can help. Get a quote in 2 easy steps! http://tagline.bidsystem.com/fc/BgLEQfI7zNZsu5fZHhXyj0UMjSNzY6ZfSaWC/

1 0

RE: IBM Branded NetApps
by Julio Calderon 15 Nov '06

15 Nov '06

I've been in the same spot having to deal with a vendorA when vendorA hd to get support from VendorB and I could never get quick and advance help from anyone! Not a fun place to be when you need help. Good luck -----Original Message----- From: Zayas, Russell [mailto:rzayas@qualcomm.com] Sent: Tue Nov 14 16:19:46 2006 To: Toasters; Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps Does that mean that support is geared to handle tough issues? Or do they pipe you into NetApp support at that point? Curious, -Russ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Toasters Sent: Tuesday, November 14, 2006 12:41 PM To: Raj Patel; toasters(a)mathworks.com Subject: RE: IBM Branded NetApps We just went with a few IBM boxes here. We have 40 or so Netapp boxes. The determining factor for us was the price. They undercut by well over 100k on what we bought. The support structure is not the same as Netapp. If you have never experienced IBM support before then I suggest you do before you decided to go the way of IBM/NAS. If your like us and don't need the "what is a snapmirror" type of support and just call when you need parts or the box is panicing then you may want to consider them. E- ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Raj Patel Sent: Tuesday, November 14, 2006 2:03 PM To: toasters(a)mathworks.com Subject: IBM Branded NetApps Hi all, A few quick queries - Has anyone had any experience with the IBM Branded NetApp's ? What does the N5200 compare with in the NetApp family ? How does service/support differ from NetApp ? Finally why would anyone go with IBM for a NetApp rather than NetApp direct (is it purely a question of vendor service quality or is there a significant price difference) ? Thanks in advance, Raj.

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Usally less than 1 second, at least a couple of times during the snapmirror job It is a Volume snapmirror Ontap 7.1 -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 13:32 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

Hmm - I honestly have never noticed that problem on any of our equipment... How long does the spike last? What version of ONTAP? VSM or QSM? Glenn -----Original Message----- From: Davies,Matt [mailto:MDAVIES@generalatlantic.com] Sent: Tuesday, November 14, 2006 7:58 AM To: Glenn Walker; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Glenn, The snapmirror.conf file is showing 1000 as the limit. Which should equate to just under 8Mb/s Overall the rate does average out for the snapmirror jobs to be below 1000KB/s There was only one snapmirror running at the time of the test I carried out to confirm the rates. Matt -----Original Message----- From: Glenn Walker [mailto:ggwalker@mindspring.com] Sent: 14 November 2006 12:47 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: SnapMirror Bandwidth restrictions I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: SnapMirror Bandwidth restrictions
by Glenn Walker 14 Nov '06

14 Nov '06

I assume that the snapmirror.conf file is showing 1000 as the limit? This is in KB (kilobyte), not kb(kilobit). Are you sure that there aren't multiple mirrors running simultaneously? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: Tuesday, November 14, 2006 5:40 AM To: toasters(a)mathworks.com Subject: SnapMirror Bandwidth restrictions Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

SnapMirror Bandwidth restrictions
by Davies,Matt 14 Nov '06

14 Nov '06

Hi, We have noticed that when a snapmirror runs between our filers with a maximum transfer rate of 1000 KB/s, we are seeing peaks of traffic of around 24Mb/s. This has been confirmed using ethereal, that it is the snapmirror traffic. We have noticed that these peaks are causing excessive amount of dropped packets on the wan interface of the routers between all the site. we have tried a number of difference queuing strategies and bandwidth shaping, but the router cannot buffer all the packets during the peaks and starts dropping them. As we also use these links for both voice and video, we are seeing high jitter rates during these peaks, and need to do something about it. We have a mixture of FAS250's and FAS270's all are showing the same problem Has anyone else experienced this ? Or does anyone have any ideas. Cheers Matt

1 0

RE: Macintosh users can't see network drive
by Glenn Walker 14 Nov '06

14 Nov '06

You might want to capture a quick packet trace on either the MAC or NetApp side (tcpdump on mac, I believe and 'pktt start all -d <directory>' on the Filer - be sure to perform 'pktt stop all' when done with the test). MAC has a notoriously horrible CIFS client - I assume you're using OSX? Glenn -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Maglinger, Paul Sent: Monday, November 13, 2006 4:31 PM To: NetApps list server Subject: FW: Macintosh users can't see network drive Running Version 6.4.5 on a 740 filer. For some reason this morning our Macintosh users cannot see the "public" CIFS drive on our NetApps filer. Because this is a "public" drive, it should be open all of our users. Everything worked fine until this morning and now they cannot see the filer in their Server folder. We've connected to a Windows server using smb://ip_address/shared_directory, but when we try to connect to the filer in the same manner we never get a logon prompt. No pretending here, I don't know diddily-squat about Mac computers. Can someone give advice on how to even start troubleshooting this? ________________________________ Paul Maglinger, A+, CA, CCA, CET, MCSE Systems Administrator Shoe Carnival Inc. (812)867-4674 pmaglinger(a)scvl.com ________________________________

1 0

FW: Macintosh users can't see network drive
by Maglinger, Paul 13 Nov '06

13 Nov '06

Running Version 6.4.5 on a 740 filer. For some reason this morning our Macintosh users cannot see the "public" CIFS drive on our NetApps filer. Because this is a "public" drive, it should be open all of our users. Everything worked fine until this morning and now they cannot see the filer in their Server folder. We've connected to a Windows server using smb://ip_address/shared_directory, but when we try to connect to the filer in the same manner we never get a logon prompt. No pretending here, I don't know diddily-squat about Mac computers. Can someone give advice on how to even start troubleshooting this? ________________________________ Paul Maglinger, A+, CA, CCA, CET, MCSE Systems Administrator Shoe Carnival Inc. (812)867-4674 pmaglinger(a)scvl.com ________________________________

1 0

ndmpcopy, jndmpcopy and multiple level 1 transfers
by Paul Letta 13 Nov '06

13 Nov '06

I am getting ready to do a big data migration. I am planning on doing an initial level 0, and then multiple level 1's leading up to the big day. I know the that native ndmpcopy in OnTap 7.1 can only do one level 1, and one level 2 for each level 0. But I have read that if you get ndmpcopy or jnmdpcopy from ndmp.org that it supports multiple level 1 dumps based on the same level 0. But I am getting the same errors I got with the native ndmpcopy when I use ndmpcopy and jndmpcopy from a unix host. I just grabbed the latest copies for ndmp.org yesterday. > java jndmpcopy.JNdmpcopy -version java version "1.4.2" gcj (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3) So I did a level 0, and then one level 1 that both worked just fine. Then I re-ran the level 1 with the same command, and get the errors below. This is the same error I get if I try this with the native ndmpcopy command on the filer itself. I am moving about 1TB of data (about 16M files) off of 1 traditional volume onto multiple new flex vols on the same filer. If what I'm doing is flat out wrong, what would others suggest ? tar wouldn't give me the incrementals and rsync would take a long time with this number of files. Suggestions ? Thanks, Paul > java jndmpcopy.JNdmpcopy filer:/vol/vol0/source filer:/vol/destination -sa xxxxxxx -da xxxxxxx -level 1 Nov 09 03:53:25 [ndmpd:0]: Now waiting for source and destination to finish. SrcFiler: Nov 09 03:53:25 [ndmpd:0]: DUMP: creating "/vol/vol0/../snapshot_for_backup.153" snapshot. SrcFiler: Nov 09 03:53:26 [ndmpd:0]: DUMP: Using subtree dump SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of this level 1 dump: Thu Nov 9 15:53:25 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of last level 0 dump: Thu Nov 9 15:48:10 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Dumping /vol/vol0/source to NDMP connection SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: mapping (Pass I)[regular files] SrcFiler: Nov 09 03:53:43 [ndmpd:0]: DUMP: mapping (Pass II)[directories] SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: estimated 18 KB. SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: dumping (Pass III) [directories] DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Thu Nov 9 15:54:13 2006: Begin incremental restore DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This incremental dump does not follow the previous dump that you have restored. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This dump is based on Thu Nov 9 15:48:10 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: It should be based on Thu Nov 9 15:50:28 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Could not use tape. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: RESTORE IS ABORTED Nov 09 03:54:13 [ndmpd:0]: DstFiler just halted. Nov 09 03:54:13 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. SrcFiler: Nov 09 03:54:13 [ndmpd:0]: DUMP: dumping (Pass IV) [regular files] SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: Network communication error SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: DUMP IS ABORTED SrcFiler: Nov 09 03:54:18 [ndmpd:0]: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.153" snapshot. Nov 09 03:54:19 [ndmpd:0]: SrcFiler just halted. Nov 09 03:54:19 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. Nov 09 03:54:19 [ndmpd:0]: jndmpcopy finished -- Paul Letta Thomas Jefferson National Accelerator Facility Computer Center Unix System Administrator EMail: letta(a)jlab.org Phone: (757)269-5106

4 3

Re: ndmpcopy, jndmpcopy and multiple level 1 transfers
by Jerry 13 Nov '06

13 Nov '06

Never underestimate the power of rsync. Here is what I did to move about 70mil files (lots of tiny files). With lots of tiny files, NDMCOPY spends a lot of time before it really transmits anything. In some cases of 10+ mil files on a busy filer I was waiting 4+ hours for files to begin copying (actual data, not just writing out sparse files for inodes). I found that if I ran a level 0 ndmpcopy on the weekend or during your off hours you can let it beat up the filer a little bit (and it will beat up both CPU and Disk util). Then, during the week run rsync's with --delete to keep things nicely in sync. This is something ndmpcopy can't do (I think).... if you do a level 0 + level 1 you won't get deletes. Doing the "catchup rsync's" gives you a good judge of timing on how long it will take you when you schedule your real outage. I recommend using rsync 2.6.6 or higher as it has significant memory efficiency compared to older versions. I used a 4 CPU Sun V440 to move a lot of data with rsync. Running a single process with a 16 million file list consumes as much as 8GB ram. I almost never found myself CPU bound and I found that CPU on the filer was much lighter than if I was doing an ndmpcopy. All of this depends on your rate of change for files, but overall I think rsync was faster for me after that initial copy. Since rsync wasn't using all my CPU, I broke it into an rsync for each top level directory, as many as 70 rsync processes in some of my migrations at the same time. This allowed me to sync up 16 million files in < 20 minutes in some cases. I suspect the level 1 ndmpcopy would be slower than that, but I'm not 100% sure. At the time of cutover, change the export to be read-only to block people from making change. This doesn't work if you have a subdirectory that is hitting a different export line. In this case I would do a second rsync and see what files were still changing and hunt down apps writing to that volume still. I'd be curious to know which method you end up going with and how it wo Original Message ---- From: Darren Dunham <ddunham(a)taos.com> To: toasters(a)mathworks.com Sent: Friday, November 10, 2006 12:37:19 PM Subject: Re: ndmpcopy, jndmpcopy and multiple level 1 transfers > > I am getting ready to do a big data migration. I am planning on doing > > an initial level 0, and then multiple level 1's leading up to the big day. > > > > It doesn't make sense to me why you would do multiple level 1 copies > because a level 1 does not copy files that have changed since the > previous level 1. To do this you must use a level 2 copy. I agree with the spirt, but I would rephrase it. :-) A level 1 *does* copy all the files that have changed since the previous level 1. It's just that they are a subset of files modified since the level 0 which are copied as well. So running a level 1 on the last day should take about the same amount of time whether any other level 1 copies had been run previously or not. Agreed on the rest about 0/1/2 ndmpcopy, but for flexibility I do like rsync as an alternative. It's just how much power you can toss at it and how long you can wait for the file list when you've got millions of files. Remember you don't have to move everything in one rsync command. Espcially if you have multiple clients, it may may sense to split it up into a few separate subtrees. -- Darren Dunham ddunham(a)taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. > ____________________________________________________________________________________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index

3 2

RE: ndmpcopy, jndmpcopy and multiple level 1 transfers
by George, Andrew 11 Nov '06

11 Nov '06

I wound up doing something similar for one of mine...doing a base level 0 ndmp and then runinning regular robocopies with the /mir flag set to keep both filesystems in synch while I was battling the change-monkeys for a window to actually move the puppy. -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Jerry Sent: Saturday, 11 November 2006 1:34 PM To: Darren Dunham; toasters(a)mathworks.com Subject: Re: ndmpcopy, jndmpcopy and multiple level 1 transfers Never underestimate the power of rsync. Here is what I did to move about 70mil files (lots of tiny files). With lots of tiny files, NDMCOPY spends a lot of time before it really transmits anything. In some cases of 10+ mil files on a busy filer I was waiting 4+ hours for files to begin copying (actual data, not just writing out sparse files for inodes). I found that if I ran a level 0 ndmpcopy on the weekend or during your off hours you can let it beat up the filer a little bit (and it will beat up both CPU and Disk util). Then, during the week run rsync's with --delete to keep things nicely in sync. This is something ndmpcopy can't do (I think).... if you do a level 0 + level 1 you won't get deletes. Doing the "catchup rsync's" gives you a good judge of timing on how long it will take you when you schedule your real outage. I recommend using rsync 2.6.6 or higher as it has significant memory efficiency compared to older versions. I used a 4 CPU Sun V440 to move a lot of data with rsync. Running a single process with a 16 million file list consumes as much as 8GB ram. I almost never found myself CPU bound and I found that CPU on the filer was much lighter than if I was doing an ndmpcopy. All of this depends on your rate of change for files, but overall I think rsync was faster for me after that initial copy. Since rsync wasn't using all my CPU, I broke it into an rsync for each top level directory, as many as 70 rsync processes in some of my migrations at the same time. This allowed me to sync up 16 million files in < 20 minutes in some cases. I suspect the level 1 ndmpcopy would be slower than that, but I'm not 100% sure. At the time of cutover, change the export to be read-only to block people from making change. This doesn't work if you have a subdirectory that is hitting a different export line. In this case I would do a second rsync and see what files were still changing and hunt down apps writing to that volume still. I'd be curious to know which method you end up going with and how it wo Original Message ---- From: Darren Dunham <ddunham(a)taos.com> To: toasters(a)mathworks.com Sent: Friday, November 10, 2006 12:37:19 PM Subject: Re: ndmpcopy, jndmpcopy and multiple level 1 transfers > > I am getting ready to do a big data migration. I am planning on > > doing an initial level 0, and then multiple level 1's leading up to the big day. > > > > It doesn't make sense to me why you would do multiple level 1 copies > because a level 1 does not copy files that have changed since the > previous level 1. To do this you must use a level 2 copy. I agree with the spirt, but I would rephrase it. :-) A level 1 *does* copy all the files that have changed since the previous level 1. It's just that they are a subset of files modified since the level 0 which are copied as well. So running a level 1 on the last day should take about the same amount of time whether any other level 1 copies had been run previously or not. Agreed on the rest about 0/1/2 ndmpcopy, but for flexibility I do like rsync as an alternative. It's just how much power you can toss at it and how long you can wait for the file list when you've got millions of files. Remember you don't have to move everything in one rsync command. Espcially if you have multiple clients, it may may sense to split it up into a few separate subtrees. -- Darren Dunham ddunham(a)taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. > ________________________________________________________________________ ____________ Want to start your own business? Learn how on Yahoo! Small Business. http://smallbusiness.yahoo.com/r-index "This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ National Bank Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."

1 0

RE: ndmpcopy, jndmpcopy and multiple level 1 transfers
by Sphar, Mike 10 Nov '06

10 Nov '06

Is the source data in qtrees at least? If you haven't already, you might see if Netapp would give you a temporary SnapMirror license. We're doing some migrations also and they gave us some 30day licenses to do the migrations with. -- Michael W. Sphar - IS&T - Lead Systems Administrator SMBU Engineering Support Services, BMC Software -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Paul Letta Sent: Friday, November 10, 2006 7:22 AM To: toasters(a)mathworks.com Subject: ndmpcopy, jndmpcopy and multiple level 1 transfers I am getting ready to do a big data migration. I am planning on doing an initial level 0, and then multiple level 1's leading up to the big day. I know the that native ndmpcopy in OnTap 7.1 can only do one level 1, and one level 2 for each level 0. But I have read that if you get ndmpcopy or jnmdpcopy from ndmp.org that it supports multiple level 1 dumps based on the same level 0. But I am getting the same errors I got with the native ndmpcopy when I use ndmpcopy and jndmpcopy from a unix host. I just grabbed the latest copies for ndmp.org yesterday. > java jndmpcopy.JNdmpcopy -version java version "1.4.2" gcj (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3) So I did a level 0, and then one level 1 that both worked just fine. Then I re-ran the level 1 with the same command, and get the errors below. This is the same error I get if I try this with the native ndmpcopy command on the filer itself. I am moving about 1TB of data (about 16M files) off of 1 traditional volume onto multiple new flex vols on the same filer. If what I'm doing is flat out wrong, what would others suggest ? tar wouldn't give me the incrementals and rsync would take a long time with this number of files. Suggestions ? Thanks, Paul > java jndmpcopy.JNdmpcopy filer:/vol/vol0/source filer:/vol/destination -sa xxxxxxx -da xxxxxxx -level 1 Nov 09 03:53:25 [ndmpd:0]: Now waiting for source and destination to finish. SrcFiler: Nov 09 03:53:25 [ndmpd:0]: DUMP: creating "/vol/vol0/../snapshot_for_backup.153" snapshot. SrcFiler: Nov 09 03:53:26 [ndmpd:0]: DUMP: Using subtree dump SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of this level 1 dump: Thu Nov 9 15:53:25 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Date of last level 0 dump: Thu Nov 9 15:48:10 2006. SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: Dumping /vol/vol0/source to NDMP connection SrcFiler: Nov 09 03:53:42 [ndmpd:0]: DUMP: mapping (Pass I)[regular files] SrcFiler: Nov 09 03:53:43 [ndmpd:0]: DUMP: mapping (Pass II)[directories] SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: estimated 18 KB. SrcFiler: Nov 09 03:54:09 [ndmpd:0]: DUMP: dumping (Pass III) [directories] DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Thu Nov 9 15:54:13 2006: Begin incremental restore DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This incremental dump does not follow the previous dump that you have restored. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: This dump is based on Thu Nov 9 15:48:10 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: It should be based on Thu Nov 9 15:50:28 2006. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: Could not use tape. DstFiler: Nov 09 03:54:13 [ndmpd:0]: RESTORE: RESTORE IS ABORTED Nov 09 03:54:13 [ndmpd:0]: DstFiler just halted. Nov 09 03:54:13 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. SrcFiler: Nov 09 03:54:13 [ndmpd:0]: DUMP: dumping (Pass IV) [regular files] SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: Network communication error SrcFiler: Nov 09 03:54:17 [ndmpd:0]: DUMP: DUMP IS ABORTED SrcFiler: Nov 09 03:54:18 [ndmpd:0]: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.153" snapshot. Nov 09 03:54:19 [ndmpd:0]: SrcFiler just halted. Nov 09 03:54:19 [ndmpd:0]: Reason: ndmp_data_halted_internal_error : Dump aborted. Nov 09 03:54:19 [ndmpd:0]: jndmpcopy finished -- Paul Letta Thomas Jefferson National Accelerator Facility Computer Center Unix System Administrator EMail: letta(a)jlab.org Phone: (757)269-5106

1 0

RE: ssh config on DOT 7
by Creek, Toby 09 Nov '06

09 Nov '06

Vmware's client doesn't have 3des-cbc in its list of available ciphers, although it is compiled in. Add "3des-cbc" to the end of the Ciphers line in the /etc/ssh.conf and it'll work fine. That's the only cipher the NetApp implements. Toby > -----Original Message----- > From: Glenn Dekhayser [mailto:gdekhayser@voyantinc.com] > Sent: Wednesday, November 08, 2006 3:20 PM > To: Darren Dunham; toasters(a)mathworks.com > Subject: RE: ssh config on DOT 7 > > Well, we had the problem when trying to get SSH scripting > working with VMWare ESX 3 and the Netapp, and we did in fact > need authorized_keys2 on both the VMWare and the Netapp box > in order to get RSA/SSHv2 working. > We tried for a long time to get that working, perhaps we were > missing something. When we mentioned this to Netapp they > confirmed that our assessment was correct and they edited the docs. > > > > Be CAREFUL here: the filename is "authorized_keys2" for the SSH > > version2 file, the docs didn't say that, I actually told > netapp about > > that Doc bug. Authorized_keys is for version 1, which you shouldn't > be > > using anymore. Netapp's SSH client is based on OpenSSH so you can > > follow any docs on that to see how it really works. > > Then that would be a difference from the standard OpenSSH releases. > OpenSSH does not require the use of "authorized_keys2" (or > hasn't for years). Both v1 and v2 keys happily live together > in the default location of ".ssh/authorized_keys". > > I haven't used this feature in OnTAP, so I don't know if it > is required for setup there. > > -- > Darren Dunham > ddunham(a)taos.com > Senior Technical Consultant TAOS > http://www.taos.com/ > Got some Dr Pepper? San Francisco, > CA bay area > < This line left intentionally blank to confuse you. > > >

1 0

RE: ssh config on DOT 7
by Potakamuri, Sreenivasa 08 Nov '06

08 Nov '06

All public keys should simply be put into the "authorized_keys" file for both SSH protocols 1 and 2 for public key authentication. This is true for all versions of ONTAP. "authorized_keys2" file was an alternative location for SSH2 protocol, and its use has been deprecated in NetApp's SSH implementation. For backward compatibility, NetApp's SSH implementation still looks for keys in "authorized_keys2" file. Regards, Sreenivasa Potakamuri Network Appliance !-----Original Message----- !From: Glenn Dekhayser [mailto:gdekhayser@voyantinc.com] !Sent: Wednesday, November 08, 2006 8:48 AM !To: toasters(a)mathworks.com !Subject: RE: ssh config on DOT 7 ! ! !Be CAREFUL here: the filename is "authorized_keys2" for the SSH !version2 file, the docs didn't say that, I actually told netapp about !that Doc bug. Authorized_keys is for version 1, which you shouldn't be !using anymore. Netapp's SSH client is based on OpenSSH so you can !follow any docs on that to see how it really works. ! !Glenn (the other one) ! !-----Original Message----- ! !> Hello all, !> !> So I finally got DOT 7 put into production so I can now use ssh !> without an addition license. My question is where do I put my !> authorized_keys file so I can use key based authentication on the !> netapp? (i.e. where is or where should I create the .ssh directory) !> !> Thanks for any help, !> ! !

1 0

ssh config on DOT 7
by Sheldon Mustard 08 Nov '06

08 Nov '06

Hello all, So I finally got DOT 7 put into production so I can now use ssh without an addition license. My question is where do I put my authorized_keys file so I can use key based authentication on the netapp? (i.e. where is or where should I create the .ssh directory) Thanks for any help, SJM -- Sheldon Mustard smustard(a)gmail.com "There will be no order, only chaos." - Pi (1998) --

6 8

Re: 7.2 stability
by Grey Friday 08 Nov '06

08 Nov '06

We are running 7.2P2 as well for bug 213380 The components in Filer View applets are disabled when SSL is enabled on the filer But only on non important things. I was hoping that this was a decent rel. but apparently not. I would like to make use of CIFS being in it's own domain but maybe not till later eh... I don't have the link off the top of my head but I think 7.0.5 is the only GD rel currently. Some one correct me if I'm wrong. If anyone else knows anything CIFS related please share it. We have more CIFS than anything in this environment. Thanks, Grey- On 11/7/06, Willeke, Jochen <Jochen.Willeke(a)wincor-nixdorf.com> wrote: Hi Grey, > > we have already deployed 7.2P2 and it has a Bug in WAFL which makes the > filer reboot for some kind of reason. > Sadly we where hit by this error just 5 hours after installing > 7.2P2...Luckily never again :D > > Netapp provided a patch 7.2P2D2 espacially for this problem which will > be installed this weekend on our systems. > > I think you should wait until it is GD. BTW there was a link on NOW > where i could see all releases and which ones are GD but i do not find > it anymore. Do you know where it can be found on NOW? > > Regards > > Jochen > > -----Original Message----- > From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] > On Behalf Of Grey Friday > Sent: Tuesday, November 07, 2006 5:43 PM > To: toasters(a)mathworks.com > Subject: 7.2 stability > > So what's the word on 7.2 ready for install or still hold off? >

1 1

Windows Explorer showing incorrect disk space totals
by Stephen C. Losen 25 Oct '06

25 Oct '06

We have a filer running DOT 7.0.1R1 with a qtree named /vol/h2/t54 with NTFS security style and this share: fccf /vol/h2/t54/fccf Here are the quota file entries that pertain to the qtree: * user@/vol/h2 - - (users unlimited in the volume) * group@/vol/h2 - - (groups unlimited in the volume) /vol/h2/t45 tree 500000M 500000K (tree quota approx 500G) * user@/vol/h2/t54 500000M 500000K (users can use all of qtree) Quotas are on and "quota report /vol/h2/t54" indicates that the qtree is using 184408104 KB out of 512000000 KB. Essentially all used space is beneath the /vol/h2/t54/fccf directory. When I map the fccf share with Windows Explorer and right click the drive letter and look at "properties" it reports the correct amount of available disk space: 327591896 KB, but it reports only 35892088 KB used (instead of 184408104) and reports the total capacity as 363483984 KB (instead of 512000000). I looked at bugs online and saw a couple bugs where Explorer or CIFS reports incorrect values, but these don't seem to apply to this situation and the bugs are fixed. Is there anything I can do besides telling folks to only pay attention to available space (which is correct) and ignore the other values ? Steve Losen scl(a)virginia.edu phone: 434-924-0640 University of Virginia ITC Unix Support

3 2

RE: Anyone know what this means?
by Fox, Adam 19 Oct '06

19 Oct '06

Best bet here is to open a case, but *in general* this occurs something happens in the initial SnapMirror handshake. It could be a network drop, it could be a really busy filer that didn't respond in time, it could be other things. Tech Support may be able to examine your logs and see if something else was going on at the time. It's not really possible to tell from just the message, however. Hope this helps. -- Adam Fox NGS Tools Developer adamfox(a)netapp.com _____ From: Bender, Marilee [mailto:Marilee.Bender@delta.com] Sent: Thursday, October 19, 2006 10:22 AM To: toasters(a)mathworks.com Subject: Anyone know what this means? Async snapmirror relationship between FAS3050 and R200. Snapmirrors starting failing one day - They have been set up for a year or more and ran without issue. Now, sometimes they complete, sometimes they appear to "hang". Ontap 7.0.4P9 on both source and dest. What could this error indicate? "Error in snapmirror sync mode negotiation" Thanks

1 0

Anyone know what this means?
by Bender, Marilee 19 Oct '06

19 Oct '06

Async snapmirror relationship between FAS3050 and R200. Snapmirrors starting failing one day - They have been set up for a year or more and ran without issue. Now, sometimes they complete, sometimes they appear to "hang". Ontap 7.0.4P9 on both source and dest. What could this error indicate? "Error in snapmirror sync mode negotiation" Thanks

1 0

replacing a broken disk in 5.3.6R2
by Frank Smith 19 Oct '06

19 Oct '06

It's been a long time since I've dealt with NetApps, so there may be a simple solution to this that I've forgotten. Had a couple of disks fail on an old F720, the first one caused a rebuild onto the hot spare, the second one put it into degraded mode. An admin pulled the dead drives (w/o first doing a 'disk remove') and inserted a two of the same size drives from a retired system that used to be its twin (same H/W and firmware), but sysconfig -r still shows the 2 disks as broken. The references I have all seem to be for later releases, as priv set advanced disk unfail <DISK> comes back with disk: Did not recognize option "unfail". Also, I seemed to recall that the replaced drives would show up as foreign volumes since they weren't zeroed on the old system, so perhaps the system just doesn't realize the drives were replaced. I also remember that years ago when I hot-swapped a drive, the 720 seemed to 'freeze' for a minute or so, but the guy who replaced the drive said he didn't notice any pause in the activity lights on the other drives as he was replacing the failed drives. I did remember that it will only run for 24 hours in degraded mode, so I've set raid.timeout to a bigger number to buy me some time. Any suggestions on how to get it to recognize the 'new' disks? Thanks, Frank -- Frank Smith fsmith(a)hoovers.com Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501

3 3

RE: NSO-151 - NetApp SAN Exam-Dumps
by Sphar, Mike 17 Oct '06

17 Oct '06

I've only taken the first basic exam, a while back at the urging of my manager, and amusingly one of the things I got dinged on was that I apparently did not know how to request replacement parts on the website, despite the fact that I've been actually *doing* that periodically for about 10 years now and have never had a problem. Personally, my opinion is that while reading/studying the manuals will probably get you close, you will have a tough time passing the tests without actual administration experience. And conversely even years of administration experience may not be enough to pass without some backup studying as well, unless your lucky enough to work someplace that utilizes almost all available Netapp products/functionality. -- Michael W. Sphar - IS&T - Lead Systems Administrator SMBU Engineering Support Services, BMC Software -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Grey Friday Sent: Tuesday, October 17, 2006 10:55 AM To: etraitel(a)gmail.com Cc: sithesvaran thirumaran; toasters(a)mathworks.com Subject: Re: NSO-151 - NetApp SAN Exam-Dumps On 10/14/06, Eyal Traitel <etraitel(a)gmail.com> wrote: > Hi ! > > Except from being illegal, there is really no need for brain dumps - read the manuals left to right and you would do fine. The questions are on commands, command options, concepts etc that are described in details in the Block Admin guides. > > Eyal. > http://filers.blogspot.com > http://stupidstorage.blogspot.com> > > > > > On 10/14/06, sithesvaran thirumaran <sithu_t2000(a)yahoo.com> wrote: > Hi, > > How r u all. I need one help. I am planning to write NSO-151-SAN Exam, I need dumps for it. Can you please help me, if anyone have it? > > Thanks, > Sithesvaran. Do you Yahoo!? > Everyone is raving about the all-new Yahoo! Mail. > > > -- > Yours, > Eyal.

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters