toasters August 2018

toasters@lists.teaparty.net

16 participants
12 discussions

Creating a readonly user to monitor NetApp cluster via check_mk
by Sylvain Robitaille 20 Dec '18

20 Dec '18

Hello fellow toasters users ... We're in the process of deploying a pair of new CDOT filers and one of the things we'd like to setup with them is having check_mk (nagios and rrd, under the hood) monitor various attributes. We have a working instance at the moment using snmp to query the filers, but check_mk also provides a "plugin" that claims to work with the filer's web interface, so we'd like to examine that. To that end, I've setup a read-only user on the filer, that I intend will be able to access the web interface: security login create -vserver fc1-na -user-or-group-name nul_cmk \ -application http -authentication-method password -role readonly \ -comment "check_mk monitoring" The user shows up when I "show": fc1-na::> security login show -vserver fc1-na -user-or-group-name nul_cmk Vserver: fc1-na Second User/Group Authentication Acct Authentication Name Application Method Role Name Locked Method -------------- ----------- ------------- ---------------- ------ -------------- nul_cmk http password readonly no none The readonly role is unchanged from the OS default: fc1-na::> security login role show -vserver fc1-na -role readonly Role Command/ Access Vserver Name Directory Query Level ---------- ------------- --------- ----------------------------------- -------- fc1-na readonly DEFAULT readonly security none security login password all security login role show-user-capability all set all 5 entries were displayed. Trouble is that if I try to login to the web interface as that user, the browser stays "stuck" at the "Loading system details..." spinner widget. It's been there now for about two hours as I type this. Of course, check_mk's plugin logs a timeout while trying to access the interface. I fully suspect that I'm missing something, of course, but I'm afraid I haven't been able to find what that is. If someone out there has experience creating such a user for readonly access, a pointer to where I should be looking and/or documentation I should be reading would be hugely appreciated. (cli is preferred because it's easier to document what I've done that way, but if someone points me to resources in the web interface I'm sure I'll be able to find what I need at the cli.) -- ---------------------------------------------------------------------- Sylvain Robitaille syl(a)encs.concordia.ca Systems analyst / AITS Concordia University Faculty of Engineering and Computer Science Montreal, Quebec, Canada ----------------------------------------------------------------------

1 1

vol clone split
by Chris Hague 12 Oct '18

12 Oct '18

Hi Toasters, I have a number of volumes running from a common snapshot and would like to split them out. All these volumes are snapmirrored to our DR site. However, when I try to perform a vol clone split, it complains about an in-use snapshot. Reading further, it appears that the split will delete all snapshots, so doesn't like the snapmirror snap. What is the best way to split these volumes? Ideally without having to re-baseline \ lose my snapmirror destination volume \ snaps. ONTAP v9.1P2 Error below; Time Node Severity Event ------------------- ---------------- ------------- --------------------------- 8/30/2017 09:02:17 filer1 INFORMATIONAL mgmtgwd.jobmgr.jobcomplete.failure: Job "Clone Split" [id 6716] (Split vol1) completed unsuccessfully: Volume has locked snapshots (1). 8/30/2017 09:02:17 filer1 DEBUG wafl.config.req.error: Config request 'VV_CONFIG_REQ_CLONE_SPLIT_START' by WAFL for volume vol1 is not being processed because of the error: '387 - CR_VV_VOLUME_SNAPSHOTS'. 8/30/2017 09:02:17 filer1 NOTICE wafl.vv.clone.splitBusySnap: Volume 'vol1 has locked snapshots, not splitting. KR, Chris.

4 8

Related to New Cmode filer Setup
by Yuvaraja Shivaram 28 Aug '18

28 Aug '18

Hi Everyone, Any one has got all the commands to setup new cluster in Dataontap C-mode environment...9.xx version... if it is in Best Standard would be fine.... Example steps: Cluster mode setup Joining cluster Creating Vlan Creating Interface Groups Creating Lifs Creating all the roles of cluster , data , node-mgmt ,intercluster cluster-mgmt Creating aggregates (both DP and Raidtec) Creating Vservers (NFS, CIFS, ISCSI) Creating failovergroups Creating volumes Creating Export-policy and rule set... Regards Yuvaraj

3 2

Be careful about snaplock
by Basil 22 Aug '18

22 Aug '18

Just going through an experience I figured I would share with the list: we ordered a small Snaplock cluster, sized so that all disks would be used for the Snaplock aggregate. Well, as it turns out, you must always size enough disk for one regular aggregate: SVM root volumes are not supported on root aggregates nor on Snaplock aggregates.

2 2

revert status
by Alexander Griesser 21 Aug '18

21 Aug '18

Hey toasters, is there any way to see the status of a node-shell revert_to command? I just had to revert one of our systems and unfortunately the SSH connection broke and I'm not seeing any progress here anymore :-/ ::> run local revert_to 9.1 You are about to revert the system to work with Data ONTAP 9.1 The system will be rebooted immediately after the conversion process completes. Make sure that you have installed Data ONTAP 9.1 onto the boot device, or you will have to run "revert_to" again. Are you sure you want to proceed? [yes/no]? yes debug.nodewatchdog.level: 2 -> 2 FlexScale PCS is not active and cannot be configured. The management gateway server restarted. Waiting to see if the connection can be reestablished I can connect to the system fine again now and I can not run the revert_to again (it says that 9.1 is not supported, only 8.3) - so I guess it's either done already or still in progress. I couldn't find any command that would show me if this task is still running - any ideas? Thanks, Alexander Griesser Head of Systems Operations ANEXIA Internetdienstleistungs GmbH E-Mail: AGriesser(a)anexia-it.com<mailto:AGriesser@anexia-it.com> Web: http://www.anexia-it.com<http://www.anexia-it.com/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

2 3

Netapp C-Mode:- how to covert Global export policy into individual users access
by Yuvaraja Shivaram 20 Aug '18

20 Aug '18

Hi Team, Is there any way to convert global access users to convert into individual access. Say for example if any of the policy is having global access of client...like 0.0.0.0/0 and we would like to convert this into individual host access by removing this global entry "0.0.0.0\0". then how can we achieve this without disruptive activity... can we add all the individual hosts one by one which are accessing through this global access and then disable the global access parameter from export-policy rule or do we need to shutdown the application/databases and then modify the global parameter.... Please let us know if any of them worked on these kind of activity, my goal is to modify this parameter without bringing down the hosts.... Regards Yuvaraja

2 5

Re: Failed Aggregate 0
by tmac 20 Aug '18

20 Aug '18

Ok, I have seen the last ASUP via this EMAIL and the current disk layout. Here they are for everyone: Last known ASUP: Aggregate aggr0 (online, raid_dp, degraded) (block checksums) Plex /aggr0/plex0 (online, normal, active) RAID group /aggr0/plex0/rg0 (double degraded, block checksums) RAID Disk Device Model Number Serial Number VBN Start VBN End --------- ------ ------------ ------------- --------- ------- dparity 0a.44 X279_HVPBP288F15 JLVBYWVC - - parity 0a.48 X279_S15K5288F15 3LM42GHN00009839N66A - - data 0a.49 X279_S15K5288F15 3LM1TA6C000098080J6N 0 69626751 data 0a.16 X279_HVPBP288F15 JLVJV9TC 69626752 139253503 data 0a.50 X279_HVIPB288F15 J8YMUD4C 139253504 208880255 data 0a.45 X279_S15K7288F15 6SJ5HVVK0000B24505C9 208880256 278507007 data 0a.51 X279_S15K5288F15 3LM1T7DZ00009808S8XF 348133760 417760511 data 0a.18 X279_HVPBP288F15 JLVJVENC 417760512 487387263 data 0a.52 X279_HVPBP288F15 JLVHYUNC 487387264 557014015 data 0a.19 X279_HVPBP288F15 JLVJWUUC 557014016 626640767 data 0a.20 X279_HVPBP288F15 JLVJWJ3C 626640768 696267519 data 0a.22 X279_HVPBP288F15 JLVK7NJC 765894272 835521023 data 0a.23 X279_HVPBP288F15 JLVJUE9C 835521024 905147775 data 0a.25 X279_HVPBP288F15 JLVJWKRC 905147776 974774527 RAID group /aggr0/plex0/rg1 (double degraded, block checksums) RAID Disk Device Model Number Serial Number VBN Start VBN End --------- ------ ------------ ------------- --------- ------- data 0a.33 X279_HVPBP288F15 JLVJUENC 974774528 1044401279 data 0a.26 X279_HVPBP288F15 JLVKVL7C 1044401280 1114028031 data 0a.34 X279_HVPBP288F15 JLVBYUWC 1114028032 1183654783 data 0a.27 X279_HVPBP288F15 JLVBZL3C 1183654784 1253281535 data 0a.35 X279_HVPBP288F15 JLVBYVYC 1253281536 1322908287 data 0a.28 X279_HVPBP288F15 JLVJV9RC 1322908288 1392535039 data 0a.36 X279_HVPBP288F15 JLVBYSBC 1392535040 1462161791 data 0a.29 X279_HVPBP288F15 JLVJWP5C 1462161792 1531788543 data 0a.37 X279_HVPBP288F15 JLVBYS1C 1531788544 1601415295 data 0a.38 X279_HVPBP288F15 JLVBYW7C 1601415296 1671042047 data 0a.39 X279_HVPBP288F15 JLVBYWWC 1671042048 1740668799 data 0a.40 X279_HVPBP288F15 JLVBYRRC 1740668800 1810295551 data 0a.41 X279_HVPBP288F15 JLVBYPWC 1810295552 1879922303 data 0a.60 X279_HVPBP288F15 JLVJ6YZC 1879922304 1949549055 RAID group /aggr0/plex0/rg2 (normal, block checksums) RAID Disk Device Model Number Serial Number VBN Start VBN End --------- ------ ------------ ------------- --------- ------- dparity 0a.54 X279_S15K5288F15 3LM42HL100009839NXS7 - - parity 0a.55 X279_S15K5288F15 3LM45JAD00009839N5JZ - - data 0a.56 X279_S15K5288F15 3LM47SPT00009840R2EL 1949549056 2019175807 data 0a.57 X279_HVIPB288F15 J8YT82MC 2019175808 2088802559 data 0a.58 X279_HVPBP288F15 JLVGNHKC 2088802560 2158429311 data 0a.59 X279_HVPBP288F15 JLVJGJVC 2158429312 2228056063 And here is the "aggr status -r" output from MAINT mode: Aggregate aggr0 (failed, Aug 19 22:47:17 [localhost:disk.failmsg:error]: Disk 0a.21 (JLVKV9SC): non-persistent message received. 0 [NETAPP X279_HVPBP288F15 NA02] S/N [JLVKV9SC] raid_dp, partial) (block checksums) Aug 19 22:47:17 [localhost:disk.failmsg:error]: Disk 0a.32 (J8VYJBHC): non-persistent message received. 0 [NETAPP X279_HVIPB288F15 NA01] S/N [J8VYJBHC] Plex /aggr0/plex0 (offline, failed, inactive) Aug 19 22:47:17 [localhost:raid.fdr.failed.ok:info]: Disk 0a.21 Shelf 1 Bay 5 [NETAPP X279_HVPBP288F15 NA02] S/N [JLVKV9SC] successfully deleted from spare pool RAID group /aggr0/plex0/rg0 (partial, block checksums) Aug 19 22:47:17 [localhost:raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP X279_HVIPB288F15 NA01] S/N [J8VYJBHC] successfully deleted from spare pool RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity FAILED N/A 272000/ - parity 0a.48 0a 3 0 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 data 0a.49 0a 3 1 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 data 0a.16 0a 1 0 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.50 0a 3 2 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.45 0a 2 13 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 data FAILED N/A 272000/ - data 0a.51 0a 3 3 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 data 0a.18 0a 1 2 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.52 0a 3 4 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.19 0a 1 3 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.20 0a 1 4 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data FAILED N/A 272000/ - data 0a.22 0a 1 6 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.23 0a 1 7 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.25 0a 1 9 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 Raid group is missing 3 disks. RAID group /aggr0/plex0/rg1 (double degraded, block checksums) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity FAILED N/A 272000/ - parity FAILED N/A 272000/ - data 0a.33 0a 2 1 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.26 0a 1 10 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.34 0a 2 2 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.27 0a 1 11 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.35 0a 2 3 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.28 0a 1 12 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.36 0a 2 4 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.29 0a 1 13 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.37 0a 2 5 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.38 0a 2 6 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.39 0a 2 7 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.40 0a 2 8 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.41 0a 2 9 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.60 0a 3 12 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 RAID group /aggr0/plex0/rg2 (degraded, block checksums) RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- dparity 0a.54 0a 3 6 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 parity FAILED N/A 272000/ - data 0a.56 0a 3 8 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 data 0a.57 0a 3 9 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.58 0a 3 10 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 data 0a.59 0a 3 11 FC:A 0 FCAL 15000 272000/557056000 274845/562884296 Unassimilated aggr0 disks RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks) --------- ------ ------------- ---- ---- ---- ----- -------------- -------------- orphaned 0a.72 0a 4 8 FC:A 0 FCAL 15000 272000/557056000 280104/573653840 What is seemingly ODD to me is the FAILED disks and they way they are displayed. See the Highlighted entries above. >From the ASUP message we know that 0a.33 and 0a.26 are dparity and parity but they are shifted and showing as data. In the last RIAD GROUP 0a.55 is failed and showing. Weird. My GUT is telling me some sort of disk enumeration bug is being encountered on this version of ONTAP. Unless someone has actually been through this before, this is going to require opening a case with NetApp in some form or fashion. There may need to be some lower level disk label corrections that need to occur but I am not 100% sure on this. Did the drives all fail at once? Was it one at a time and never replaced? just curious. You *might* be able to get a one-time support call. If you are willing to pony up the $$ to re-enable support, they might help you out now for free. OR you can purchase a one-time support. I think I have heard is like $5k-$10k or something. --tmac

2 1

Failed Aggregate 0
by dhichandler 19 Aug '18

19 Aug '18

I have a FAS3140 running OnTap 8.2.5 and I have had a double partity disk failure in one of our raid groups and looking for help recovering. The unit is out of warranty and support. Any help is appreciated. ----- Christopher Chandler CEO DH Innovations, LLC -- Sent from: http://network-appliance-toasters.10978.n7.nabble.com/

4 14

Re: volume move and SFO
by Francis Kim 11 Aug '18

11 Aug '18

Here’s a repost, with a link to the TR rather than the actual pdf. https://www.netapp.com/us/media/tr-4075.pdf Francis Kim Cell: 415-606-2525 Direct: 510-644-1599 x334 fkim(a)berkcom.com<mailto:fkim@berkcom.com> www.berkcom.com On Aug 11, 2018, at 12:02 PM, Francis Kim <fkim(a)BERKCOM.com<mailto:fkim@BERKCOM.com>> wrote: Ian, <tr-4075 DataMotion for Volumes NetApp clustered Data ONTAP 8.2 and 8.3.pdf> With recurring controller and shelf refreshes, I’ve become a frequent vol mover (and a fan of it) over the last two years. NDO aspect of this task is obviously a big draw. Attached TR-4075, dated March 2015, has the most detailed explanation of vol moves I’ve been able to find, but its discussion is limited to 8.2 vs 8.3. Nothing about ONTAP 9 in this TR, so YMMV. In this TR much is made about the two phases of a vol move, the iterative (baseline and updates) and cutover, with respective to what other operations are (in)compatible with the progress of a vol move. During the iterative phase, a vol move in 8.2 would have to be restarted after a FO/GB while in 8.3 they would resume at its most recent checkpoint. However, once the cutover phase has been entered, a vol move in 8.2 would survive a FO/GB if it had crossed its “point of no return” checkpoint, while an 8.3 vol move is mutually exclusive with a FO/GB, suggesting a cutover would have to be reattempted afterward. I’ve not been able to find under 9.1 (even in diag mode) any information specific to these checkpoints. Not sure whether “Bytes sent” is a checkpoint. Since documentation on vol moves is generally skinny and this TR is now over three years and five releases old, a lab run is probably a good move if you have access to gear. Francis Kim Cell: 415-606-2525 Direct: 510-644-1599 x334 fkim(a)berkcom.com<mailto:fkim@berkcom.com> www.berkcom.com<http://www.berkcom.com> On Aug 10, 2018, at 2:55 AM, Ian Ehrenwald <Ian.Ehrenwald(a)hbgusa.com<mailto:Ian.Ehrenwald@hbgusa.com>> wrote: Based on a Justin Parisi article that Douglas pointed out earlier in the thread, I'm going with the assumption that the cluster would veto the giveback if a move was in progress. I actually just got a new FAS2720 HA pair for my lab this week, so when it's set up and I have a bunch of junk data and volumes on it for testing, I will see what happens if we were to do a move, takeover, and giveback in Real Life (tm). I can't make any promises on an ETA for answer, but as soon as I do it I'll send an update. ________________________________________ From: RUIS Henk <henk.ruis(a)axians.com<mailto:henk.ruis@axians.com>> Sent: Friday, August 10, 2018 4:28:30 AM To: Ian Ehrenwald; Mike Gossett Cc: Toasters Subject: RE: volume move and SFO Hi, Great, but now we still don't know if it's possible to move a volume during maintenance ;-( Met vriendelijke groet / Kind regards, Henk Ruis Technical Consultant -----Oorspronkelijk bericht----- Van: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> <toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net>> Namens Ian Ehrenwald Verzonden: donderdag 9 augustus 2018 18:13 Aan: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> CC: Toasters <toasters(a)teaparty.net<mailto:toasters@teaparty.net>> Onderwerp: Re: volume move and SFO Good afternoon Update, as promised. The volume move completed early this morning so there will be no conflict with our maintenance tonight. Inline compaction is AWESOME - it has saved us over 16TB on this new SSD aggregate. Crazy stuff. ________________________________________ From: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> Sent: Wednesday, August 8, 2018 2:37:35 PM To: Ian Ehrenwald Cc: Douglas Siggins; Toasters Subject: Re: volume move and SFO Hi Ian, The good news is that, forgetting about what it is estimating, we've seen that in 24 hours 21TB has been copied and. Hopefully another 30 hours or whatever is sufficient for the remaining 9.5TB - thanks for sharing. I'm interested to know the result of the SFO - but if was me I'd try to push the maintenance back to be safe. Or open a ticket with support and see if they can tell you what to expect On Aug 8, 2018, at 1:29 PM, Ian Ehrenwald <Ian.Ehrenwald(a)hbgusa.com<mailto:Ian.Ehrenwald@hbgusa.com>> wrote: Hi Mike Somewhat redacted output from a few minutes ago: MyCluster1::> volume move show -instance Vserver Name: mySvm Volume Name: aVeryLargeVolume Actual Completion Time: - Bytes Remaining: 9.44TB Destination Aggregate: aggr_ssd_3800g_c1n4 Detailed Status: Transferring data: 20.96TB sent. Estimated Time of Completion: Thu Aug 09 01:58:40 2018 Managing Node: Clus1-Node1 Percentage Complete: 68% Move Phase: replicating Estimated Remaining Duration: 11:45:25 Replication Throughput: 233.9MB/s Duration of Move: 23:47:43 Source Aggregate: aggr_sas_600g_c1n1 Start Time of Move: Tue Aug 07 14:25:38 2018 Move State: healthy Is Source Volume Encrypted: false Encryption Key ID of Source Volume: - Is Destination Volume Encrypted: false Encryption Key ID of Destination Volume: - MyCluster1::> Depending on source filer load, I've seen Replication Throughput anywhere from 25MB/s through 400MB/s and higher. The source aggregate is 192x600GB SAS on a filer with 2TB FlashCache and it sometimes sees periods of 30K IOPS. I guess the point is ETA has been anywhere from just about now all the way out to this coming Sunday. There's a good chance that the move will complete before tomorrow evening according to these numbers though. If it doesn't complete on time, I guess we'll find out what effect, if any, SFO has (unless I can easily reschedule this HW maintenance). ________________________________________ From: Mike Gossett <cmgossett(a)gmail.com<mailto:cmgossett@gmail.com>> Sent: Wednesday, August 8, 2018 13:59 To: Douglas Siggins Cc: Ian Ehrenwald; Toasters Subject: Re: volume move and SFO How long did it take to get to your 60% point that you reference? Some quick math says 30TB at 400MB/s should complete in about 22 hours. If you're 60 percent done, and you have a day and a half, then it should complete before tomorrow night, right? _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters<https://protect-us.mimecast.com/s/bF00ClYkMYUo703Os9r2z6?domain=teaparty.net> _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

1 0

volume move and SFO
by Ian Ehrenwald 10 Aug '18

10 Aug '18

Good morning I have a four node cluster, nodes1/2 are SAS/SATA and nodes 3/4 are AFF. I have a long running volume move going from node 1 to node 4. Long running, like 30TB+, and it's about 60% done. I need to do some hardware maintenance on nodes 1 and 2 tomorrow evening (install additional FlashCache cards). Will a takeover of node 2 by node 1, then a takeover of node 1 by node 2, interrupt this volume move? I can't seem to find much in the way of documentation about what happens during a SFO and a volume move, but it's possible I'm just not looking hard enough. Thanks for any insights. Ian Ehrenwald Senior Infrastructure Engineer Hachette Book Group, Inc. 1.617.263.1948 / ian.ehrenwald(a)hbgusa.com

4 9

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters August 2018