Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s. I need to do some snapmirrors to re-balance some disk space. The -k option to throttle the transfer doesn't seem to be having any effect. I've tried modifying the placement of the -k but it doesn't seem to matter. I also tried to modify it after it was running and it doesn't seem to help either. Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system. Here is a cut of a sysstat 3 after starting it:
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age
73% 598 0 0 2091 349080 338016 271 0 0 7 71% 1019 0 0 2046 319892 324019 13114 0 0 0s 71% 2800 0 0 3880 330527 343528 17379 0 0 7 69% 1440 0 0 3405 330279 392647 22343 0 0 0s 87% 1614 0 0 2128 320151 607753 168553 0 0 0s 87% 827 0 0 5652 244701 584436 371689 0 0 0s 91% 897 0 0 4242 344072 680454 386373 0 0 0s
As you can see, the disk read/write counts go way up. This is causing some noticeable latency in the nfs access for clients. While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k n] -S source_system:source_volume
[dest_system:]dest_volume
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7
а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s
а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.com wrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Jeff,
AFAIK as previously mentioned, only _network_ speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume /prodvol /level=high system=low* Set the priority scheduling policy for volume /prodvol /to /high /compared to other volumes. Also prioritize system operations for the volume /low /compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one? - Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise <sklise@hotmail.com mailto:sklise@hotmail.com> wrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs.. Good luck. snapmirror update [-k /n/] -S /source_system/:/source_volume/ // [/dest_system/:]/dest_volume/ ------------------------------------------------------------------------ Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com <mailto:jeff.cleverley@avagotech.com> To: Toasters@teaparty.net <mailto:Toasters@teaparty.net> Greetings, I'm running 8.1.2P4, 7-mode on some 6290s.? I need to do some snapmirrors to re-balance some disk space.? The -k option to throttle the transfer doesn't seem to be having any effect.? I've tried modifying the placement of the -k but it doesn't seem to matter.? I also tried to modify it after it was running and it doesn't seem to help either.? Here is the command I'm running: snapmirror initialize -S sm15_3 -k 10000 new_sm15_3 If I'm understanding correctly, this should be allowing 10MB/s. The source and destination are on the same file system.? Here is a cut of a sysstat 3 after starting it: ?CPU???? NFS??? CIFS??? HTTP???? Net?? kB/s??? Disk?? kB/s??? Tape?? kB/s? Cache ????????????????????????????????? in??? out??? read? write??? read? write??? age 73%???? 598?????? 0?????? 0??? 2091 349080? 338016??? 271?????? 0????? 0???? 7 ?71%??? 1019?????? 0?????? 0??? 2046 319892? 324019? 13114?????? 0????? 0???? 0s ?71%??? 2800?????? 0?????? 0??? 3880 330527? 343528? 17379?????? 0????? 0???? 7 ?69%??? 1440?????? 0?????? 0??? 3405 330279? 392647? 22343?????? 0????? 0???? 0s ?87%??? 1614?????? 0?????? 0??? 2128 320151? 607753 168553?????? 0????? 0???? 0s ?87%???? 827?????? 0?????? 0??? 5652 244701? 584436 371689?????? 0????? 0???? 0s ?91%???? 897?????? 0?????? 0??? 4242 344072? 680454 386373?????? 0????? 0???? 0s As you can see, the disk read/write counts go way up.? This is causing some noticeable latency in the nfs access for clients.? While I really like the new hardware can pump data around, I need to be able to control it. What am I doing wrong? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
AFAIK as previously mentioned, only *network* speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume prodvol level=high system=low* Set the priority scheduling policy for volume *prodvol *to *high *compared to other volumes. Also prioritize system operations for the volume *low *compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one? - Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.com wrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
Hi Jeff,
just remember: system=... changes priority of system vs. user _in this volume_ Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume _level_ priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=*1* for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=*8*).
Therefore you better use *priority set default option=value [option=value...] * The /priority set default/ command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: *level *Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. *system* Set the relative priority for system-related operations (such as *SnapMirror *transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze <spgoetze@gmail.com mailto:spgoetze@gmail.com> wrote:
Hi Jeff, AFAIK as previously mentioned, only _network_ speed is affected. But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user. *priority set volume /prodvol /level=high system=low* Set the priority scheduling policy for volume /prodvol /to /high /compared to other volumes. Also prioritize system operations for the volume /low /compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued. So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror. Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior. HTH (Hi Oldtimers, still know this one? - Hope That Helps...) Sebastian On 01.08.2013 07:15, Jeff Cleverley wrote:I did try different positions for the -k. It didn't seem to matter. Jeff On Wed, Jul 31, 2013 at 8:45 PM, steve klise <sklise@hotmail.com <mailto:sklise@hotmail.com>> wrote: I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs.. Good luck. snapmirror update [-k /n/] -S /source_system/:/source_volume/ // [/dest_system/:]/dest_volume/ ------------------------------------------------------------------------ Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com <mailto:jeff.cleverley@avagotech.com> To: Toasters@teaparty.net <mailto:Toasters@teaparty.net> Greetings, I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running: snapmirror initialize -S sm15_3 -k 10000 new_sm15_3 If I'm understanding correctly, this should be allowing 10MB/s. The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it: аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age 73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it. What am I doing wrong? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 _______________________________________________ Toasters mailing list Toasters@teaparty.net <mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Sebastian,
What I got from NetApp today was change the snapmirror.volume.local_nwk_bypass.enable option to off. It is then supposed to be able to throttle using the -k argument. I haven't had a chance to try it out yet. I'll have to see what type of impact it appears to have on the network interface when these run.
Jeff
On Thu, Aug 1, 2013 at 3:18 AM, Sebastian Goetze spgoetze@gmail.com wrote:
Hi Jeff,
just remember: system=... changes priority of system vs. user *in this volume* Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume *level* priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=*1*for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=*8*).
Therefore you better use *priority set default option=value [option=value...] * The *priority set default* command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: *level *Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. *system* Set the relative priority for system-related operations (such as *SnapMirror *transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
AFAIK as previously mentioned, only *network* speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume prodvol level=high system=low* Set the priority scheduling policy for volume *prodvol *to *high *compared to other volumes. Also prioritize system operations for the volume *low *compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one? - Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.com wrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Hi Jeff,
I had the same issue with 8.1.2P4. The way I ended up working around the issue was to do the following:
1. Kick off a snapmirror initilaize w/out the -k option. 2. Use the snapmirror throttle command to throttle the snapmirror initialize down. Snapmirror throttle adjusts the throttleing of snapmirrors currently in progress. 3. (optional) if this will be an ongoing snapmirror relationship, update the snapmirror.conf file with the rate at which you would like to throttle the transfer going forward.
Essentially, there was a brief moment when the snapmirror initialize was not throttled between steps 1 ad 2. Not the best solution but it worked.
Phil
On Fri, Aug 2, 2013 at 6:36 PM, Jeff Cleverley <jeff.cleverley@avagotech.com
wrote:
Sebastian,
What I got from NetApp today was change the snapmirror.volume.local_nwk_bypass.enable option to off. It is then supposed to be able to throttle using the -k argument. I haven't had a chance to try it out yet. I'll have to see what type of impact it appears to have on the network interface when these run.
Jeff
On Thu, Aug 1, 2013 at 3:18 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
just remember: system=... changes priority of system vs. user *in this volume* Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume *level* priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=*1*for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=*8*).
Therefore you better use *priority set default option=value [option=value...] * The *priority set default* command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: *level *Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. *system* Set the relative priority for system-related operations (such as *SnapMirror *transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
AFAIK as previously mentioned, only *network* speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume prodvol level=high system=low* Set the priority scheduling policy for volume *prodvol *to *high *compared to other volumes. Also prioritize system operations for the volume *low *compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one?
- Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.com wrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Philbert,
I'll have to give that a try and verify. I thought I tried that on one, but I don't remember for sure.
Thanks for the pointer.
Jeff
On Mon, Aug 5, 2013 at 11:56 AM, Philbert Rupkins <philbertrupkins@gmail.com
wrote:
Hi Jeff,
I had the same issue with 8.1.2P4. The way I ended up working around the issue was to do the following:
- Kick off a snapmirror initilaize w/out the -k option.
- Use the snapmirror throttle command to throttle the snapmirror
initialize down. Snapmirror throttle adjusts the throttleing of snapmirrors currently in progress. 3. (optional) if this will be an ongoing snapmirror relationship, update the snapmirror.conf file with the rate at which you would like to throttle the transfer going forward.
Essentially, there was a brief moment when the snapmirror initialize was not throttled between steps 1 ad 2. Not the best solution but it worked.
Phil
On Fri, Aug 2, 2013 at 6:36 PM, Jeff Cleverley < jeff.cleverley@avagotech.com> wrote:
Sebastian,
What I got from NetApp today was change the snapmirror.volume.local_nwk_bypass.enable option to off. It is then supposed to be able to throttle using the -k argument. I haven't had a chance to try it out yet. I'll have to see what type of impact it appears to have on the network interface when these run.
Jeff
On Thu, Aug 1, 2013 at 3:18 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
just remember: system=... changes priority of system vs. user *in this volume* Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume *level* priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=*1*for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=*8*).
Therefore you better use *priority set default option=value [option=value...] * The *priority set default* command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: *level *Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. *system* Set the relative priority for system-related operations (such as *SnapMirror *transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
AFAIK as previously mentioned, only *network* speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume prodvol level=high system=low* Set the priority scheduling policy for volume *prodvol *to *high *compared to other volumes. Also prioritize system operations for the volume *low *compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one?
- Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.comwrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Hi Jeff,
You could also consider filer level throttling ( options replication ) and lock down the overall transfer rate either out or in bound. It's less granular but if you're concerned about impact to user traffic it's also the best way to catch multiple simultaneous mirroring jobs and prevent them from getting out of hand.
Colin Bieberstein
On 2013-08-05, at 11:56 AM, Philbert Rupkins philbertrupkins@gmail.com wrote:
Hi Jeff,
I had the same issue with 8.1.2P4. The way I ended up working around the issue was to do the following:
- Kick off a snapmirror initilaize w/out the -k option.
- Use the snapmirror throttle command to throttle the snapmirror initialize down. Snapmirror throttle adjusts the throttleing of snapmirrors currently in progress.
- (optional) if this will be an ongoing snapmirror relationship, update the snapmirror.conf file with the rate at which you would like to throttle the transfer going forward.
Essentially, there was a brief moment when the snapmirror initialize was not throttled between steps 1 ad 2. Not the best solution but it worked.
Phil
On Fri, Aug 2, 2013 at 6:36 PM, Jeff Cleverley jeff.cleverley@avagotech.com wrote:
Sebastian,
What I got from NetApp today was change the snapmirror.volume.local_nwk_bypass.enable option to off. It is then supposed to be able to throttle using the -k argument. I haven't had a chance to try it out yet. I'll have to see what type of impact it appears to have on the network interface when these run.
Jeff
On Thu, Aug 1, 2013 at 3:18 AM, Sebastian Goetze spgoetze@gmail.com wrote:
Hi Jeff,
just remember: system=... changes priority of system vs. user in this volume Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume level priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=1 for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=8).
Therefore you better use priority set default option=value [option=value...] The priority set default command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: level Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. system Set the relative priority for system-related operations (such as SnapMirror transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.com wrote:
Hi Jeff,
AFAIK as previously mentioned, only network speed is affected.
But did you think of the priority command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
priority set volume prodvol level=high system=low Set the priority scheduling policy for volume prodvol to high compared to other volumes. Also prioritize system operations for the volume low compared to user operations on the same volume. These options are enabled by this operation if priority on has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one? - Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.com wrote: > I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs.. > > Good luck. > > snapmirror update [-k n] -S source_system:source_volume > > > [dest_system:]dest_volume > > > Date: Wed, 31 Jul 2013 19:54:46 -0600 > Subject: Snapmirror throttle not working > From: jeff.cleverley@avagotech.com > To: Toasters@teaparty.net > > Greetings, > > I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running: > > > snapmirror initialize -S sm15_3 -k 10000 new_sm15_3 > > If I'm understanding correctly, this should be allowing 10MB/s. > > The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it: > > аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache > ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age > > 73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 > а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s > а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 > а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s > а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s > а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s > а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s > > As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it. > > What am I doing wrong? > > Thanks, > > Jeff > > -- > Jeff Cleverley > Unix Systems Administrator > 4380 Ziegler Road > Fort Collins, Colorado 80525 > 970-288-4611 > > _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Jeff,
No problem. The other neat thing with the snapmirror throttle command is that you can adjust it up or down on the fly. I had a snapmirror initialize that was running for days and there were times in which I needed to throttle it up and other times in which I needed to throttle it down. It ended up working out pretty well.
-Phil
On Tue, Aug 6, 2013 at 12:32 AM, Colin Bieberstein colin@bieberstein.cawrote:
Hi Jeff,
You could also consider filer level throttling ( options replication ) and lock down the overall transfer rate either out or in bound. It's less granular but if you're concerned about impact to user traffic it's also the best way to catch multiple simultaneous mirroring jobs and prevent them from getting out of hand.
Colin Bieberstein
On 2013-08-05, at 11:56 AM, Philbert Rupkins philbertrupkins@gmail.com wrote:
Hi Jeff,
I had the same issue with 8.1.2P4. The way I ended up working around the issue was to do the following:
- Kick off a snapmirror initilaize w/out the -k option.
- Use the snapmirror throttle command to throttle the snapmirror
initialize down. Snapmirror throttle adjusts the throttleing of snapmirrors currently in progress. 3. (optional) if this will be an ongoing snapmirror relationship, update the snapmirror.conf file with the rate at which you would like to throttle the transfer going forward.
Essentially, there was a brief moment when the snapmirror initialize was not throttled between steps 1 ad 2. Not the best solution but it worked.
Phil
On Fri, Aug 2, 2013 at 6:36 PM, Jeff Cleverley < jeff.cleverley@avagotech.com> wrote:
Sebastian,
What I got from NetApp today was change the snapmirror.volume.local_nwk_bypass.enable option to off. It is then supposed to be able to throttle using the -k argument. I haven't had a chance to try it out yet. I'll have to see what type of impact it appears to have on the network interface when these run.
Jeff
On Thu, Aug 1, 2013 at 3:18 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
just remember: system=... changes priority of system vs. user *in this volume* Use this if users are working on this vol and are impacted (e.g. the source vol).
Use the volume *level* priority to prioritize between volumes, either giving higher priority to volumes being used (recommended), or by giving lower priority to the SnapMirrored volumes.
But beware: if you have 50 vols, they will all be in the default bucket (prio Medium=50) with possibly lower combined prio (effectively prio=*1*for every volume in the default bucket) than the 2 SnapMirror vols (source & destination, e.g. VeryLow=*8*).
Therefore you better use *priority set default option=value [option=value...] * The *priority set default* command manages the default priority policy, which is applied to volumes without any specific priority policy. The following options may be specified: *level *Set the priority level for operations are sent to the volume when compared to other volumes. The value might be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 8 (VeryLow) to 92 ( VeryHigh). A volume with a higher priority level receives more resources than a volume with a lower priority level. The default value is Medium. *system* Set the relative priority for system-related operations (such as *SnapMirror *transfers) that are sent to the volume when compared to user operations that are sent to the volume. The valuemight be one of VeryHigh, High, Medium, Low, VeryLow, or a numeric value from 4 (VeryLow) to 96 (VeryHigh). The default value is Medium.
Also be aware that system operations include also things like WAFL-tree updates (metadata...). So if you observe negative effects (e.g. in a high-file-count volume), give this volume a higher system priority or switch priority off when not needed.
HTH
Sebastian
On 01.08.2013 10:04, Jeff Cleverley wrote:
Sebastian,
I had not thought about the priority option. I'll see if it makes sense tomorrow. The slowness seems to affect access to the entire filer. Changing the system priority to low on the source and destination volumes might do the trick though. Thanks for the idea.
Jeff
On Thu, Aug 1, 2013 at 12:58 AM, Sebastian Goetze spgoetze@gmail.comwrote:
Hi Jeff,
AFAIK as previously mentioned, only *network* speed is affected.
But did you think of the *priority *command? There you can change relative (!) priorities, e.g. system (-> SnapMirror) vs. user.
*priority set volume prodvol level=high system=low* Set the priority scheduling policy for volume *prodvol *to *high *compared to other volumes. Also prioritize system operations for the volume *low *compared to user operations on the same volume. These options are enabled by this operation if *priority on* has been previous issued.
So you would set 'level=high' on the volumes where the users are impacted and 'system=low' and maybe "level=low" (if the source isn't the one where the users are impacted) to the volumes involved in the SnapMirror.
Don't forget to set 'priority on'. And maybe 'priority off' after the snapmirror is through if you want to go back to the previous behavior.
HTH (Hi Oldtimers, still know this one?
- Hope That Helps...)
Sebastian
On 01.08.2013 07:15, Jeff Cleverley wrote:
I did try different positions for the -k. It didn't seem to matter.
Jeff
On Wed, Jul 31, 2013 at 8:45 PM, steve klise sklise@hotmail.comwrote:
I am not sure, but you may want to change your syntax to put the -k before the -S; Not sure if that really matters, but this is what I found in one of the docs..
Good luck.
snapmirror update [-k *n*] -S *source_system*:*source_volume*
[*dest_system*:]*dest_volume*
Date: Wed, 31 Jul 2013 19:54:46 -0600 Subject: Snapmirror throttle not working From: jeff.cleverley@avagotech.com To: Toasters@teaparty.net
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s.а I need to do some snapmirrors to re-balance some disk space.а The -k option to throttle the transfer doesn't seem to be having any effect.а I've tried modifying the placement of the -k but it doesn't seem to matter.а I also tried to modify it after it was running and it doesn't seem to help either.а Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system.а Here is a cut of a sysstat 3 after starting it:
аCPUаааа NFSааа CIFSааа HTTPаааа Netаа kB/sааа Diskаа kB/sааа Tapeаа kB/sа Cache ааааааааааааааааааааааааааааааааа inааа outааа readа writeааа readа writeааа age
73%аааа 598аааааа 0аааааа 0ааа 2091 349080а 338016ааа 271аааааа 0ааааа 0аааа 7 а71%ааа 1019аааааа 0аааааа 0ааа 2046 319892а 324019а 13114аааааа 0ааааа 0аааа 0s а71%ааа 2800аааааа 0аааааа 0ааа 3880 330527а 343528а 17379аааааа 0ааааа 0аааа 7 а69%ааа 1440аааааа 0аааааа 0ааа 3405 330279а 392647а 22343аааааа 0ааааа 0аааа 0s а87%ааа 1614аааааа 0аааааа 0ааа 2128 320151а 607753 168553аааааа 0ааааа 0аааа 0s а87%аааа 827аааааа 0аааааа 0ааа 5652 244701а 584436 371689аааааа 0ааааа 0аааа 0s а91%аааа 897аааааа 0аааааа 0ааа 4242 344072а 680454 386373аааааа 0ааааа 0аааа 0s
As you can see, the disk read/write counts go way up.а This is causing some noticeable latency in the nfs access for clients.а While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing listToasters@teaparty.nethttp://www.teaparty.net/mailman/listinfo/toasters
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Maybe a bug? Try creating an entry in /etc/snapmirror.conf with the kbs=10000 option to see if it makes a difference.
--tmac
*Tim McCarthy* *Principal Consultant*
Clustered ONTAP Clustered ONTAP NCDA ID: XK7R3GEKC1QQ2LVD RHCE5 805007643429572 NCSIE ID: C14QPHE21FR4YWD4 Expires: 08 November 2014 Expires w/release of RHEL7 Expires: 08 November 2014
On Wed, Jul 31, 2013 at 9:54 PM, Jeff Cleverley < jeff.cleverley@avagotech.com> wrote:
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s. I need to do some snapmirrors to re-balance some disk space. The -k option to throttle the transfer doesn't seem to be having any effect. I've tried modifying the placement of the -k but it doesn't seem to matter. I also tried to modify it after it was running and it doesn't seem to help either. Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system. Here is a cut of a sysstat 3 after starting it:
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age
73% 598 0 0 2091 349080 338016 271 0 0 7 71% 1019 0 0 2046 319892 324019 13114 0 0 0s 71% 2800 0 0 3880 330527 343528 17379 0 0 7 69% 1440 0 0 3405 330279 392647 22343 0 0 0s 87% 1614 0 0 2128 320151 607753 168553 0 0 0s 87% 827 0 0 5652 244701 584436 371689 0 0 0s 91% 897 0 0 4242 344072 680454 386373 0 0 0s
As you can see, the disk read/write counts go way up. This is causing some noticeable latency in the nfs access for clients. While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I have two possible explanations.
1) That because there is no network involved in this transfer, the filer is ignoring what you're specifying as the network bandwidth throttle. The text in the man page does say "The -k option sets the maximum speed at which data is transferred *over the network* in kilobytes per second."
2) Is the destination growing faster than your specified threshold? Maybe (for some layout reason, etc.) it requires a LOT of disk IO, in order to achieve the amount of actual data flow that you specified?
(I'm betting more on option #1 though.)
Davin.
On Wed, Jul 31, 2013 at 9:54 PM, Jeff Cleverley < jeff.cleverley@avagotech.com> wrote:
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s. I need to do some snapmirrors to re-balance some disk space. The -k option to throttle the transfer doesn't seem to be having any effect. I've tried modifying the placement of the -k but it doesn't seem to matter. I also tried to modify it after it was running and it doesn't seem to help either. Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system. Here is a cut of a sysstat 3 after starting it:
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age
73% 598 0 0 2091 349080 338016 271 0 0 7 71% 1019 0 0 2046 319892 324019 13114 0 0 0s 71% 2800 0 0 3880 330527 343528 17379 0 0 7 69% 1440 0 0 3405 330279 392647 22343 0 0 0s 87% 1614 0 0 2128 320151 607753 168553 0 0 0s 87% 827 0 0 5652 244701 584436 371689 0 0 0s 91% 897 0 0 4242 344072 680454 386373 0 0 0s
As you can see, the disk read/write counts go way up. This is causing some noticeable latency in the nfs access for clients. While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I wondered about option 1 since I also saw the same description in the man pages. It still seems there should be a way to control internal transfers.
As for option 2, it is running considerably faster than expected. If I did the math correctly, 10MB/s would be 360GB/hour. The snapmirror completed and averaged over 900GB/hour.
I'll probably open a call with NetApp in the morning.
Thanks,
Jeff
On Wed, Jul 31, 2013 at 8:57 PM, Davin Milun davin.milun@gmail.com wrote:
I have two possible explanations.
- That because there is no network involved in this transfer, the filer
is ignoring what you're specifying as the network bandwidth throttle. The text in the man page does say "The -k option sets the maximum speed at which data is transferred *over the network* in kilobytes per second."
- Is the destination growing faster than your specified threshold? Maybe
(for some layout reason, etc.) it requires a LOT of disk IO, in order to achieve the amount of actual data flow that you specified?
(I'm betting more on option #1 though.)
Davin.
On Wed, Jul 31, 2013 at 9:54 PM, Jeff Cleverley < jeff.cleverley@avagotech.com> wrote:
Greetings,
I'm running 8.1.2P4, 7-mode on some 6290s. I need to do some snapmirrors to re-balance some disk space. The -k option to throttle the transfer doesn't seem to be having any effect. I've tried modifying the placement of the -k but it doesn't seem to matter. I also tried to modify it after it was running and it doesn't seem to help either. Here is the command I'm running:
snapmirror initialize -S sm15_3 -k 10000 new_sm15_3
If I'm understanding correctly, this should be allowing 10MB/s.
The source and destination are on the same file system. Here is a cut of a sysstat 3 after starting it:
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age
73% 598 0 0 2091 349080 338016 271 0 0 7 71% 1019 0 0 2046 319892 324019 13114 0 0 0s 71% 2800 0 0 3880 330527 343528 17379 0 0 7 69% 1440 0 0 3405 330279 392647 22343 0 0 0s 87% 1614 0 0 2128 320151 607753 168553 0 0 0s 87% 827 0 0 5652 244701 584436 371689 0 0 0s 91% 897 0 0 4242 344072 680454 386373 0 0 0s
As you can see, the disk read/write counts go way up. This is causing some noticeable latency in the nfs access for clients. While I really like the new hardware can pump data around, I need to be able to control it.
What am I doing wrong?
Thanks,
Jeff
-- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-
Colin Bieberstein -
Davin Milun -
Jeff Cleverley -
Philbert Rupkins -
Sebastian Goetze -
steve klise -
tmac