Hello
Im having some really weird problems with backing up our snapmirrors and wondering if anyone has any ideas.
Here is the first setup..
We have a 3050 filer (filerB) with 16TB split up in 2 aggrs
filer B is used as a snapmirror source where our other filers snapmirror all their data onto.
Problem:
on filer A we have 3 sepearte flexvols with 1 qtree in each flex vol. Each qtree has around 160gb and 4 million files in it.
eg.
/vol/vola/vola /vol/volb/volb /vol/volc/volc
on filer B i created a 600gb flex vol called filerA to snapmirror into
We were using Qtree snapmirrors onto filer B so the results looked like this
qtree: This command is deprecated; using qtree status. Volume Tree Style Oplocks Status -------- -------- ----- -------- --------- vol0 unix enabled normal filerA unix enabled normal filerA vola unix enabled snapmirrored filerA volb unix enabled snapmirrored filerA volc unix enabled snapmirrored
filerB> df /vol/filerA/ 838860800 488597596 350263204 58% /vol/filerA/ /vol/filerA/.snapshot 209715200 14934800 194780400 7% /vol/filerA/.snapshot
filerB> df -i
Filesystem iused ifree %iused Mounted on /vol/filerA/ 17202153 14674536 54% /vol/filerA/
filerB> snapmirror status Source Destination State Lag Status filerA:/vol/vola/vola filerB:/vol/filerA/vola Snapmirrored 152:02:24 Idle
When we backup /vol/filerA we are getting around 20-26meg a second to tape.
After a few weeks we see that the snapmirrors are taking a very long time (2-3 hours) to sync as each time it snapmirrors its doing a file copy. Ie works out what inodes have changed then copies data over.
So we decide to cutover to Volume based snapmirrors as its heaps faster.
now on filerB i create 3 restricted volumes and we are snapmirroring to each volume to their restricted volume. This is very very fast as its block level and its syncing in under 15 min
this is now how looks like the following on filerB
/vol/filerAvola/ 251658240 153210208 98448032 61% /vol/filerAvola/ /vol/filerAvola/.snapshot 62914560 14631824 48282736 23% /vol/filerAvola/.snapshot /vol/filerAvolb/ 251658240 154023020 97635220 61% /vol/filerAvolb/ /vol/filerAvolb/.snapshot 62914560 14196416 48718144 23% /vol/filerAvolb/.snapshot /vol/filerAvolc/ 251658240 154578208 97080032 61% /vol/filerAvolc/ /vol/filerAvolc/.snapshot 62914560 14897572 48016988 24% /vol/filerAvolc/.snapshot
a vol status looks like the following
filerAvola online raid_dp, flex snapmirrored=on, create_ucode=on, snapmirrored maxdirsize=20971, read-only fs_size_fixed=on, guarantee=volume(disabled)
we have 3 of these snapmirrored volumes..
filerB:vola filerA:vola Snapmirrored 04:20:26 Idle filerB:volb filerA:volb Snapmirrored 04:20:26 Idle filerB:volc filerA:volc Snapmirrored 04:20:26 Idle
now the problem is when we go to back these upto tape we are only getting 3 - 7 meg a second to tape.
This is the problem.
How come when we backup the large volume (/vol/filerA/) we get 26 meg a sec but when we backup the smaller restricted volumes (/vol/filerAvola/) we are only getting 3-7 ?
we are using Netbackup NDMP to a local tape drive
we have another restricted volume on filerB which is 140Gb of oracle data and that gets 30-50meg a second to tape.
Qtree snapmirror writes data on destination sequentially, thus your files get optimized for sequential load like backups.
Volume snapmirror transfers and writes data 1-to-1 with source volume, thus retaining original fragmentation. Depending on how heavy original volume is fragmented, that well explains the difference, especially if there are many small files.
Oracle is normally using small number of large files; large files are processed really more efficiently during backups.
With best regards
andrey
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Greg Wilson Sent: Wednesday, August 02, 2006 10:35 AM To: toasters@mathworks.com Subject: Snapmirror and backup weirdness
Hello
Im having some really weird problems with backing up our snapmirrors and wondering if anyone has any ideas.
Here is the first setup..
We have a 3050 filer (filerB) with 16TB split up in 2 aggrs
filer B is used as a snapmirror source where our other filers snapmirror all their data onto.
Problem:
on filer A we have 3 sepearte flexvols with 1 qtree in each flex vol. Each qtree has around 160gb and 4 million files in it.
eg.
/vol/vola/vola /vol/volb/volb /vol/volc/volc
on filer B i created a 600gb flex vol called filerA to snapmirror into
We were using Qtree snapmirrors onto filer B so the results looked like this
qtree: This command is deprecated; using qtree status. Volume Tree Style Oplocks Status
vol0 unix enabled normal filerA unix enabled normal filerA vola unix enabled snapmirrored filerA volb unix enabled snapmirrored filerA volc unix enabled snapmirrored
filerB> df /vol/filerA/ 838860800 488597596 350263204 58% /vol/filerA/ /vol/filerA/.snapshot 209715200 14934800 194780400 7% /vol/filerA/.snapshot
filerB> df -i
Filesystem iused ifree %iused Mounted on /vol/filerA/ 17202153 14674536 54% /vol/filerA/
filerB> snapmirror status Source Destination State Lag Status filerA:/vol/vola/vola filerB:/vol/filerA/vola Snapmirrored 152:02:24 Idle
When we backup /vol/filerA we are getting around 20-26meg a second to tape.
After a few weeks we see that the snapmirrors are taking a very long time (2-3 hours) to sync as each time it snapmirrors its doing a file copy. Ie works out what inodes have changed then copies data over.
So we decide to cutover to Volume based snapmirrors as its heaps faster.
now on filerB i create 3 restricted volumes and we are snapmirroring to each volume to their restricted volume. This is very very fast as its block level and its syncing in under 15 min
this is now how looks like the following on filerB
/vol/filerAvola/ 251658240 153210208 98448032 61% /vol/filerAvola/ /vol/filerAvola/.snapshot 62914560 14631824 48282736 23% /vol/filerAvola/.snapshot /vol/filerAvolb/ 251658240 154023020 97635220 61% /vol/filerAvolb/ /vol/filerAvolb/.snapshot 62914560 14196416 48718144 23% /vol/filerAvolb/.snapshot /vol/filerAvolc/ 251658240 154578208 97080032 61% /vol/filerAvolc/ /vol/filerAvolc/.snapshot 62914560 14897572 48016988 24% /vol/filerAvolc/.snapshot
a vol status looks like the following
filerAvola online raid_dp, flex snapmirrored=on, create_ucode=on, snapmirrored maxdirsize=20971, read-only fs_size_fixed=on, guarantee=volume(disabled)
we have 3 of these snapmirrored volumes..
filerB:vola filerA:vola Snapmirrored 04:20:26 Idle filerB:volb filerA:volb Snapmirrored 04:20:26 Idle filerB:volc filerA:volc Snapmirrored 04:20:26 Idle
now the problem is when we go to back these upto tape we are only getting 3
- 7 meg a second to tape.
This is the problem.
How come when we backup the large volume (/vol/filerA/) we get 26 meg a sec but when we backup the smaller restricted volumes (/vol/filerAvola/) we are only getting 3-7 ?
we are using Netbackup NDMP to a local tape drive
we have another restricted volume on filerB which is 140Gb of oracle data and that gets 30-50meg a second to tape.
-- Greg Wilson Senior System Administrator greg.wilson@aapt.com.au
Check the vol options for "minra" in that volume. ideally it should be "off" for backup purposes. You need to change the vol options of the source volume to affect the destination, since it is volme level mirrored.
Btw. "minra" vol option has a direct impact to the volume's I/O performance.
Brian.
Greg Wilson <gwilson@connect. To: toasters@mathworks.com com.au> cc: Sent by: Subject: Snapmirror and backup weirdness owner-toasters@ma thworks.com
08/02/2006 12:05 PM
Hello
Im having some really weird problems with backing up our snapmirrors and wondering if anyone has any ideas.
Here is the first setup..
We have a 3050 filer (filerB) with 16TB split up in 2 aggrs
filer B is used as a snapmirror source where our other filers snapmirror all their data onto.
Problem:
on filer A we have 3 sepearte flexvols with 1 qtree in each flex vol. Each qtree has around 160gb and 4 million files in it.
eg.
/vol/vola/vola /vol/volb/volb /vol/volc/volc
on filer B i created a 600gb flex vol called filerA to snapmirror into
We were using Qtree snapmirrors onto filer B so the results looked like this
qtree: This command is deprecated; using qtree status. Volume Tree Style Oplocks Status -------- -------- ----- -------- --------- vol0 unix enabled normal filerA unix enabled normal filerA vola unix enabled snapmirrored filerA volb unix enabled snapmirrored filerA volc unix enabled snapmirrored
filerB> df /vol/filerA/ 838860800 488597596 350263204 58% /vol/filerA/ /vol/filerA/.snapshot 209715200 14934800 194780400 7% /vol/filerA/.snapshot
filerB> df -i
Filesystem iused ifree %iused Mounted on /vol/filerA/ 17202153 14674536 54% /vol/filerA/
filerB> snapmirror status Source Destination State Lag Status filerA:/vol/vola/vola filerB:/vol/filerA/vola Snapmirrored 152:02:24 Idle
When we backup /vol/filerA we are getting around 20-26meg a second to tape.
After a few weeks we see that the snapmirrors are taking a very long time (2-3 hours) to sync as each time it snapmirrors its doing a file copy. Ie works out what inodes have changed then copies data over.
So we decide to cutover to Volume based snapmirrors as its heaps faster.
now on filerB i create 3 restricted volumes and we are snapmirroring to each volume to their restricted volume. This is very very fast as its block level and its syncing in under 15 min
this is now how looks like the following on filerB
/vol/filerAvola/ 251658240 153210208 98448032 61% /vol/filerAvola/ /vol/filerAvola/.snapshot 62914560 14631824 48282736 23% /vol/filerAvola/.snapshot /vol/filerAvolb/ 251658240 154023020 97635220 61% /vol/filerAvolb/ /vol/filerAvolb/.snapshot 62914560 14196416 48718144 23% /vol/filerAvolb/.snapshot /vol/filerAvolc/ 251658240 154578208 97080032 61% /vol/filerAvolc/ /vol/filerAvolc/.snapshot 62914560 14897572 48016988 24% /vol/filerAvolc/.snapshot
a vol status looks like the following
filerAvola online raid_dp, flex snapmirrored=on, create_ucode=on, snapmirrored maxdirsize=20971, read-only fs_size_fixed=on, guarantee=volume(disabled)
we have 3 of these snapmirrored volumes..
filerB:vola filerA:vola Snapmirrored 04:20:26 Idle filerB:volb filerA:volb Snapmirrored 04:20:26 Idle filerB:volc filerA:volc Snapmirrored 04:20:26 Idle
now the problem is when we go to back these upto tape we are only getting 3 - 7 meg a second to tape.
This is the problem.
How come when we backup the large volume (/vol/filerA/) we get 26 meg a sec
but when we backup the smaller restricted volumes (/vol/filerAvola/) we are only getting 3-7 ?
we are using Netbackup NDMP to a local tape drive
we have another restricted volume on filerB which is 140Gb of oracle data and that gets 30-50meg a second to tape.
-- Greg Wilson Senior System Administrator greg.wilson@aapt.com.au
This mail has been scanned by InterScan-MSS/kbsl
This Mail Has Been Scanned For Virus By Scanmail For Lotus Notes Enterprise/Kbsl
-
Borzenkov, Andrey (FSC) -
Brian Pascal -
Greg Wilson