Hello,
I wonder what transferrate I could expect for snapmirror volumes (an/or qtrees) between 2 FAS270 directly connected with Gigabit Ethernet.
We use one filer as NFS backing store for our Mailsystem (Communigate Pro) with mdir mailbox format, so we have many small files here.
I see abnormal high lags with really moderate sizes of the transferred data.
For instance we need about 20 to 40 minutes to transfer 100 to 600 MBytes!
The filesystems a 50% filled 250 GByte Volume/qtree with 10.3 million files (with 7 mio free inodes).
so the average filesize is about 12KByte.
We use Ontap 7.0.4 here and will upgrade to 7.0.5 next week. But I don't really have much hope that this will change the problem.
Could anybody give me some data about his/her transfer rates? I really can't believe, that this slow transfers are normal.
An repeatedly snapmirror status command on the filer shows this:
at first the transfer startet and transferred about 68 MByte in 80 seconds. after that it paused for 15 minutes (output constantly "transferring 68MB done"). from minute 16 to 26 it says "transferring inodes". in these 11 minutes it transferred 225.000 inodes. after that it transfers data again. in minute 28 it has transferred 550 MBytes. after that one minute pause and then the status is idle.
Has netapp/snapmirror a big problem with small files? I don't want to switch to mailbox format because I like the advantages of mdir (single copy mailstore, easier backup?)
Thanks for any hints, thomas
Hello,
I wonder what transferrate I could expect for snapmirror volumes (an/or qtrees) between 2 FAS270 directly connected with Gigabit Ethernet.
Could you detail more about your mirroring setup ...
Volume or Qtree snapmirror?
What is the snapmirror schedule?
Synch or Async?
What sort of volume setup? (traditional, flex)?
Are the disks identical at both ends? What type and quantity of disks?
are you following the recommendations in the snapmirror best practices guide?
What sort of cpu utilization are you seeing on the source and destination?
I have a maildir setup with slightly more files than yours and about 100GB more data. I'm Volume snapmirroring over 100Mbps ethernet, and It takes ~10-15 Minutes to mirror the deltas, which are about 6-8 GB. I'm running 7.0.5. If memory serves me correctly, I upgraded because I saw some ugly Snapmirror bugs associated with 7.0.4 as a preventive measure.
Otherwise snapmirror works at the block level for transfers, file sizes shouldn't matter.
Regards, Max
We use one filer as NFS backing store for our Mailsystem (Communigate Pro) with mdir mailbox format, so we have many small files here.
I see abnormal high lags with really moderate sizes of the transferred data.
For instance we need about 20 to 40 minutes to transfer 100 to 600 MBytes!
The filesystems a 50% filled 250 GByte Volume/qtree with 10.3 million files (with 7 mio free inodes).
so the average filesize is about 12KByte.
We use Ontap 7.0.4 here and will upgrade to 7.0.5 next week. But I don't really have much hope that this will change the problem.
Could anybody give me some data about his/her transfer rates? I really can't believe, that this slow transfers are normal.
An repeatedly snapmirror status command on the filer shows this:
at first the transfer startet and transferred about 68 MByte in 80 seconds. after that it paused for 15 minutes (output constantly "transferring 68MB done"). from minute 16 to 26 it says "transferring inodes". in these 11 minutes it transferred 225.000 inodes. after that it transfers data again. in minute 28 it has transferred 550 MBytes. after that one minute pause and then the status is idle.
Has netapp/snapmirror a big problem with small files? I don't want to switch to mailbox format because I like the advantages of mdir (single copy mailstore, easier backup?)
Thanks for any hints, thomas
Hallo list, thanks for all responses.
On Fri, 29 Dec 2006, slinkymax0r wrote:
Could you detail more about your mirroring setup ...
Volume or Qtree snapmirror?
I started with volume and had similar problems with big delays. As I remember, the destination was doing constantly a "deswizzling". Our lokal support (not netapp) told me to migrate to qtrees about this deswizzling but this did not help.
What is the snapmirror schedule?
I did not use sync because I expected slow performance of NFS because the write will be accked after writing the blocks to both netapps. so I started with one minute:-) but had increased that quickly to 5 and the 15 minutes. now i use hourly scheduled snapmirror to better be able to find the reason of the problem.
Synch or Async?
asynch. is anybody using synch snapmirroring?
What sort of volume setup? (traditional, flex)?
flex
Are the disks identical at both ends? What type and quantity of disks?
both files are identical and purchsed at the same time on beginning of 2006. 14 300GB FCAL disks, DP, one spare
are you following the recommendations in the snapmirror best practices guide?
I don't know them, any link?
What sort of cpu utilization are you seeing on the source and destination?
I don't know what is normal. I see loads from 20 to 70 % on both filers. that is another problem for me. because ontap is constantly doing some maintenance on the filesystem (which souns good) I don't see what the real load is.
I have a maildir setup with slightly more files than yours and about 100GB more data. I'm Volume snapmirroring over 100Mbps ethernet, and It takes ~10-15 Minutes to mirror the deltas, which are about 6-8 GB. I'm running
That sounds to me as you have a moderate snapmirror schedule. If you have 7 GB deltas for about 250 GB I assume that you are doing the snapmirror only every some hours??? or you have very busy users:-)
I am (was) so naive to expect that I could be able to let the second filer follow the first with one or two minutes delay. It should not be a great problem to transfer the writes to the second filer and doing the same operations there on the fly. but now I think, that netapp ist doing snapshots for snapmirror and then calculates the changes and then transfers them to the destination. this is a lot of work to do for the small poor filer and it will not scale well with much small files. But perhaps I completly misunderstand the whole process.
7.0.5. If memory serves me correctly, I upgraded because I saw some ugly Snapmirror bugs associated with 7.0.4 as a preventive measure.
That gives me some hope about the future when we upgrade to 7.0.5.
Otherwise snapmirror works at the block level for transfers, file sizes shouldn't matter.
I don't think, that the transfer is the problem. It looks like some overhead in the filesystem for me.
Thanks for all responses yet. Everybody on the list I wish a fine 2007, thomas
Volume or Qtree snapmirror?
I started with volume and had similar problems with big delays. As I remember, the destination was doing constantly a "deswizzling". Our lokal support (not netapp) told me to migrate to qtrees about this deswizzling but this did not help.
Occassionally there is some noticable activity on the destination and the source node after a transfer, but it should cause the delays you are experiencing. Deswizzling is normal on destination volumes.
deswizzling should only be an issue if your accessing data on the destination volume, and seeing as you switched to qtree snapmirror and noticed no change, this might not be the problem.
What is the snapmirror schedule?
I did not use sync because I expected slow performance of NFS because the write will be accked after writing the blocks to both netapps. so I started with one minute:-) but had increased that quickly to 5 and the 15 minutes. now i use hourly scheduled snapmirror to better be able to find the reason of the problem.
Hourly seems reasonable.
Synch or Async?
asynch. is anybody using synch snapmirroring?
I've used it. Typically, if you're considering using a really small window, (say snap every five minutes or everyone one minute) then I'd recommend changing to sync if the filers are directly connected. Not applicable in every case, but certainly an option.
Are the disks identical at both ends? What type and quantity of disks?
both files are identical and purchsed at the same time on beginning of 2006. 14 300GB FCAL disks, DP, one spare
Excellent!
I don't know them, any link?
http://www.netapp.com/library/tr/3446.pdf
What sort of cpu utilization are you seeing on the source and destination?
I don't know what is normal. I see loads from 20 to 70 % on both filers. that is another problem for me. because ontap is constantly doing some maintenance on the filesystem (which souns good) I don't see what the real load is.
Unless it's constantly pegged at 100%, you're usually ok. Granted, we haven't dived into a statit session, but nothing sounds out of the ordinary.
My suggestion at this point would be to double check the links between the two filers to make sure you aren't dealing with a speed/duplex mismatch or other issue. (Run netdiag -v).
Otherwise, you've probably encountered a bug. (Doing a quick now search for snapmirror bugs is illuminating. There are a fair number of #2 slowness bugs related to snapmirror) I suggest upgrading to 7.0.5 on both the source and destination.
Regards, Max
I have a maildir setup with slightly more files than yours and about 100GB more data. I'm Volume snapmirroring over 100Mbps ethernet, and It takes ~10-15 Minutes to mirror the deltas, which are about 6-8 GB. I'm running
That sounds to me as you have a moderate snapmirror schedule. If you have 7 GB deltas for about 250 GB I assume that you are doing the snapmirror only every some hours??? or you have very busy users:-)
I am (was) so naive to expect that I could be able to let the second filer follow the first with one or two minutes delay. It should not be a great problem to transfer the writes to the second filer and doing the same operations there on the fly. but now I think, that netapp ist doing snapshots for snapmirror and then calculates the changes and then transfers them to the destination. this is a lot of work to do for the small poor filer and it will not scale well with much small files. But perhaps I completly misunderstand the whole process.
7.0.5. If memory serves me correctly, I upgraded because I saw some ugly Snapmirror bugs associated with 7.0.4 as a preventive measure.
That gives me some hope about the future when we upgrade to 7.0.5.
Otherwise snapmirror works at the block level for transfers, file sizes shouldn't matter.
I don't think, that the transfer is the problem. It looks like some overhead in the filesystem for me.
Thanks for all responses yet. Everybody on the list I wish a fine 2007, thomas
Are you using flexvol? Are you using qtree snapmirror or volume snapmirror? You also need to find out what is the CPU utilization of the filers. Qtree snapmirror is logical and takes up more CPU to start and if the filers are busy, which I think for a mail application it could potentially be, then you might be having problem due to the systems being overloaded.
I have not done much snapmirror on FAS270 but on a FAS940 cluster I was routinely getting ~100 GB per hour for qtree snapmirror. Not sure if that helps.
Derek
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Thomas Bleek Sent: Friday, December 29, 2006 6:19 AM To: Toaster Admins Subject: Snapmirror transferrates??
Hello,
I wonder what transferrate I could expect for snapmirror volumes (an/or qtrees) between 2 FAS270 directly connected with Gigabit Ethernet.
We use one filer as NFS backing store for our Mailsystem (Communigate Pro) with mdir mailbox format, so we have many small files here.
I see abnormal high lags with really moderate sizes of the transferred data.
For instance we need about 20 to 40 minutes to transfer 100 to 600 MBytes!
The filesystems a 50% filled 250 GByte Volume/qtree with 10.3 million files (with 7 mio free inodes).
so the average filesize is about 12KByte.
We use Ontap 7.0.4 here and will upgrade to 7.0.5 next week. But I don't really have much hope that this will change the problem.
Could anybody give me some data about his/her transfer rates? I really can't believe, that this slow transfers are normal.
An repeatedly snapmirror status command on the filer shows this:
at first the transfer startet and transferred about 68 MByte in 80 seconds. after that it paused for 15 minutes (output constantly "transferring 68MB done"). from minute 16 to 26 it says "transferring inodes". in these 11 minutes it transferred 225.000 inodes. after that it transfers data again. in minute 28 it has transferred 550 MBytes. after that one minute pause and then the status is idle.
Has netapp/snapmirror a big problem with small files? I don't want to switch to mailbox format because I like the advantages of mdir (single copy mailstore, easier backup?)
Thanks for any hints, thomas
Hello List,
I'm happy to be able to say that the problem seems to be solved.
The upgrade from 7.0.4 to 7.0.5 did not improve the performance of Qtree Snapmirroring (about the same 30 minutes). But today after converting to VSM I was really impressed.
The initial sync of 170 GB (incl. snapshots) took about 170 minutes which is MUCH better than before. I remember about 16 hours or even more.
After that the differential transfers took took about 1-3 minutes for a schedule of 10 minutes.
I switched to synchrous for test and DID NOT see the expected performance disaster. Our mailserver did not notice any difference. Of course I will do some measurements in future but for now I left it in the sync. state. Perhaps I will have to switch to semi-sync or even back to async but the problem itself seems to have gone away!!!
Thanks for all helpful hints, thomas
On Fri, 29 Dec 2006, Thomas Bleek wrote:
Hello,
I wonder what transferrate I could expect for snapmirror volumes (an/or qtrees) between 2 FAS270 directly connected with Gigabit Ethernet.
We use one filer as NFS backing store for our Mailsystem (Communigate Pro) with mdir mailbox format, so we have many small files here.
I see abnormal high lags with really moderate sizes of the transferred data.
For instance we need about 20 to 40 minutes to transfer 100 to 600 MBytes!
The filesystems a 50% filled 250 GByte Volume/qtree with 10.3 million files (with 7 mio free inodes).
so the average filesize is about 12KByte.
We use Ontap 7.0.4 here and will upgrade to 7.0.5 next week. But I don't really have much hope that this will change the problem.
Could anybody give me some data about his/her transfer rates? I really can't believe, that this slow transfers are normal.
An repeatedly snapmirror status command on the filer shows this:
at first the transfer startet and transferred about 68 MByte in 80 seconds. after that it paused for 15 minutes (output constantly "transferring 68MB done"). from minute 16 to 26 it says "transferring inodes". in these 11 minutes it transferred 225.000 inodes. after that it transfers data again. in minute 28 it has transferred 550 MBytes. after that one minute pause and then the status is idle.
Has netapp/snapmirror a big problem with small files? I don't want to switch to mailbox format because I like the advantages of mdir (single copy mailstore, easier backup?)
Thanks for any hints, thomas
Dr. Thomas Bleek, Netzwerkadministrator GeoForschungsZentrum Potsdam,Rechenzentrum,Telegrafenberg G261,D-14473 Potsdam Tel.: +49 331 288- 1818/1681 Fax.: 1730 Mobil: +49 172 1543233 E-Mail: bl@gfz-potsdam.de
Great news! Lesson from all this, use a block based replication for a data set with loads and loads of small files.
-Blake
On 1/9/07, Thomas Bleek bl@gfz-potsdam.de wrote:
Hello List,
I'm happy to be able to say that the problem seems to be solved.
The upgrade from 7.0.4 to 7.0.5 did not improve the performance of Qtree Snapmirroring (about the same 30 minutes). But today after converting to VSM I was really impressed.
The initial sync of 170 GB (incl. snapshots) took about 170 minutes which is MUCH better than before. I remember about 16 hours or even more.
After that the differential transfers took took about 1-3 minutes for a schedule of 10 minutes.
I switched to synchrous for test and DID NOT see the expected performance disaster. Our mailserver did not notice any difference. Of course I will do some measurements in future but for now I left it in the sync. state. Perhaps I will have to switch to semi-sync or even back to async but the problem itself seems to have gone away!!!
Thanks for all helpful hints, thomas
On Fri, 29 Dec 2006, Thomas Bleek wrote:
Hello,
I wonder what transferrate I could expect for snapmirror volumes (an/or qtrees) between 2 FAS270 directly connected with Gigabit Ethernet.
We use one filer as NFS backing store for our Mailsystem (Communigate Pro) with mdir mailbox format, so we have many small files here.
I see abnormal high lags with really moderate sizes of the transferred data.
For instance we need about 20 to 40 minutes to transfer 100 to 600 MBytes!
The filesystems a 50% filled 250 GByte Volume/qtree with 10.3 million files (with 7 mio free inodes).
so the average filesize is about 12KByte.
We use Ontap 7.0.4 here and will upgrade to 7.0.5 next week. But I don't really have much hope that this will change the problem.
Could anybody give me some data about his/her transfer rates? I really can't believe, that this slow transfers are normal.
An repeatedly snapmirror status command on the filer shows this:
at first the transfer startet and transferred about 68 MByte in 80 seconds. after that it paused for 15 minutes (output constantly "transferring 68MB done"). from minute 16 to 26 it says "transferring inodes". in these 11 minutes it transferred 225.000 inodes. after that it transfers data again. in minute 28 it has transferred 550 MBytes. after that one minute pause and then the status is idle.
Has netapp/snapmirror a big problem with small files? I don't want to switch to mailbox format because I like the advantages of mdir (single copy mailstore, easier backup?)
Thanks for any hints, thomas
Dr. Thomas Bleek, Netzwerkadministrator GeoForschungsZentrum Potsdam,Rechenzentrum,Telegrafenberg G261,D-14473 Potsdam Tel.: +49 331 288- 1818/1681 Fax.: 1730 Mobil: +49 172 1543233 E-Mail: bl@gfz-potsdam.de
-
Blake Golliher -
Derek Lai -
Max Reid -
slinkymax0r -
Thomas Bleek