We have a scenario very similar to yours — NDMP over 10GbE with NetBackup. I was seeing backup speeds from our FAS2650 of several hundred MB/s in the beginning (first several months) but that has gradually slowed over time to about 35 MB/s. It is surely disappointing. It’s also disappointing that the Filer doesn’t have enough interfaces for us to attach the tape drives directly like we were doing with our old FAS3040 but if we could get back to the several hundred MB/s speeds that we had in the beginning then it would be an overall win.
From: Toasters toasters-bounces@teaparty.net on behalf of Heino Walther hw@beardmann.dk Date: Wednesday, June 2, 2021 at 08:46 To: "toasters@teaparty.net" toasters@teaparty.net Subject: [EXTERNAL] NDMP Backup speeds
Hi there
We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup. For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers. We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup. The LTO8 drives should be able to do much better (350MB/sec. before compression)… The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups… Could it be that the protocol itself it as cause here? In the “old days” we did NDMP backup directly from the controllers to the tape drives. We are now thinking of converting two 10G ports to fibrechannel ports… if possible… So I have two questions 😊
The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ? (I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)
Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?
Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)
/Heino
Heino, When the backups are running have you confirmed its going over the 10G interfaces (end to end -- do a packet trace)? I had a heck of a time getting it working without intercluster interfaces.
Get harvest/OCUM running and check if there are bottlenecks when the backups kick in. Also look up multi streams.
I had a similar issue to what you are describing -- it was defaulting to 1G mgmt, I had to configure a cluster wide 10G management interface and define it before it started working -- the support recommended node mgmt ints did not work.
Regards, Douglas
On Wed, Jun 2, 2021 at 10:26 AM Heino Walther hw@beardmann.dk wrote:
So maybe a volume reallocate would make it speed up?
One of our systems is quite new and has not been over 80% on its aggregates, so I would think that reallocate would not make much sense… also it is a snapvault target and no snapshots have been deleted yet, so data has just been added over time…
I would guess the issue can be that if the volume is dedupe enabled, the data will no doubt be “spread” around the disks somehow…. And these volumes have about 70% dedupe rates 😊
I have read a bit about snapmirror to tape, maybe that is the approach? Has anyone tried this?
I do not have much faith to reallocation… so we are looking into FC attached if it is possible via FCoE (which I doubt)?
Another approach would be to mount up the volumes (they are NFS) to a Linux hosts that has a NetBackup Client installed, and then backup the data from there…
We have so much data, that we would need to do a full, and then only do incrementals from there… (the volumes are vmware datastores)
(I know that other setups are possible, like FabricPool etc.. and that might be the way forward in the future)
/Heino
*Fra: *Rhorer, Kyle L. (JSC-OD)[THE BOEING COMPANY] < kyle.l.rhorer@nasa.gov> *Dato: *onsdag, 2. juni 2021 kl. 16.07 *Til: *Heino Walther hw@beardmann.dk, toasters@teaparty.net < toasters@teaparty.net> *Emne: *Re: [EXTERNAL] NDMP Backup speeds
We have a scenario very similar to yours — NDMP over 10GbE with NetBackup. I was seeing backup speeds from our FAS2650 of several hundred MB/s in the beginning (first several months) but that has gradually slowed over time to about 35 MB/s. It is surely disappointing. It’s also disappointing that the Filer doesn’t have enough interfaces for us to attach the tape drives directly like we were doing with our old FAS3040 but if we could get back to the several hundred MB/s speeds that we had in the beginning then it would be an overall win.
*From: *Toasters toasters-bounces@teaparty.net on behalf of Heino Walther hw@beardmann.dk *Date: *Wednesday, June 2, 2021 at 08:46 *To: *"toasters@teaparty.net" toasters@teaparty.net *Subject: *[EXTERNAL] NDMP Backup speeds
Hi there
We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup.
For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers.
We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup.
The LTO8 drives should be able to do much better (350MB/sec. before compression)…
The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups…
Could it be that the protocol itself it as cause here?
In the “old days” we did NDMP backup directly from the controllers to the tape drives.
We are now thinking of converting two 10G ports to fibrechannel ports… if possible…
So I have two questions 😊
The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ?
(I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)
Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?
Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)
/Heino _______________________________________________ Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
How about this for the win?
https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88...
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
On Wed, Jun 2, 2021 at 9:41 AM Heino Walther hw@beardmann.dk wrote:
Hi there
We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup.
For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers.
We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup.
The LTO8 drives should be able to do much better (350MB/sec. before compression)…
The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups…
Could it be that the protocol itself it as cause here?
In the “old days” we did NDMP backup directly from the controllers to the tape drives.
We are now thinking of converting two 10G ports to fibrechannel ports… if possible…
So I have two questions 😊
The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ?
(I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)
Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?
Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)
/Heino _______________________________________________ Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Another thing we’ve observed is that since NDMP has to enumerate the entire file system, backups of locations with lots of inodes can take a long time and drive high levels of CPU and memory utilization.
On Wed, Jun 2, 2021 at 3:45 PM tmac tmacmd@gmail.com wrote:
How about this for the win?
https://docs.netapp.com/ontap-9/topic/com.netapp.doc.dot-cm-ptbrg/GUID-6EC88...
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
On Wed, Jun 2, 2021 at 9:41 AM Heino Walther hw@beardmann.dk wrote:
Hi there
We are doing NDMP backups from out FAS2750 with NL-SAS disks onto LTO8 tape drives via NetBackup.
For other reasons we are doing this via 10G Ethernet, and not directly connected tapedrives to the NetApp controllers.
We are seeing backup speeds of about 40-50MB/sec. which is kinda slow, the backup is a Windows host that is doing nothing… no CPU or memory load as we backup.
The LTO8 drives should be able to do much better (350MB/sec. before compression)…
The aggregates are RAID-TEC with 17 drives in it. We are on ONTAP 9.7. We are not seeing a lot of load as we are running the backups…
Could it be that the protocol itself it as cause here?
In the “old days” we did NDMP backup directly from the controllers to the tape drives.
We are now thinking of converting two 10G ports to fibrechannel ports… if possible…
So I have two questions 😊
The FAS2750 is configured with RJ45 10G ports, but would it be possible to use FCoE and somehow connect this to a FC/Ethernet switch and be able to connect it to the tape drive which is 8G FC (LC Connectors) ?
(I have not worked that much with FCoE… if the FAS2750 was configured with CNA-Ports, it would have been easy to convert that into FC)
Has anyone tried to backup NL-SAS volumes similar to this over directly connected NDMP, and what kind of speeds are you seeing?
Finally if anyone have any idea if the NetBackup setup can be tweaked somehow, that would also be useful… (we have tweaked it a bit)
/Heino _______________________________________________ Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
Toasters mailing list Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters
-
Basil -
Douglas Siggins -
Heino Walther -
Rhorer, Kyle L. (JSC-OD)[THE BOEING COMPANY] -
tmac