Hello *,
we are in the process of consolidating several NetApp filers and NearStore systems onto a single FAS30x0HA system. The system's main function is to serve primary data (part from FCAL disks, part from SATA disks), it's secondary function is to act as the NearStore system for a second site, storing the copy of the other site's data on SATA disks.
Now we would like to try A-SIS to reduce the required physical storage.
It looks as if in order to add A-SIS (a_sis license), I also need to install the NearStore license. Now my concern is, that by giving the system a NearStore personality I might impact the system's primary role - serving primary storage data to users.
Has anyone experience with how a system changes when the NearStore licenses is enabled? Any negative impact for serving data to users?
Is the NearStore personality activated by installing the nearstore license, or could we install the nearstore and a_sis license, but keep the nearstore personality deactivated?
For data that we volume SnapMirror, would we need an A-SIS license on both sides - source and target system?
Will the volume SnapMirror transfer less data (assuming that A-SIS de-dupe of data saved some of the required data blocks)? yes?!)
Any other experience (good or bad) or concerns when using A-SIS?
Any reason to not use it on all (home directory type data) volumes?
Any limitation encountered?
Any experience how much savings can be achieved (e.g. for home directories volumes, volume storing tons of application software packages)?
Cheers --pwo
-- Peter W. Osel eMail: pwo@Qimonda.COM Principal Development Systems Phone: +1.919.677.6333 Qimonda North America Corp. Cell : +1.408.348.6735 3000 CentreGreen Way eFax : +1.804.952.3905 Cary, NC 27513-5760, USA www : http://pwo.de/
Hi Peter,
You need the nearstore personality license AND the asis licence.
As I have understand about the nearstore personality: there is no impact in the performance behaviour of your FAS system. But the other parameter that also changes at the same time is the amount of simultaneous snapmirror/snapvault connections. When you do a lot of snapvaults or QSM's this can cause at the end some performance concerns. But you can solve those very easy to play with the schedules or with flexshare.
For snapmirror, I think that you can do it with only asis on the source. But of course, when you want to activate the snapmirrored volume, and you want still have the benefit of asis for new data, you need also asis on the destination. I have no experience with this scenario.
ASIS works great (in my case) for dumps to disks, or archive of our invoices, email archive, office data. Today, I only use it on my R200's (so I don't have the problem with the nearstore personality).
The savings varies between 5 and 80%. The only problem is that asis is a batch process. So for dumps, you must play with the schedules (dumping and asis) to have a great result.
I haven't tried the home dirs, but I think this must be perfect for asis.
Greetings,
Reinoud
it-manager infrastructure & operations
UZ Leuven
Belgium
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Peter W. Osel Sent: vrijdag 27 juli 2007 23:32 To: toasters@mathworks.com Subject: A-SIS (NetApp De-Duplication)
Hello *,
we are in the process of consolidating several NetApp filers and
NearStore systems onto a single FAS30x0HA system. The system's main
function is to serve primary data (part from FCAL disks, part from SATA
disks), it's secondary function is to act as the NearStore system for a
second site, storing the copy of the other site's data on SATA disks.
Now we would like to try A-SIS to reduce the required physical
storage.
It looks as if in order to add A-SIS (a_sis license), I also need to
install the NearStore license. Now my concern is, that by giving the
system a NearStore personality I might impact the system's primary role
- serving primary storage data to users.
Has anyone experience with how a system changes when the NearStore
licenses is enabled? Any negative impact for serving data to users?
Is the NearStore personality activated by installing the nearstore
license, or could we install the nearstore and a_sis license, but keep
the nearstore personality deactivated?
For data that we volume SnapMirror, would we need an A-SIS license on
both sides - source and target system?
Will the volume SnapMirror transfer less data (assuming that A-SIS
de-dupe of data saved some of the required data blocks)?
yes?!)
Any other experience (good or bad) or concerns when using A-SIS?
Any reason to not use it on all (home directory type data) volumes?
Any limitation encountered?
Any experience how much savings can be achieved (e.g. for home
directories volumes, volume storing tons of application software
packages)?
Cheers
--pwo
--
Peter W. Osel eMail: pwo@Qimonda.COM
Principal Development Systems Phone: +1.919.677.6333
Qimonda North America Corp. Cell : +1.408.348.6735
3000 CentreGreen Way eFax : +1.804.952.3905
Cary, NC 27513-5760, USA www : http://pwo.de/
Peter,
Hello and how are you my friend?
The enabling of the Nearstore personality does not reduce the performance of a system as long as your maintain the number of replication streams (SnapMirror / SnapVault) to the non Nearstore limits for that platform. You can reference NOW for the limits.
In order to replicate de-duplicated data one will require A-SIS licenses on both the sopurce and destination FAS systems. VSM replication is my personal preference.
Vaughn
Peter W. Osel wrote:
Hello *,
we are in the process of consolidating several NetApp filers and NearStore systems onto a single FAS30x0HA system. The system's main function is to serve primary data (part from FCAL disks, part from SATA disks), it's secondary function is to act as the NearStore system for a second site, storing the copy of the other site's data on SATA disks.
Now we would like to try A-SIS to reduce the required physical storage.
It looks as if in order to add A-SIS (a_sis license), I also need to install the NearStore license. Now my concern is, that by giving the system a NearStore personality I might impact the system's primary role
- serving primary storage data to users.
Has anyone experience with how a system changes when the NearStore licenses is enabled? Any negative impact for serving data to users?
Is the NearStore personality activated by installing the nearstore license, or could we install the nearstore and a_sis license, but keep the nearstore personality deactivated?
For data that we volume SnapMirror, would we need an A-SIS license on both sides - source and target system?
Will the volume SnapMirror transfer less data (assuming that A-SIS de-dupe of data saved some of the required data blocks)? yes?!)
Any other experience (good or bad) or concerns when using A-SIS?
Any reason to not use it on all (home directory type data) volumes?
Any limitation encountered?
Any experience how much savings can be achieved (e.g. for home directories volumes, volume storing tons of application software packages)?
Cheers --pwo
-- Peter W. Osel eMail: pwo@Qimonda.COM Principal Development Systems Phone: +1.919.677.6333 Qimonda North America Corp. Cell : +1.408.348.6735 3000 CentreGreen Way eFax : +1.804.952.3905 Cary, NC 27513-5760, USA www : http://pwo.de/
Hello,
here my summary on A-SIS, based on all the responses I received.
* at this point in time you need both, the nearstore license as well as the a_sis license.
* installing the nearstore license enables the nearstore personality, biggest impact is that the number of allowed parallel replication streams is increased (e.g. for an FAS30xx from 8 to 64).
* performance impact is negligible as long as you do not use the additional data streams (some scheduling magic might be necessary).
* while A-SIS seems to be fairly low overhead, a maxed out filer is probably not the best system to use for A-SIS ;-)
* for data volumes with mostly read access, A-SIS will add less overhead than for data volumes where data constantly changes.
* when using Volume SnapMirror (VSM) to replicate a de-duped volume, the update traffic does benefit from the de-duplication (you might want to coordinate when A-SIS de-dupes the volume and when SnapMirror transfers it ;-). It is recommended to have a-sis licenses on the source and the destination filer, e.g. in case you have to reverse the SnapMirror relationship.
* you can also qtree SnapMirror form a non-de-duped volume and just de-duplicate the target volume (obviously source volume and transfer volume will not be reduced in this case)
* The Data ONTAP 7.2 Data Protection Online Backup and Recovery Guide has a chapter on A-SIS. http://now.netapp.com/NOW/knowledge/docs/ontap/rel722/pdfs/ontap/onlinebk.pdf (now account needed)
* There is an excellent A-SIS De-duplication FAQ, and a Technical Report (TR-3505) in the making (not yet available on the public library). Ask your NetApp SE, they can also help with estimating the savings that A-SIS would achieve on your data volumes.
Work with your NetApp SE, they have access to more information and tools that really helps to plan your A-SIS deployment. I will definitely give A-SIS a try, and can hopefully share some experiences later this year.
A big thank you to all who responded: Bill Holland, Larry Freeman, M. Vaughn Stewart, Michael Barrow, Reinoud Reynders, and Sels Roger.
Cheers --pwo
For reference, here my original posting:
*** Peter W. Osel pwo@qimonda.com *** [2007-07-27 18:00:52]: Hello *,
we are in the process of consolidating several NetApp filers and NearStore systems onto a single FAS30x0HA system. The system's main function is to serve primary data (part from FCAL disks, part from SATA disks), it's secondary function is to act as the NearStore system for a second site, storing the copy of the other site's data on SATA disks.
Now we would like to try A-SIS to reduce the required physical storage.
It looks as if in order to add A-SIS (a_sis license), I also need to install the NearStore license. Now my concern is, that by giving the system a NearStore personality I might impact the system's primary role
- serving primary storage data to users.
Has anyone experience with how a system changes when the NearStore licenses is enabled? Any negative impact for serving data to users?
number of allowed parallel data streams increases. impact negligible as long as you limit the number of parallel data streams.
Is the NearStore personality activated by installing the nearstore license, or could we install the nearstore and a_sis license, but keep the nearstore personality deactivated?
yes, installing the nearstore license activates the nearstore personality, biggest change: number of parallel data streams is increased.
For data that we volume SnapMirror, would we need an A-SIS license on both sides - source and target system?
strictly needed only for source, recommended is source and target, e.g to be prepared when you want to revert the direction of the SnapMirror.
Will the volume SnapMirror transfer less data (assuming that A-SIS de-dupe of data saved some of the required data blocks)? yes?!)
definitely yes.
Any other experience (good or bad) or concerns when using A-SIS?
Any reason to not use it on all (home directory type data) volumes?
A-SIS makes more sense for volumes where not all data changes all the time, so if you have a volume that is very write intensive, with all data changing all the time, it might be a bad candidate for any de-duplication technology.
Any limitation encountered?
right now (Data ONTAP 7.2.x) there are limits on how big a volume can be, depending on the platform. Ask your SE for current limitations. They might change or be completely lifted in a future version of ONTAP.
Any experience how much savings can be achieved (e.g. for home directories volumes, volume storing tons of application software packages)?
Heavily depends on the data, 20% to 85% have been seen for regular data volumes, even more for highly redundant volumes (e.g. when storing weekly full backup images you might - over time - see 20:1 reduction). Your mileage may vary! Try it out yourself.
Cheers --pwo
-- Peter W. Osel eMail: pwo@Qimonda.COM Principal Development Systems Phone: +1.919.677.6333 Qimonda North America Corp. Cell : +1.408.348.6735 3000 CentreGreen Way eFax : +1.804.952.3905 Cary, NC 27513-5760, USA www : http://pwo.de/
One place where we hope to find some value from deduplication on our fairly loaded production filers, is we're constantly fighting with the amount of space our build engineers need for product builds. They're doing several snapshot builds a day, nightly builds, milestone builds, and they "need" to keep many of those online for weeks. Our hope is all of these nearly-identical build trees can be deduplicated to reduce disk space. I'm working with our SE to get the tool for estimating savings first, we'll see what that says.
If it works for that purpose too we might get a little more leverage for buying netapps for remote offices, since VSM can take advantage of A-SIS to reduce replication bandwidth.
Just some devils advocate "full disclosure comments"
1. ASIS is only at the volume level, and you can't hit a volume over 2TB (sometimes smaller depending on how many files you have) 2. When spinning to tape your data balloons up to its original non de-duped size. 3. You cannot ASIS a snap vault.
I really am excited about this technology. I think this could be a killer app for NetApp. Best of all right now it's free with your nearstore personality :)
Who knows how long that will last.
IF/When the ASIS guys get to a aggregate level then ASIS is going to be a must have in the data world, and I for one welcome our de-duplication overlords.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Sphar, Mike Sent: Wednesday, August 01, 2007 9:11 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
One place where we hope to find some value from deduplication on our fairly loaded production filers, is we're constantly fighting with the amount of space our build engineers need for product builds. They're doing several snapshot builds a day, nightly builds, milestone builds, and they "need" to keep many of those online for weeks. Our hope is all of these nearly-identical build trees can be deduplicated to reduce disk space. I'm working with our SE to get the tool for estimating savings first, we'll see what that says.
If it works for that purpose too we might get a little more leverage for buying netapps for remote offices, since VSM can take advantage of A-SIS to reduce replication bandwidth.
On your points 1 and 3, they are incorrect.
I'm running ASIS on snapvault destination volumes as large as 7TB. Several of the source volumes I snapshot have needed maxfiles increased, so the file count doesn't seem to be an issue.
The thing that is really lame about ASIS is that the freed blocks end up in the snapshot, and on a snapvault destination with 30+ days of snapshot, it takes a long time to actually save any space.
John
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Langdon, Laughlin T. (Lock) Sent: Wednesday, August 01, 2007 9:50 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
Just some devils advocate "full disclosure comments"
1. ASIS is only at the volume level, and you can't hit a volume over 2TB (sometimes smaller depending on how many files you have) 2. When spinning to tape your data balloons up to its original non de-duped size. 3. You cannot ASIS a snap vault.
I really am excited about this technology. I think this could be a killer app for NetApp. Best of all right now it's free with your nearstore personality :)
Who knows how long that will last.
IF/When the ASIS guys get to a aggregate level then ASIS is going to be a must have in the data world, and I for one welcome our de-duplication overlords.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Sphar, Mike Sent: Wednesday, August 01, 2007 9:11 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
One place where we hope to find some value from deduplication on our fairly loaded production filers, is we're constantly fighting with the amount of space our build engineers need for product builds. They're doing several snapshot builds a day, nightly builds, milestone builds, and they "need" to keep many of those online for weeks. Our hope is all of these nearly-identical build trees can be deduplicated to reduce disk space. I'm working with our SE to get the tool for estimating savings first, we'll see what that says.
If it works for that purpose too we might get a little more leverage for buying netapps for remote offices, since VSM can take advantage of A-SIS to reduce replication bandwidth.
Apologies if any of my info was incorrect. I'm just speaking from my experiences. I'm using a 3050.
The following table lists the maximum volume size for each storage system: Maximum A-SIS deduplication volume size allowed FAS6070* - 14.37 TB (to allow for WAFL reserve) FAS6030* - 10 TB FAS3070* - 6 TB FAS3050* - 2 TB FAS3040* - 3 TB FAS3020* - 1 TB R200 - 4 TB
I thought "officially" that snapvaults were not supported (they work, but like a lot of Netapp's stuff they haven't fully tested it, and are being conservative in saying it doesn't work)
Thanks for keepin me honest :)
-----Original Message----- From: John Clear [mailto:Johnn.Clear@amd.com] Sent: Wednesday, August 01, 2007 2:18 PM To: Langdon, Laughlin T. (Lock); toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
On your points 1 and 3, they are incorrect.
I'm running ASIS on snapvault destination volumes as large as 7TB. Several of the source volumes I snapshot have needed maxfiles increased, so the file count doesn't seem to be an issue.
The thing that is really lame about ASIS is that the freed blocks end up in the snapshot, and on a snapvault destination with 30+ days of snapshot, it takes a long time to actually save any space.
John
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Langdon, Laughlin T. (Lock) Sent: Wednesday, August 01, 2007 9:50 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
Just some devils advocate "full disclosure comments"
1. ASIS is only at the volume level, and you can't hit a volume over 2TB (sometimes smaller depending on how many files you have) 2. When spinning to tape your data balloons up to its original non de-duped size. 3. You cannot ASIS a snap vault.
I really am excited about this technology. I think this could be a killer app for NetApp. Best of all right now it's free with your nearstore personality :)
Who knows how long that will last.
IF/When the ASIS guys get to a aggregate level then ASIS is going to be a must have in the data world, and I for one welcome our de-duplication overlords.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Sphar, Mike Sent: Wednesday, August 01, 2007 9:11 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
One place where we hope to find some value from deduplication on our fairly loaded production filers, is we're constantly fighting with the amount of space our build engineers need for product builds. They're doing several snapshot builds a day, nightly builds, milestone builds, and they "need" to keep many of those online for weeks. Our hope is all of these nearly-identical build trees can be deduplicated to reduce disk space. I'm working with our SE to get the tool for estimating savings first, we'll see what that says.
If it works for that purpose too we might get a little more leverage for buying netapps for remote offices, since VSM can take advantage of A-SIS to reduce replication bandwidth.
Make sure you get the 64bit version of the ASIS evaluation tool, you can run the test against much bigger volumes, and get a better picture of your savings.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Sphar, Mike Sent: Wednesday, August 01, 2007 9:11 AM To: toasters@mathworks.com Subject: RE: SUMMARY: A-SIS (NetApp De-Duplication)
One place where we hope to find some value from deduplication on our fairly loaded production filers, is we're constantly fighting with the amount of space our build engineers need for product builds. They're doing several snapshot builds a day, nightly builds, milestone builds, and they "need" to keep many of those online for weeks. Our hope is all of these nearly-identical build trees can be deduplicated to reduce disk space. I'm working with our SE to get the tool for estimating savings first, we'll see what that says.
If it works for that purpose too we might get a little more leverage for buying netapps for remote offices, since VSM can take advantage of A-SIS to reduce replication bandwidth.
-
John Clear -
Langdon, Laughlin T. (Lock) -
M. Vaughn Stewart -
Peter W. Osel -
Reinoud Reynders -
Sphar, Mike