Your understanding / expectations of thin provisioning are a bit off. TP saves on Provisioned yet unused capacity. A deleted VMDK on VMFS is used capacity. If you have a high number of VMDKs which you recover from you shoudl consider RDMs or VMDKs on NFS and SnapRestore .

NFS eliminates a layer of virtualization and the overhead associated with it. NFS provides the highest storage utilization of the storage options. Comparable is TP's\d RDMs.

If storage utilization is your goal consider thee other storage options along with A-SIS.

V

drenthjd@wxs.nl wrote:

Forgot to mention, we have disabled all snapshotting on the NetApp, and space reservation configuration is as was mentioned.

DF output shows the 180 GB in use (while ESX reports 60 on the VFMS).

The TR talks about auto deletion and auto grow, which are fine and very helpful in general but do not adress this issue we are having.

It would seem while a static amound of VMDKs is in use, the strategy works for you. But when you delete and recreate a vmdk (either from scratch or a dumb copy) VMware will put this new file anywhere in the thin provisioned LUN - thus not reusing old vmdk space, thus taking up more space than strictly necessary (of course at the end of the empty void that is thin provisioned I suppose it would start using old removed spave, but that would render thin provisioning useless).

Kind regards, Delorean

---

*Van:* Kenneth Heal [mailto:kheal@hotmail.com] *Verzonden:* di 17-7-2007 23:40 *Aan:* drenthjd@wxs.nl; toasters@mathworks.com *Onderwerp:* RE: NetApp Thin Provisioning and VMware ESX

Hi

Don't know if you have seen http://www.netapp.com/library/tr/3428.pdf. This talks a bit more about ESX and Netapp together. Reading this description it sounds like there may be old snapshots and/or space reservations at play here. In the case of the former snapshot auto-deletion may prove to be a help.

Could you check this and grab us a df output?

I haven't tried this combo, so there may be someone out there who has already done this and could say more.

regards Kenneth Heal Storage Consultant Senet Eindhoven bv

------------------------------------------------------------------------
Subject: NetApp Thin Provisioning and VMware ESX
Date: Tue, 17 Jul 2007 22:54:03 +0200
From: drenthjd@wxs.nl
To: toasters@mathworks.com

Hi Toasters,

We created something of an interesting dilemma for ourselves.

The original problem was the lack of a means to extend existing
VMFS LUNs through ESX without an extent and or downtime.

The idea was to use NetApp thin provisioning as a way to offer up
a large disk to ESX without the need to back that up with real
disks, as a way to monitor real disk usage we created a hard
volume and monitor with df the real use of disk against a hard
limit (being the volume).

So a 1 TB Lun (thin) was created inside a 200 GB (hard) volume,
subsequently we wanted to keep track of ESX 3's usage through
controlling the size of the VMDK files we would create. As this
environment hosts desktops and was a testing ground at first there
were a whole bunch of VMDK being created and destroyed.

The situation we end up with is VMware thinking it has 60 GB in
use of a 1 TB LUN while NetApp reports 180 GB in use of 200 GB
(being the hard limit at this time).

Basically the question becomes whether we can force ESX to
actually reuse any of the [netapp in use] - [real use according to
ESX] = 130 GB of space.

Probably we need to clean up and start over with a new clean LUN,
which we could then either hard provision and forget about thin
provisioning altogether, because in our scenario it is no help. A
broader question becomes were we went wrong? Our situation seems
to suggest that we waste a whole lot of disk space - which VMware
wont reuse on its own, and which NetApp can't touch (how would it
know it was now freed up space) In that sense thin provisioning is
just a space waster in an environment with VMDK's being destroyed
and recreated from scratch.

Of course we could always go flexcloning... but that would force
netapp world on a whole lot of admins.

Best regards,
Delorean

---

Change is good. See what's different about Windows Live Hotmail. Check it out! http://www.windowslive-hotmail.com/learnmore/default.html?locale=en-us&ocid=RMT_TAGLM_HMWL_reten_changegood_0607

Check your DNS / name resolving settings. 90% of the time, that's where your problem lies :-)

On 7/19/07, Jack Lyons jack1729@gmail.com wrote:

I am new to using esx on nfs. I have create an nfs export but when I try to mount it from the esx box, it is really slow - it takes 2- 3 minutes to mount?

I assume that isn't normal, but I am not sure where to start looking for problems. I assume it is probably doing something and then waits until a timeout occurs before proceeding with the mount. I can nfs mount the same filer / export on another box no problem (i.e. quickly)

You might want to look a bit more closely at the network design of you ESX hosts, if you want to go down the NFS or ISCSI route. And from the explanation you have given of your networking, I would say it is most likely to be the cause of the problem

Where ever you use a VMkernel port for storage it is always a good idea to put a Service console port with it, for some reason it always seems to work better.

The best practise I got from VMware regards ISCSI should stand with NFS as well. We do however have 6 Nics in each of our ESX hosts.

Vswitch0 (Service console and VMkernel for Vmotion) with 2 Physical Nics, load balance using Virtual port ID

Vswitch1 (VM Network) with 2 Physical Nics, load balance using Virtual port ID

Vswitch2 (Service console and VMkernel for ISCS/NFS) with 2 Physical Nics, load balance using IP HASH

Now with 4 Nics becomes difficult, as I have yet to find any best practise information based on ISCSI or NFS, but based on some documents I have from the TSX event earlier this year I will make a stab.

Vswitch0 2 Physical Nics with Trunking Configured with 2 Vlans and with the following

(Service console and VMkernel for Vmotion (Seperate Vlan prefer Nic 1)) (Service console and VMkernel for Iscsi/NFS (separate Vlan, prefer Nic 2))

Vswitch1 (VM Network) with 2 Physical Nics, load balance using Virtual port ID

Hope this helps.

I will also send you the document I have directly. If anyone else would like it, let me know off list I will happily send it over

Cheers

Matt

-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Jack Lyons Sent: 20 July 2007 04:03 To: Nils Vogels Cc: toasters@mathworks.com Subject: Re: NetApp Thin Provisioning and VMware ESX

I can ping the filer from the esx host and vice versa, but I have 4 interfaces on my esx servers, which one needs to resolve

1 for vmotion 1 for service console <---- I believe this is the network that the NFS traffic will be using??? 1 for production network 1 for dmz network

which leads me to more questions 1) i will be needing to connect to separate test network, user acceptance network and possibly one more. in addition to the ones above. Any problems I need to worry about for having 6-7 network interfaces. 2) Right now, I have all VM's datastores on SAN via FC HBAs. If I max out a server (30-35 VM's per host), will I need more than 1 GigE for NFS

traffic if I move all VM datastores to NFS?

Thanks

Nils Vogels wrote:

Check your DNS / name resolving settings. 90% of the time, that's where your problem lies :-)

On 7/19/07, Jack Lyons jack1729@gmail.com wrote:

...

I am new to using esx on nfs. I have create an nfs export but when I try to mount it from the esx box, it is really slow - it takes 2- 3 minutes to mount?

I assume that isn't normal, but I am not sure where to start looking

for

...

problems. I assume it is probably doing something and then waits

until

...

a timeout occurs before proceeding with the mount. I can nfs mount

the

...

same filer / export on another box no problem (i.e. quickly)

_____________________________________________________________ This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com mailto:help@generalatlantic.com. Thank You.

Any LUN block that was touched at least once remains in use forever. NetApp has no knowledge whether this block is on free list of file system on top of LUN or belongs to some file inside. In your case VMware did touch 180GB during its lifetime. Of them 120GB is likely on free list (or similar) of vmfs, and are accounted as "free" by it; but they still occupy space on NetApp.

Unless you can force VMware to reuse old blocks in preference to allocating new once, I do not see what can be done. Using raw disks is probably an option as was already mentioned.

? ????????? / With best regards / Mit freundlichen Grü?en

--- Andrey Borzenkov Senior system engineer

________________________________ From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of drenthjd@wxs.nl Sent: Wednesday, July 18, 2007 1:54 AM To: Kenneth Heal; toasters@mathworks.com Subject: RE: NetApp Thin Provisioning and VMware ESX

Forgot to mention, we have disabled all snapshotting on the NetApp, and space reservation configuration is as was mentioned.

DF output shows the 180 GB in use (while ESX reports 60 on the VFMS).

The TR talks about auto deletion and auto grow, which are fine and very helpful in general but do not adress this issue we are having.

It would seem while a static amound of VMDKs is in use, the strategy works for you. But when you delete and recreate a vmdk (either from scratch or a dumb copy) VMware will put this new file anywhere in the thin provisioned LUN - thus not reusing old vmdk space, thus taking up more space than strictly necessary (of course at the end of the empty void that is thin provisioned I suppose it would start using old removed spave, but that would render thin provisioning useless).

Kind regards, Delorean

________________________________ Van: Kenneth Heal [mailto:kheal@hotmail.com] Verzonden: di 17-7-2007 23:40 Aan: drenthjd@wxs.nl; toasters@mathworks.com Onderwerp: RE: NetApp Thin Provisioning and VMware ESX Hi

Don't know if you have seen http://www.netapp.com/library/tr/3428.pdf. This talks a bit more about ESX and Netapp together. Reading this description it sounds like there may be old snapshots and/or space reservations at play here. In the case of the former snapshot auto-deletion may prove to be a help.

Could you check this and grab us a df output?

I haven't tried this combo, so there may be someone out there who has already done this and could say more.

regards Kenneth Heal Storage Consultant Senet Eindhoven bv ________________________________ Subject: NetApp Thin Provisioning and VMware ESX Date: Tue, 17 Jul 2007 22:54:03 +0200 From: drenthjd@wxs.nl To: toasters@mathworks.com Hi Toasters,

We created something of an interesting dilemma for ourselves.

The original problem was the lack of a means to extend existing VMFS LUNs through ESX without an extent and or downtime.

The idea was to use NetApp thin provisioning as a way to offer up a large disk to ESX without the need to back that up with real disks, as a way to monitor real disk usage we created a hard volume and monitor with df the real use of disk against a hard limit (being the volume).

So a 1 TB Lun (thin) was created inside a 200 GB (hard) volume, subsequently we wanted to keep track of ESX 3's usage through controlling the size of the VMDK files we would create. As this environment hosts desktops and was a testing ground at first there were a whole bunch of VMDK being created and destroyed.

The situation we end up with is VMware thinking it has 60 GB in use of a 1 TB LUN while NetApp reports 180 GB in use of 200 GB (being the hard limit at this time).

Basically the question becomes whether we can force ESX to actually reuse any of the [netapp in use] - [real use according to ESX] = 130 GB of space.

Probably we need to clean up and start over with a new clean LUN, which we could then either hard provision and forget about thin provisioning altogether, because in our scenario it is no help. A broader question becomes were we went wrong? Our situation seems to suggest that we waste a whole lot of disk space - which VMware wont reuse on its own, and which NetApp can't touch (how would it know it was now freed up space) In that sense thin provisioning is just a space waster in an environment with VMDK's being destroyed and recreated from scratch.

Of course we could always go flexcloning... but that would force netapp world on a whole lot of admins.

Best regards, Delorean

________________________________ Change is good. See what's different about Windows Live Hotmail. Check it out!http://www.windowslive-hotmail.com/learnmore/default.html?locale=en-us&ocid=RMT_TAGLM_HMWL_reten_changegood_0607