Hello Toasters,
Anybody have any issues with seemingly random ESXi 5.5 NFS datastore disconnects during heavy load?
Our Environment:
ESXi 5.5 F3240 ONTAP 8.1.2P4
It doesn't happen all the time. Only during heavy load but even then there is no guarantee that it will happen. We have yet to find a consistent trigger.
Datastores are mounted via shortname. We are planning to mount via IP address to rule out any name resolution issues but that will take some time. DNS is generally solid so we are doubtful DNS has anything to do with it but we should align ourselves with best practices.
We serve all of our NFS through vfilers. Some of our vfilers host 5 NFS datastores from a single IP address. I mention this because I have come across documentation recommending a 1:1 ratio of datastores to IP addresses.
vmkernel.log just shows that the connection was lost to the NFS server. It recovers w/in 10 seconds. We have 11 nodes in this particular ESX cluster.
Not all 11 ESXi nodes lose connectivity to the datastore at the same time. I've seen it affect just one ESXi node's connectivity to a single datastore. I've also seen it affect more than one ESXi node and multiple datastores on the same filer.
Until recently, it was only observed during storage vmotions. We recently discovered it happening during vmotion activity managed by DRS after a node was brought out of maintenance mode. As I said before, it is generally a rare occurrence so it is difficult to trigger on our own.
Thanks in advance for any insight/experiences.
Phil