Hi Everyone,

 

Over the weekend we had a snapvault backup fail with memory

allocation errors on the snapvault destination.  This error

occurred twice on the same volume, then hasn’t come back.

 

  Can  not allocate memory.

  (memory limit reached) (Cannot allocate memory)

 

I run snapmirror and snapvault on all our volumes,

but do it one operation at a time – NO concurrent

snapvault or snapmirror operations.

 

I search NOW and didn’t come up with anything.

 

This occurred on this particular volume twice in a row

(6 hours apart – backups run every 6hours), not on any

other volumes.

 

This volume is 410GB with 900k inodes – not even close

to our biggest volumes. So this error is not related to

size (capacity or num inodes), but seems specific to this volume.

 

  Filers:   IBM N3220 (NetApp FAS2240-2)

  OnTap:    8.1.2P1 7-Mode

 

 

Here are the full messages on the SV destination filer.

 

 

___ Sat  09 20 07:43:05 EDT [rshd_0:debug]: root@[172.17.170.34_60065]:IN:ssh2 shell:SSH INPUT COMMAND is snapvault snap create -w v_fnce08p_cifs v_fnce08p_cifs

tgt Sat  09 20 07:43:05 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_start

dst Sat  09 20 07:43:06 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Request (Update)

dst Sat  09 20 07:43:07 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Start

___ Sat  09 20 07:50:43 EDT [isnabk0665a:replication.dst.waflErr:error]: SnapVault: destination transfer from isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs to /vol/v_fnce08p_cifs/q_fnce08p_cifs : memory limit reached:Can  not allocate memory.

dst Sat  09 20 07:50:43 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Defer (memory limit reached) (Cannot allocate memory)

dst Sat  09 20 07:51:00 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Request (Retry)

dst Sat  09 20 07:51:01 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Restart (@ 929692 KB)

dst Sat  09 20 07:51:04 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Abort (memory limit reached) (Cannot allocate memory)

dst Sat  09 20 07:51:04 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Rollback_start

slk Sat  09 20 07:51:30 EDT state.qtree_softlock.v_fnce08p_cifs.0001b0a2.252.q_fnce08p_cifs.src-qt.isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs.00000000.-01.1 Softlock_add (Transfer)

dst Sat  09 20 07:51:37 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Rollback_end (42928 KB)

slk Sat  09 20 07:51:37 EDT state.qtree_softlock.v_fnce08p_cifs.0001b0a2.252.q_fnce08p_cifs.src-qt.isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs.00000000.-01.1 Softlock_delete (Transfer)

dst Sat  09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_end

dst Sat  09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_start (isnabk0665a(1896301030)_v_fnce08p_cifs-base.2)

dst Sat  09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_start (isnabk0665a(1896301030)_v_fnce08p_cifs-cleanup.0)

dst Sat  09 20 07:51:40 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_end

___ Sat  09 20 07:51:41 EDT [rshd_0:debug]: root@[172.17.170.34_60540]:IN:ssh2 shell:SSH INPUT COMMAND is snapvault status -l /vol/v_fnce08p_cifs/q_fnce08p_cifs

tgt Sat  09 20 07:51:41 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_create_snapshot (v_fnce08p_cifs.0)

tgt Sat  09 20 07:51:41 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_end

vol Sat  09 20 07:54:43 EDT v_fnce08p_cifs Snapshot_replace (v_fnce08p_cifs.0)

 

 

 

Thanks!

 

Rick

The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.