Hi Everyone,
Over the weekend we had a snapvault backup fail with memory allocation errors on the snapvault destination. This error occurred twice on the same volume, then hasn't come back.
Can not allocate memory. (memory limit reached) (Cannot allocate memory)
I run snapmirror and snapvault on all our volumes, but do it one operation at a time - NO concurrent snapvault or snapmirror operations.
I search NOW and didn't come up with anything.
This occurred on this particular volume twice in a row (6 hours apart - backups run every 6hours), not on any other volumes.
This volume is 410GB with 900k inodes - not even close to our biggest volumes. So this error is not related to size (capacity or num inodes), but seems specific to this volume.
Filers: IBM N3220 (NetApp FAS2240-2) OnTap: 8.1.2P1 7-Mode
Here are the full messages on the SV destination filer.
___ Sat 09 20 07:43:05 EDT [rshd_0:debug]: root@[172.17.170.34_60065]:IN:ssh2 shell:SSH INPUT COMMAND is snapvault snap create -w v_fnce08p_cifs v_fnce08p_cifs tgt Sat 09 20 07:43:05 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_start dst Sat 09 20 07:43:06 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Request (Update) dst Sat 09 20 07:43:07 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Start ___ Sat 09 20 07:50:43 EDT [isnabk0665a:replication.dst.waflErr:error]: SnapVault: destination transfer from isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs to /vol/v_fnce08p_cifs/q_fnce08p_cifs : memory limit reached:Can not allocate memory. dst Sat 09 20 07:50:43 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Defer (memory limit reached) (Cannot allocate memory) dst Sat 09 20 07:51:00 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Request (Retry) dst Sat 09 20 07:51:01 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Restart (@ 929692 KB) dst Sat 09 20 07:51:04 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Abort (memory limit reached) (Cannot allocate memory) dst Sat 09 20 07:51:04 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Rollback_start slk Sat 09 20 07:51:30 EDT state.qtree_softlock.v_fnce08p_cifs.0001b0a2.252.q_fnce08p_cifs.src-qt.isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs.00000000.-01.1 Softlock_add (Transfer) dst Sat 09 20 07:51:37 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Rollback_end (42928 KB) slk Sat 09 20 07:51:37 EDT state.qtree_softlock.v_fnce08p_cifs.0001b0a2.252.q_fnce08p_cifs.src-qt.isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs.00000000.-01.1 Softlock_delete (Transfer) dst Sat 09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_end dst Sat 09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_start (isnabk0665a(1896301030)_v_fnce08p_cifs-base.2) dst Sat 09 20 07:51:38 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_start (isnabk0665a(1896301030)_v_fnce08p_cifs-cleanup.0) dst Sat 09 20 07:51:40 EDT isnapr0664a:/vol/v_fnce08p_cifs/q_fnce08p_cifs isnabk0665a:/vol/v_fnce08p_cifs/q_fnce08p_cifs Coalesce_end ___ Sat 09 20 07:51:41 EDT [rshd_0:debug]: root@[172.17.170.34_60540]:IN:ssh2 shell:SSH INPUT COMMAND is snapvault status -l /vol/v_fnce08p_cifs/q_fnce08p_cifs tgt Sat 09 20 07:51:41 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_create_snapshot (v_fnce08p_cifs.0) tgt Sat 09 20 07:51:41 EDT v_fnce08p_cifs xfer:v_fnce08p_cifs Target_end vol Sat 09 20 07:54:43 EDT v_fnce08p_cifs Snapshot_replace (v_fnce08p_cifs.0)
Thanks!
Rick
----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.