I have lived through this harrowing experience. Yes there were warning signs but as I was busy performing other tasks I did not see them. The moral of my story is to read your weekly message logs and have good back ups. I also file every weekly autosupport message. I have message logs from everyone of my filers, we currently have twenty-one, dating back to 1996 when I started this position.
I have a nightly cron job that emails me the interesting lines from /etc/messages. Our filer gets tons of disk quota exceeded messages because we have over 20,000 users and each one has a quota. And I also filter out those hourly status messages. There's no way I can wade through those weekly emails because /etc/messages is usually about 5000 lines long. I'd probably forget to check the logs by hand, but I always read my email.
This is roughly how my cron script emails me:
MAILTO=mailid@mailhost
( echo "Subject: netapp daily log" echo egrep -v 'NFS ops|quota exceeded|message repeated' /na/etc/messages ) | /usr/lib/sendmail $MAILTO
Steve Losen scl@virginia.edu phone: 804-924-0640
University of Virginia ITC Unix Support