Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that. It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
Li, Jackie (Yanhui) wrote:
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
more details please
this would occur on any storage IMHO. if your filesystem is 100% full it cannot write to it and if that filesystem contains things like redo logs, archive logs, and/or system tables generally bad things occur.
my first question would be why are they reaching 100% and do you have any filesystem monitoring in place? there is no reason your oracle filesystems should be reaching 100%
to me this is not a netapp/nfs issue but an operations issue--it would occur on nfs, san, or local disk.
--daniel
-- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404
310-309-4999 desk 310-430-0536 cell
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Li, Jackie (Yanhui) Sent: Fri 12/5/2008 10:33 AM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
We had experience with SAN attached device, normally the database will just start fine after you increase the file system capacity.
________________________________ From: Leeds, Daniel [mailto:dleeds@edmunds.com] Sent: Friday, December 05, 2008 11:35 AM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
this would occur on any storage IMHO. if your filesystem is 100% full it cannot write to it and if that filesystem contains things like redo logs, archive logs, and/or system tables generally bad things occur.
my first question would be why are they reaching 100% and do you have any filesystem monitoring in place? there is no reason your oracle filesystems should be reaching 100%
to me this is not a netapp/nfs issue but an operations issue--it would occur on nfs, san, or local disk.
--daniel
-- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404
310-309-4999 desk 310-430-0536 cell
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Li, Jackie (Yanhui) Sent: Fri 12/5/2008 10:33 AM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
Just because the database *may* start up OK after such an event, by no means should you assume that it *should* start up OK. The ability to come back up is completely dependent upon what state the database was in (where it was in transactions, etc) when the filesystem filled up. The more active the database was when the filesystem choked, the higher likely you'll have corruption that you can't recover from.
I completely agree with the assessments of everyone else on the list- NEVER let your filesystem fill up.
That being said- you should look at the volume "auto grow" which will (sorry) automatically grow the volume when it hits certain thresholds. Could prevent disaster in the future.
Glenn Dekhayser VoyantStrategies
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Li, Jackie (Yanhui) Sent: Friday, December 05, 2008 2:50 PM To: Leeds, Daniel; toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
We had experience with SAN attached device, normally the database will just start fine after you increase the file system capacity.
________________________________ From: Leeds, Daniel [mailto:dleeds@edmunds.com] Sent: Friday, December 05, 2008 11:35 AM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
this would occur on any storage IMHO. if your filesystem is 100% full it cannot write to it and if that filesystem contains things like redo logs, archive logs, and/or system tables generally bad things occur.
my first question would be why are they reaching 100% and do you have any filesystem monitoring in place? there is no reason your oracle filesystems should be reaching 100%
to me this is not a netapp/nfs issue but an operations issue--it would occur on nfs, san, or local disk.
--daniel
-- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404
310-309-4999 desk 310-430-0536 cell
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Li, Jackie (Yanhui) Sent: Fri 12/5/2008 10:33 AM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
This is also why you don't enable autoextend on tablespaces, and put your archive logs on a separate filesystem. Space management is part of the day to day responsibility of a DBA. Running out of space is a Very Bad Thing for databases.
Thanks,
Matt
________________________________
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Glenn Dekhayser Sent: Monday, December 08, 2008 1:49 PM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
Just because the database *may* start up OK after such an event, by no means should you assume that it *should* start up OK. The ability to come back up is completely dependent upon what state the database was in (where it was in transactions, etc) when the filesystem filled up. The more active the database was when the filesystem choked, the higher likely you'll have corruption that you can't recover from.
I completely agree with the assessments of everyone else on the list- NEVER let your filesystem fill up.
That being said- you should look at the volume "auto grow" which will (sorry) automatically grow the volume when it hits certain thresholds. Could prevent disaster in the future.
Glenn Dekhayser
VoyantStrategies
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Li, Jackie (Yanhui) Sent: Friday, December 05, 2008 2:50 PM To: Leeds, Daniel; toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
We had experience with SAN attached device, normally the database will just start fine after you increase the file system capacity.
________________________________
From: Leeds, Daniel [mailto:dleeds@edmunds.com] Sent: Friday, December 05, 2008 11:35 AM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
this would occur on any storage IMHO. if your filesystem is 100% full it cannot write to it and if that filesystem contains things like redo logs, archive logs, and/or system tables generally bad things occur.
my first question would be why are they reaching 100% and do you have any filesystem monitoring in place? there is no reason your oracle filesystems should be reaching 100%
to me this is not a netapp/nfs issue but an operations issue--it would occur on nfs, san, or local disk.
--daniel
-- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404
310-309-4999 desk 310-430-0536 cell
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Li, Jackie (Yanhui) Sent: Fri 12/5/2008 10:33 AM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
Configure and Monitor the autogrow carefully - iit won't notify you and you could grow your volume until the aggr is full. (Spoken from experience.:-) )
Also, note that it is not an 'on-demand' growth. The system polls the size periodically - iirc every minute.
On 12/8/08, Glenn Dekhayser gdekhayser@voyantinc.com wrote:
Just because the database *may* start up OK after such an event, by no means should you assume that it *should* start up OK. The ability to come back up is completely dependent upon what state the database was in (where it was in transactions, etc) when the filesystem filled up. The more active the database was when the filesystem choked, the higher likely you'll have corruption that you can't recover from.
I completely agree with the assessments of everyone else on the list- NEVER let your filesystem fill up.
That being said- you should look at the volume "auto grow" which will (sorry) automatically grow the volume when it hits certain thresholds. Could prevent disaster in the future.
Glenn Dekhayser VoyantStrategies
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Li, Jackie (Yanhui) Sent: Friday, December 05, 2008 2:50 PM To: Leeds, Daniel; toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
We had experience with SAN attached device, normally the database will just start fine after you increase the file system capacity.
From: Leeds, Daniel [mailto:dleeds@edmunds.com] Sent: Friday, December 05, 2008 11:35 AM To: Li, Jackie (Yanhui); toasters@mathworks.com Subject: RE: oracle coruption caused by NFS file system is full
this would occur on any storage IMHO. if your filesystem is 100% full it cannot write to it and if that filesystem contains things like redo logs, archive logs, and/or system tables generally bad things occur.
my first question would be why are they reaching 100% and do you have any filesystem monitoring in place? there is no reason your oracle filesystems should be reaching 100%
to me this is not a netapp/nfs issue but an operations issue--it would occur on nfs, san, or local disk.
--daniel
-- Daniel Leeds Manager, Storage Operations Edmunds, Inc. 1620 26th Street, Suite 400 South Santa Monica, CA 90404
310-309-4999 desk 310-430-0536 cell
-----Original Message----- From: owner-toasters@mathworks.com on behalf of Li, Jackie (Yanhui) Sent: Fri 12/5/2008 10:33 AM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
Li, Jackie (Yanhui) wrote:
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
First rule is, *never* let your database filesystem get full! At the very least, make sure your Redo logs are on a separate filesystem that can *never* get full. This way any data changes resulting from your datafiles running out of space (or other corruption) can be reversed safely.
Like any other storage system, whether is it NAS (NFS, CIFS) or SAN (SCSI, FC, iSCSI, FCoE), you should never, ever, ever allow it to become completely full. Especially when using structured apps like databases. Oracle crashes when the file system gets full (it would do the same with a SAN attached lun) because Oracle constantly needs to have disk space available to write to. Once full and crashed, it can't start up for the same reason. No space available to write anything - data, logs, control information updates, etc...
The simple solution is not to let that happen to you. If there is space available in the containing aggregate you can use the volume autogrow option to increase the size of the exported volume in definable increments up to a definable maximum. Eventually though, you will run out of space or reach the maximum and the same thing will happen. Whenever your aggregate gets 80% full or higher you should start the process to either free up space or add capacity to the system. Once the system aggregate goes above 90% full you're in the red zone. If you let it get more than 95% full that's your bad.
The percentage figures quoted here are my own professional opinion. They are not company policy numbers endorsed by NetApp or any other storage vendor (although I doubt they would disagree).
Hope this helps
Paul Brosseau Systems Engineer N/A East - Chesapeake Dist. 301-351-5165 Mobile Paul.Brosseau@netapp.com mailto:Paul.Brosseau@netapp.com www.netapp.com http://www.netapp.com/
From: Li, Jackie (Yanhui) [mailto:YLi@ea.com] Sent: Friday, December 05, 2008 1:34 PM To: toasters@mathworks.com Subject: oracle coruption caused by NFS file system is full
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
Thanks
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Li, Jackie (Yanhui) wrote:
Our firm runs oracle 9i using NFS file system from Netapp(3040 running 7.2), we have noticed few cases now that when the NFS file system is full, oracle database crash and can not even start after that.
It turns out that database has logical corruption, does anybody experience similar issues?
If you log Oracle to the same vol with Oracle data, ruh roh.
I recommend splitting Oracle online (and archive) logs to their own volumes to mitigate damage should Oracle data vols fill. This should allow you to replay to a consistent point-in-time once youve resolved the full data volume problem. I also recommend multi-plexing the logs to NAS and local disk just to be paranoid.
You can also flip a bit on your filer (vol autosize; done on a per-volume basis) to auto-grow a volume a specified value should you hit a threshold. This assumed you have spare aggr capacity. Use wisely.
Hope this helps your avoid future issues.
- -- Nick Silkey
-
Brosseau, Paul -
Chaim Rieger -
Glenn Dekhayser -
Jack Lyons -
Leeds, Daniel -
Li, Jackie (Yanhui) -
Matthew Zito -
Nick Silkey