Thanks Jeffrey,
Do you know how I could go about getting more info on those bugs? I just checked on the now site and it does seem like we are running the latest version of SMO, 3.1.
Thanks, Dan
________________________________ From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Sent: Friday, September 30, 2011 6:27 AM Subject: RE: SMO dumping java hprof files
There is a bug or two fixed in one of the SMO P releases that can lead to java crashes during operations, and the result is this big hprof files left behind. You definitely should investigate. That only happens when a java process crashes. From:Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Thursday, September 29, 2011 7:09 PM To: toasters@teaparty.net Subject: SMO dumping java hprof files Hello, We recently noticed that our / partition on one of our oracle RAC servers was filling up. While investigating I noticed that snap manager for oracle seems to be dumping files under /opt/Netapp/smo named java_pidXXXX.hprof. These files are all around 90 meg. It appears that this has something to do with java being run with the "-XX:+HeapDumpOnOutOfMemoryError". I'm very new to this environment but from what I'm being told, there is no issues with SMO, it's functioning correctly and as it always has. I am a little concerned though that these dumps are being generated due to an out of memory error. Someone did mention that they thought during the initial setup they may have turned on debug logging but they aren't positive and they don't remember how it was turned it on. Has anyone else ever run into this? We are running: SMO 3.1 SnapDrive 4.2 Oracle 11g RHEL 5.4 Using both NFS and FCP ASM is mounting the FCP luns Thanks, Dan
I usually use SMO 3.1P1, but if you've already got that, then there's one other thing to consider. Some customers have an unusually large number of archive logs and SMO can have problems cataloging them all.
First, locate the launch-java file in /opt/NetApp or /opt/netapp and locate a line that is similar to this:
java -Xmx96m -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -enableassertions ${JVM_ARGS} -cp ${CLASSPATH} ${1+"$@"}
The "96" there is the max heap size. Try changing that to 128 or 256 and see if the problem disappears. If not, then the next step would be to open a support case.
From: Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Friday, September 30, 2011 7:30 PM To: Steiner, Jeffrey Cc: toasters@teaparty.net Subject: Re: SMO dumping java hprof files
Thanks Jeffrey,
Do you know how I could go about getting more info on those bugs? I just checked on the now site and it does seem like we are running the latest version of SMO, 3.1.
Thanks,
Dan
________________________________
From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Sent: Friday, September 30, 2011 6:27 AM Subject: RE: SMO dumping java hprof files
There is a bug or two fixed in one of the SMO P releases that can lead to java crashes during operations, and the result is this big hprof files left behind. You definitely should investigate. That only happens when a java process crashes.
From: Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Thursday, September 29, 2011 7:09 PM To: toasters@teaparty.net Subject: SMO dumping java hprof files
Hello,
We recently noticed that our / partition on one of our oracle RAC servers was filling up. While investigating I noticed that snap manager for oracle seems to be dumping files under /opt/Netapp/smo named java_pidXXXX.hprof. These files are all around 90 meg. It appears that this has something to do with java being run with the "-XX:+HeapDumpOnOutOfMemoryError".
I'm very new to this environment but from what I'm being told, there is no issues with SMO, it's functioning correctly and as it always has. I am a little concerned though that these dumps are being generated due to an out of memory error.
Someone did mention that they thought during the initial setup they may have turned on debug logging but they aren't positive and they don't remember how it was turned it on.
Has anyone else ever run into this?
We are running:
SMO 3.1
SnapDrive 4.2
Oracle 11g
RHEL 5.4
Using both NFS and FCP
ASM is mounting the FCP luns
Thanks,
Dan
Would 5100 be an unusually large number of archive logs? We didn't have any process in place to clean them up so we were storing them for much longer than we needed to. I just had my DBA remove everything except for this months and I'm hoping that will resolve these out of memory errors.
________________________________ From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Cc: toasters@teaparty.net Sent: Friday, September 30, 2011 12:37 PM Subject: RE: SMO dumping java hprof files
I usually use SMO 3.1P1, but if you’ve already got that, then there’s one other thing to consider. Some customers have an unusually large number of archive logs and SMO can have problems cataloging them all. First, locate the launch-java file in /opt/NetApp or /opt/netapp and locate a line that is similar to this: java -Xmx96m -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -enableassertions ${JVM_ARGS} -cp ${CLASSPATH} ${1+"$@"} The “96” there is the max heap size. Try changing that to 128 or 256 and see if the problem disappears. If not, then the next step would be to open a support case. From:Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Friday, September 30, 2011 7:30 PM To: Steiner, Jeffrey Cc: toasters@teaparty.net Subject: Re: SMO dumping java hprof files Thanks Jeffrey, Do you know how I could go about getting more info on those bugs? I just checked on the now site and it does seem like we are running the latest version of SMO, 3.1. Thanks, Dan
________________________________
From:"Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Sent: Friday, September 30, 2011 6:27 AM Subject: RE: SMO dumping java hprof files There is a bug or two fixed in one of the SMO P releases that can lead to java crashes during operations, and the result is this big hprof files left behind. You definitely should investigate. That only happens when a java process crashes. From:Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Thursday, September 29, 2011 7:09 PM To: toasters@teaparty.net Subject: SMO dumping java hprof files Hello, We recently noticed that our / partition on one of our oracle RAC servers was filling up. While investigating I noticed that snap manager for oracle seems to be dumping files under /opt/Netapp/smo named java_pidXXXX.hprof. These files are all around 90 meg. It appears that this has something to do with java being run with the "-XX:+HeapDumpOnOutOfMemoryError". I'm very new to this environment but from what I'm being told, there is no issues with SMO, it's functioning correctly and as it always has. I am a little concerned though that these dumps are being generated due to an out of memory error. Someone did mention that they thought during the initial setup they may have turned on debug logging but they aren't positive and they don't remember how it was turned it on. Has anyone else ever run into this? We are running: SMO 3.1 SnapDrive 4.2 Oracle 11g RHEL 5.4 Using both NFS and FCP ASM is mounting the FCP luns Thanks, Dan
Patch releases are not listed on the NOW site along with the GA/GD releases of software. You have to grab them from the main Software Download page at the bottom under To Access a Specific [software name] Version, enter it here [version] and play guess the latest P release.
The BURT that sounds like your problem and what Jeff is referring to is here “Burt # 488753 -- SMO/SMSAP should not fail with java heap errors during refreshing backup protection states or listing backups.” and is fixed in the SMO 3.1P1 release of the software. You can grab that version here: https://now.netapp.com/NOW/download/software/snapmanager_oracle_unix/3.1P1/
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Dan Finn Sent: Friday, September 30, 2011 1:47 PM To: Steiner, Jeffrey Cc: toasters@teaparty.net Subject: Re: SMO dumping java hprof files
Would 5100 be an unusually large number of archive logs? We didn't have any process in place to clean them up so we were storing them for much longer than we needed to. I just had my DBA remove everything except for this months and I'm hoping that will resolve these out of memory errors.
________________________________ From: "Steiner, Jeffrey" <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com> To: Dan Finn <dan_j_finn@yahoo.commailto:dan_j_finn@yahoo.com> Cc: toasters@teaparty.netmailto:toasters@teaparty.net Sent: Friday, September 30, 2011 12:37 PM Subject: RE: SMO dumping java hprof files I usually use SMO 3.1P1, but if you’ve already got that, then there’s one other thing to consider. Some customers have an unusually large number of archive logs and SMO can have problems cataloging them all.
First, locate the launch-java file in /opt/NetApp or /opt/netapp and locate a line that is similar to this:
java -Xmx96m -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -enableassertions ${JVM_ARGS} -cp ${CLASSPATH} ${1+"$@"}
The “96” there is the max heap size. Try changing that to 128 or 256 and see if the problem disappears. If not, then the next step would be to open a support case.
From: Dan Finn [mailto:dan_j_finn@yahoo.com]mailto:[mailto:dan_j_finn@yahoo.com] Sent: Friday, September 30, 2011 7:30 PM To: Steiner, Jeffrey Cc: toasters@teaparty.netmailto:toasters@teaparty.net Subject: Re: SMO dumping java hprof files
Thanks Jeffrey,
Do you know how I could go about getting more info on those bugs? I just checked on the now site and it does seem like we are running the latest version of SMO, 3.1.
Thanks, Dan
________________________________ From: "Steiner, Jeffrey" <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com> To: Dan Finn <dan_j_finn@yahoo.commailto:dan_j_finn@yahoo.com> Sent: Friday, September 30, 2011 6:27 AM Subject: RE: SMO dumping java hprof files There is a bug or two fixed in one of the SMO P releases that can lead to java crashes during operations, and the result is this big hprof files left behind. You definitely should investigate. That only happens when a java process crashes.
From: Dan Finn [mailto:dan_j_finn@yahoo.com]mailto:[mailto:dan_j_finn@yahoo.com] Sent: Thursday, September 29, 2011 7:09 PM To: toasters@teaparty.netmailto:toasters@teaparty.net Subject: SMO dumping java hprof files
Hello,
We recently noticed that our / partition on one of our oracle RAC servers was filling up. While investigating I noticed that snap manager for oracle seems to be dumping files under /opt/Netapp/smo named java_pidXXXX.hprof. These files are all around 90 meg. It appears that this has something to do with java being run with the "-XX:+HeapDumpOnOutOfMemoryError".
I'm very new to this environment but from what I'm being told, there is no issues with SMO, it's functioning correctly and as it always has. I am a little concerned though that these dumps are being generated due to an out of memory error.
Someone did mention that they thought during the initial setup they may have turned on debug logging but they aren't positive and they don't remember how it was turned it on.
Has anyone else ever run into this?
We are running: SMO 3.1 SnapDrive 4.2 Oracle 11g RHEL 5.4 Using both NFS and FCP ASM is mounting the FCP luns
Thanks, Dan
5100 is on the high side, and it’s in the range where this problem can occur. I’m not entirely sure why some SMO installations have seen this problems while others done.
I also got an email from engineering that 3.1P2 has further patches that can prevent this.
From: Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Friday, September 30, 2011 7:47 PM To: Steiner, Jeffrey Cc: toasters@teaparty.net Subject: Re: SMO dumping java hprof files
Would 5100 be an unusually large number of archive logs? We didn't have any process in place to clean them up so we were storing them for much longer than we needed to. I just had my DBA remove everything except for this months and I'm hoping that will resolve these out of memory errors.
________________________________
From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Cc: toasters@teaparty.net Sent: Friday, September 30, 2011 12:37 PM Subject: RE: SMO dumping java hprof files
I usually use SMO 3.1P1, but if you’ve already got that, then there’s one other thing to consider. Some customers have an unusually large number of archive logs and SMO can have problems cataloging them all.
First, locate the launch-java file in /opt/NetApp or /opt/netapp and locate a line that is similar to this:
java -Xmx96m -Djava.net.preferIPv4Stack=true -XX:+HeapDumpOnOutOfMemoryError -enableassertions ${JVM_ARGS} -cp ${CLASSPATH} ${1+"$@"}
The “96” there is the max heap size. Try changing that to 128 or 256 and see if the problem disappears. If not, then the next step would be to open a support case.
From: Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Friday, September 30, 2011 7:30 PM To: Steiner, Jeffrey Cc: toasters@teaparty.net Subject: Re: SMO dumping java hprof files
Thanks Jeffrey,
Do you know how I could go about getting more info on those bugs? I just checked on the now site and it does seem like we are running the latest version of SMO, 3.1.
Thanks,
Dan
________________________________
From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Dan Finn dan_j_finn@yahoo.com Sent: Friday, September 30, 2011 6:27 AM Subject: RE: SMO dumping java hprof files
There is a bug or two fixed in one of the SMO P releases that can lead to java crashes during operations, and the result is this big hprof files left behind. You definitely should investigate. That only happens when a java process crashes.
From: Dan Finn [mailto:dan_j_finn@yahoo.com] Sent: Thursday, September 29, 2011 7:09 PM To: toasters@teaparty.net Subject: SMO dumping java hprof files
Hello,
We recently noticed that our / partition on one of our oracle RAC servers was filling up. While investigating I noticed that snap manager for oracle seems to be dumping files under /opt/Netapp/smo named java_pidXXXX.hprof. These files are all around 90 meg. It appears that this has something to do with java being run with the "-XX:+HeapDumpOnOutOfMemoryError".
I'm very new to this environment but from what I'm being told, there is no issues with SMO, it's functioning correctly and as it always has. I am a little concerned though that these dumps are being generated due to an out of memory error.
Someone did mention that they thought during the initial setup they may have turned on debug logging but they aren't positive and they don't remember how it was turned it on.
Has anyone else ever run into this?
We are running:
SMO 3.1
SnapDrive 4.2
Oracle 11g
RHEL 5.4
Using both NFS and FCP
ASM is mounting the FCP luns
Thanks,
Dan
-
Chris Muellner -
Dan Finn -
Steiner, Jeffrey