What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
That sounds strange. An aligned Oracle database ought to be doing whole block writes under all circumstances, with the exception of a tiny amount of CP read behavior with the archive and redo logs. You can occasionally get into trouble with SAN filesystems where the block size of the filesystem itself results in misalignment, but not here.
Can you share the load average and top 5 wait events section of a one-hour AWR report encompassing this 15 minute cycle so we have an idea what kind of IO is occuring and the latencies you are experiencing?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fred Grieco Sent: Friday, September 13, 2013 8:07 PM To: Toasters@teaparty.net Subject: high CP_reads/writes ratio
What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
[Description: Image removed by sender.]
I'll check with the DBAs on that.
I'm looking at the replication target filer and it's actually even worse, with cp_reads almost matching user_writes on the physical disks. The replication method is dataguard.
________________________________ From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Fred Grieco fredgrieco@yahoo.com; "Toasters@teaparty.net" Toasters@teaparty.net Sent: Friday, September 13, 2013 2:21 PM Subject: RE: high CP_reads/writes ratio
That sounds strange. An aligned Oracle database ought to be doing whole block writes under all circumstances, with the exception of a tiny amount of CP read behavior with the archive and redo logs. You can occasionally get into trouble with SAN filesystems where the block size of the filesystem itself results in misalignment, but not here. Can you share the load average and top 5 wait events section of a one-hour AWR report encompassing this 15 minute cycle so we have an idea what kind of IO is occuring and the latencies you are experiencing? From:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fred Grieco Sent: Friday, September 13, 2013 8:07 PM To: Toasters@teaparty.net Subject: high CP_reads/writes ratio What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
Aha, that changes things. A standby database is an especially demanding series of alternating reads and writes. I think your best bet is probably a support case to ask them to identify the bottleneck. A couple hours of perfstat at 10 minute intervals should cover it, but you'll have to see what they say for sure. It should be pretty easy to identify the exact reason for the limitation, given the use of DataGuard.
From: Fred Grieco [mailto:fredgrieco@yahoo.com] Sent: Friday, September 13, 2013 8:58 PM To: Steiner, Jeffrey; Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
I'll check with the DBAs on that.
I'm looking at the replication target filer and it's actually even worse, with cp_reads almost matching user_writes on the physical disks. The replication method is dataguard.
________________________________ From: "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com To: Fred Grieco fredgrieco@yahoo.com; "Toasters@teaparty.net" Toasters@teaparty.net Sent: Friday, September 13, 2013 2:21 PM Subject: RE: high CP_reads/writes ratio
That sounds strange. An aligned Oracle database ought to be doing whole block writes under all circumstances, with the exception of a tiny amount of CP read behavior with the archive and redo logs. You can occasionally get into trouble with SAN filesystems where the block size of the filesystem itself results in misalignment, but not here.
Can you share the load average and top 5 wait events section of a one-hour AWR report encompassing this 15 minute cycle so we have an idea what kind of IO is occuring and the latencies you are experiencing?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fred Grieco Sent: Friday, September 13, 2013 8:07 PM To: Toasters@teaparty.net Subject: high CP_reads/writes ratio
What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
[Description: Image removed by sender.]
Or, just reallocate all volumes in the affected aggregate and save two weeks of hassle expecting support to suddenly become performance consultants.
To then be told to reallocate the volumes in the affected aggregate.
*popcorn*
IMHO, it's quarterly preventative maintenance on a netapp.
Sent from my iPhone
On Sep 13, 2013, at 3:07 PM, "Steiner, Jeffrey" Jeffrey.Steiner@netapp.com wrote:
I am pretty sure netapp took on the role of performance consultant on their own accord when they made the decision not to allow us stupid customers to use perfstat viewer ourselves.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Friday, September 13, 2013 4:23 PM To: Steiner, Jeffrey Cc: Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
Or, just reallocate all volumes in the affected aggregate and save two weeks of hassle expecting support to suddenly become performance consultants.
To then be told to reallocate the volumes in the affected aggregate.
*popcorn*
IMHO, it's quarterly preventative maintenance on a netapp.
Sent from my iPhone
On Sep 13, 2013, at 3:07 PM, "Steiner, Jeffrey" <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com> wrote: Aha, that changes things. A standby database is an especially demanding series of alternating reads and writes. I think your best bet is probably a support case to ask them to identify the bottleneck. A couple hours of perfstat at 10 minute intervals should cover it, but you’ll have to see what they say for sure. It should be pretty easy to identify the exact reason for the limitation, given the use of DataGuard.
From: Fred Grieco [mailto:fredgrieco@yahoo.com] Sent: Friday, September 13, 2013 8:58 PM To: Steiner, Jeffrey; Toasters@teaparty.netmailto:Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
I'll check with the DBAs on that.
I'm looking at the replication target filer and it's actually even worse, with cp_reads almost matching user_writes on the physical disks. The replication method is dataguard.
________________________________ From: "Steiner, Jeffrey" <Jeffrey.Steiner@netapp.commailto:Jeffrey.Steiner@netapp.com> To: Fred Grieco <fredgrieco@yahoo.commailto:fredgrieco@yahoo.com>; "Toasters@teaparty.netmailto:Toasters@teaparty.net" <Toasters@teaparty.netmailto:Toasters@teaparty.net> Sent: Friday, September 13, 2013 2:21 PM Subject: RE: high CP_reads/writes ratio
That sounds strange. An aligned Oracle database ought to be doing whole block writes under all circumstances, with the exception of a tiny amount of CP read behavior with the archive and redo logs. You can occasionally get into trouble with SAN filesystems where the block size of the filesystem itself results in misalignment, but not here.
Can you share the load average and top 5 wait events section of a one-hour AWR report encompassing this 15 minute cycle so we have an idea what kind of IO is occuring and the latencies you are experiencing?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fred Grieco Sent: Friday, September 13, 2013 8:07 PM To: Toasters@teaparty.netmailto:Toasters@teaparty.net Subject: high CP_reads/writes ratio
What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
<image001.jpg>
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Well, I did mean support specifically. Separate from PS perf consulting or knowledge housed within the customer base.
Sent from my iPhone
On Sep 13, 2013, at 3:29 PM, Jordan Slingerland Jordan.Slingerland@independenthealth.com wrote:
Wafl free space fragmentation most likely. Reallocate all volumes in the affected aggregate.
Do not reallocate the _aggregate_.
Sent from my iPhone
On Sep 13, 2013, at 1:06 PM, Fred Grieco fredgrieco@yahoo.com wrote:
Are your aggregates more then 65-85% full?
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Friday, September 13, 2013 3:09 PM To: Fred Grieco Cc: Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
Wafl free space fragmentation most likely. Reallocate all volumes in the affected aggregate.
Do not reallocate the _aggregate_.
Sent from my iPhone
On Sep 13, 2013, at 1:06 PM, Fred Grieco <fredgrieco@yahoo.commailto:fredgrieco@yahoo.com> wrote: What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
The prod one is 87%. The replication one is 71% but it was much higher during disaster recovery testing.
I'll run a reallocate measure on the DR side this weekend and see what comes back.
________________________________ From: Jordan Slingerland Jordan.Slingerland@independenthealth.com To: Jeff Mohler speedtoys.racing@gmail.com; Fred Grieco fredgrieco@yahoo.com Cc: "Toasters@teaparty.net" Toasters@teaparty.net Sent: Friday, September 13, 2013 3:12 PM Subject: RE: high CP_reads/writes ratio
Are your aggregates more then 65-85% full? From:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Friday, September 13, 2013 3:09 PM To: Fred Grieco Cc: Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio Wafl free space fragmentation most likely. Reallocate all volumes in the affected aggregate. Do not reallocate the _aggregate_.
Sent from my iPhone
On Sep 13, 2013, at 1:06 PM, Fred Grieco fredgrieco@yahoo.com wrote: What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've
attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
_______________________________________________
You can always run a measure to see where you are at as well..
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Fred Grieco Sent: Friday, September 13, 2013 12:16 PM To: Jordan Slingerland; Jeff Mohler Cc: Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
The prod one is 87%. The replication one is 71% but it was much higher during disaster recovery testing.
I'll run a reallocate measure on the DR side this weekend and see what comes back.
________________________________ From: Jordan Slingerland <Jordan.Slingerland@independenthealth.commailto:Jordan.Slingerland@independenthealth.com> To: Jeff Mohler <speedtoys.racing@gmail.commailto:speedtoys.racing@gmail.com>; Fred Grieco <fredgrieco@yahoo.commailto:fredgrieco@yahoo.com> Cc: "Toasters@teaparty.netmailto:Toasters@teaparty.net" <Toasters@teaparty.netmailto:Toasters@teaparty.net> Sent: Friday, September 13, 2013 3:12 PM Subject: RE: high CP_reads/writes ratio
Are your aggregates more then 65-85% full?
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Friday, September 13, 2013 3:09 PM To: Fred Grieco Cc: Toasters@teaparty.netmailto:Toasters@teaparty.net Subject: Re: high CP_reads/writes ratio
Wafl free space fragmentation most likely. Reallocate all volumes in the affected aggregate.
Do not reallocate the _aggregate_.
Sent from my iPhone
On Sep 13, 2013, at 1:06 PM, Fred Grieco <fredgrieco@yahoo.commailto:fredgrieco@yahoo.com> wrote: What kinds of things can cause a high CP_reads/writes ratio? And what can I do about it?
I have an Oracle system with ASM that doing writes in a 15 minute cycle, and it causes a lot of writes with CP reads. The oracle ASM luns are basically the only thing on the aggregate. Disk alignment is fine (which is the first thing I thought of). I've attached a screenshot of the Oncommand graph for one of the disks in this aggr-- not sure if that will come through.
The filer is a 3220 with 8.1.2 7-mode. Fibre channel attach, and there are about 20 luns under two volumes.
TIA,
Fred
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
In my experience 87% is likely to cause some write issues. Even if the volumes are ok the aggregate is short enough on space it's going to spend time looking for somewhere to stick incoming data.
On 09/13/2013 03:16 PM, Fred Grieco wrote:
Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
Reallocate measure won't tell you anything. It is measuring an entirely different issue altogether.
Sent from my iPhone
On Sep 13, 2013, at 2:16 PM, Fred Grieco fredgrieco@yahoo.com wrote:
-
Fred Grieco -
Jeff Mohler -
Jeremy Page -
Jordan Slingerland -
Klise, Steve -
Steiner, Jeffrey