We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described?
thanks, Fletcher
Thats good numbers!
You wont get per volume cache hits/etc, its a system wide pool..but you can use 'priority' to say that PER volume, you want it sticky (keep) neutral (reuse) or never (never) to store those vol blocks in cache.
Cache is always random IO, sequential IO is kept on disk..plenty fast from there and doesn't pollute the flash cache.
Your next step is a chart showing the cache hit types. Level0, metadata, directory...etc.
:)
Good stuff you're showing there.
On Mon, May 14, 2012 at 12:56 PM, Fletcher Cocquyt fcocquyt@stanford.eduwrote:
We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described?
thanks, Fletcher
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
There's another view you can add into Perf Advisor that provides another more overall and possibly more useful perspective than the standard three that you normally find in the documents. It breaks down what percentage of the total reads are coming out of the main system cache, the extended cache (Flash Cache), and the disk.
In the NetApp Management Console do the following:
Setup Hosts > Select a controller > Data Collection > Edit
Disabled Counters wafl > read_io_type (select and add to the right column to Enabled Counters)
Data Collection Details Modify wafl retention to be longer than the default of 1 week if you'd like.
Custom Views Select an existing Flash Cache view and edit it Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph > Category: Wafl; Storage System: Controller_Name > Available Counters: Wafl > Cache + Ext_Cache + Disk (Add all three to the right). Save the view.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Monday, May 14, 2012 3:04 PM To: Fletcher Cocquyt Cc: Mailing Lists Subject: Re: What is Flash Cache caching?
Thats good numbers!
You wont get per volume cache hits/etc, its a system wide pool..but you can use 'priority' to say that PER volume, you want it sticky (keep) neutral (reuse) or never (never) to store those vol blocks in cache.
Cache is always random IO, sequential IO is kept on disk..plenty fast from there and doesn't pollute the flash cache.
Your next step is a chart showing the cache hit types. Level0, metadata, directory...etc.
:)
Good stuff you're showing there.
On Mon, May 14, 2012 at 12:56 PM, Fletcher Cocquyt <fcocquyt@stanford.edumailto:fcocquyt@stanford.edu> wrote: We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described?
[cid:image001.png@01CD31EA.36DFFE50]
thanks, Fletcher
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
Ya..this is the kind of insight that I like seeing people get into.
Good stuff.
On Mon, May 14, 2012 at 1:57 PM, Chris Muellner chris@northlandusa.comwrote:
There’s another view you can add into Perf Advisor that provides another more overall and possibly more useful perspective than the standard three that you normally find in the documents. It breaks down what percentage of the total reads are coming out of the main system cache, the extended cache (Flash Cache), and the disk.****
In the NetApp Management Console do the following:****
*Setup*
Hosts > Select a controller > Data Collection > Edit****
*Disabled Counters*
wafl > read_io_type (select and add to the right column to Enabled Counters)****
*Data Collection Details*
Modify wafl retention to be longer than the default of 1 week if you’d like.****
*Custom Views*****
Select an existing Flash Cache view and edit it****
Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph
Category: Wafl; Storage System: Controller_Name > Available Counters:
Wafl > Cache + Ext_Cache + Disk (Add all three to the right). Save the view.
*From:* toasters-bounces@teaparty.net [mailto: toasters-bounces@teaparty.net] *On Behalf Of *Jeff Mohler *Sent:* Monday, May 14, 2012 3:04 PM *To:* Fletcher Cocquyt *Cc:* Mailing Lists *Subject:* Re: What is Flash Cache caching?****
Thats good numbers!
You wont get per volume cache hits/etc, its a system wide pool..but you can use 'priority' to say that PER volume, you want it sticky (keep) neutral (reuse) or never (never) to store those vol blocks in cache.
Cache is always random IO, sequential IO is kept on disk..plenty fast from there and doesn't pollute the flash cache.
Your next step is a chart showing the cache hit types. Level0, metadata, directory...etc.
:)
Good stuff you're showing there.
On Mon, May 14, 2012 at 12:56 PM, Fletcher Cocquyt fcocquyt@stanford.edu wrote:****
We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832.****
Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content.*
So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting.****
We also have oracle instances, VMware volumes and would like per volume stats if possible****
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced ****
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500****
Does this look like what others are seeing in a mixed workload environment like I described?****
thanks,****
Fletcher****
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters****
--
Gustatus Similis Pullus****
I screwed up the Available Counter sub-category where you find Cache, Ext_Cache, and Disk. After Available Counters the sub-category should be "read_io_type" in order to add the disk, cache, and ext_cache counters. Sorry for any confusion for people staring at their screen going where is it? Also, I believe you need to be running 8.0.2 or later to access these counters. I'm not sure when exactly they were added (maybe 8.0.1?). Custom Views Select an existing Flash Cache view and edit it Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph > Category: Wafl; Storage System: Controller_Name > Available Counters: read_io_type > cache + ext_cache + disk (Add all three to the right). Save the view.
From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Monday, May 14, 2012 4:44 PM To: Chris Muellner Cc: Fletcher Cocquyt; Mailing Lists Subject: Re: What is Flash Cache caching?
Ya..this is the kind of insight that I like seeing people get into.
Good stuff.
On Mon, May 14, 2012 at 1:57 PM, Chris Muellner <chris@northlandusa.commailto:chris@northlandusa.com> wrote: There's another view you can add into Perf Advisor that provides another more overall and possibly more useful perspective than the standard three that you normally find in the documents. It breaks down what percentage of the total reads are coming out of the main system cache, the extended cache (Flash Cache), and the disk.
In the NetApp Management Console do the following:
Setup Hosts > Select a controller > Data Collection > Edit
Disabled Counters wafl > read_io_type (select and add to the right column to Enabled Counters)
Data Collection Details Modify wafl retention to be longer than the default of 1 week if you'd like.
Custom Views Select an existing Flash Cache view and edit it Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph > Category: Wafl; Storage System: Controller_Name > Available Counters: Wafl > Cache + Ext_Cache + Disk (Add all three to the right). Save the view.
From: toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Monday, May 14, 2012 3:04 PM To: Fletcher Cocquyt Cc: Mailing Lists Subject: Re: What is Flash Cache caching?
Thats good numbers!
You wont get per volume cache hits/etc, its a system wide pool..but you can use 'priority' to say that PER volume, you want it sticky (keep) neutral (reuse) or never (never) to store those vol blocks in cache.
Cache is always random IO, sequential IO is kept on disk..plenty fast from there and doesn't pollute the flash cache.
Your next step is a chart showing the cache hit types. Level0, metadata, directory...etc.
:)
Good stuff you're showing there.
On Mon, May 14, 2012 at 12:56 PM, Fletcher Cocquyt <fcocquyt@stanford.edumailto:fcocquyt@stanford.edu> wrote: We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described?
[cid:image001.png@01CD31F6.9E5E52F0]
thanks, Fletcher
_______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
-- --- Gustatus Similis Pullus
-- --- Gustatus Similis Pullus
Hi Chris
Some interesting observations I'm trying to explain below - thanks to your graph I have a theory (below)
I'm trying to interpret what I see here - starting around 7pm last night the Disk Reads replaced dropped from 4500 to < 100 - I'm investigating what 7pm jobs might affect this…
but the interesting thing is your graph shows some Reads Served stopped coming from disk and started coming more from cache (controller cache?) - while the ext_cache level remains about the same…(about 10-15%)
One explanation follows the strict definition of "Disk Reads Replaced" - since there are few disk reads to be replaced (they shifted to controller cache) - the "Disk Reads Replaced" plummets.
Does that make some sense?
thanks
On May 14, 2012, at 3:25 PM, Chris Muellner wrote:
I screwed up the Available Counter sub-category where you find Cache, Ext_Cache, and Disk. After Available Counters the sub-category should be “read_io_type” in order to add the disk, cache, and ext_cache counters. Sorry for any confusion for people staring at their screen going where is it? Also, I believe you need to be running 8.0.2 or later to access these counters. I’m not sure when exactly they were added (maybe 8.0.1?). Custom Views Select an existing Flash Cache view and edit it Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph > Category: Wafl; Storage System: Controller_Name > Available Counters: read_io_type > cache + ext_cache + disk (Add all three to the right). Save the view.
From: Jeff Mohler [mailto:speedtoys.racing@gmail.com] Sent: Monday, May 14, 2012 4:44 PM To: Chris Muellner Cc: Fletcher Cocquyt; Mailing Lists Subject: Re: What is Flash Cache caching?
Ya..this is the kind of insight that I like seeing people get into.
Good stuff.
On Mon, May 14, 2012 at 1:57 PM, Chris Muellner chris@northlandusa.com wrote: There’s another view you can add into Perf Advisor that provides another more overall and possibly more useful perspective than the standard three that you normally find in the documents. It breaks down what percentage of the total reads are coming out of the main system cache, the extended cache (Flash Cache), and the disk.
In the NetApp Management Console do the following:
Setup Hosts > Select a controller > Data Collection > Edit
Disabled Counters wafl > read_io_type (select and add to the right column to Enabled Counters)
Data Collection Details Modify wafl retention to be longer than the default of 1 week if you’d like.
Custom Views Select an existing Flash Cache view and edit it Add Chart > Name the Chart (ex: Reads Served % or something) > Line Graph > Category: Wafl; Storage System: Controller_Name > Available Counters: Wafl > Cache + Ext_Cache + Disk (Add all three to the right). Save the view.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Jeff Mohler Sent: Monday, May 14, 2012 3:04 PM To: Fletcher Cocquyt Cc: Mailing Lists Subject: Re: What is Flash Cache caching?
Thats good numbers!
You wont get per volume cache hits/etc, its a system wide pool..but you can use 'priority' to say that PER volume, you want it sticky (keep) neutral (reuse) or never (never) to store those vol blocks in cache.
Cache is always random IO, sequential IO is kept on disk..plenty fast from there and doesn't pollute the flash cache.
Your next step is a chart showing the cache hit types. Level0, metadata, directory...etc.
:)
Good stuff you're showing there.
On Mon, May 14, 2012 at 12:56 PM, Fletcher Cocquyt fcocquyt@stanford.edu wrote: We setup a custom view in NMC to show us the hit % and Disk Reads replaced per the Flash Cache Best Practices doc TR-3832. Is it possible to get more granular information about what is being cached?
These are NFS data stores and one of the volumes is 500Gb of web content. So the web team would be very interested to know the net benefit to them - eg how much are the apache instances benefitting. We also have oracle instances, VMware volumes and would like per volume stats if possible
I guess I am looking for the ability to apply the custom view to different vFilers and see per volume hits and hit %, Disk Reads replaced
We have the 1Tb card installed and see the Disk Reads Replaced topping out around 5000 - with one spike up to ~8500 Does this look like what others are seeing in a mixed workload environment like I described?
<image001.png>
thanks, Fletcher
Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
--
Gustatus Similis Pullus
--
Gustatus Similis Pullus
-
Chris Muellner -
Fletcher Cocquyt -
Jeff Mohler