I'm trying to analyze the performance of a Flash Cache (PAMII) module and can't seem to find a detailed definition of the "inval" and "evict" metrics. All the documentation simply states the obvious with the very unhelpful "blocks evicted/invalidated from cache per second". Both eviction and invalidation are removal of blocks from the cache, but what I want to know is why are they removed. Essentially what causes an eviction verses an invalidation? Is a block "evicted" because it was never used and a timer expired? Is a block "invalidated" when the cache is full and a useful block is thrown out to make room for a new block?
Thanks,
-Robert
stats show -p flexscale-access
Cache Reads Writes Disk Reads Usage Hit Meta Miss Hit Evict Inval Insert Chain Blocks Chain Blocks Replaced % /s /s /s % /s /s /s /s /s /s /s /s 0 0 0 0 0 0 0 0 0 0 0 0 0
Here are the definitions for those counters: Cache Usage: How much data is currently stored in the module(s) Hit: The 4kB disk block cache hit per second Meta: The 4kB metadata disk block cache hit per second Miss: The 4kB disk block cache missed per second Hit %: The percentage of total hit/miss Evict: The 4kB disk blocks evicted from the cache per second Inval: The 4kB disk blocks invalidated from the cache per second Insert: The 4kB disk blocks inserted into the cache per second Reads Chain: The number of read I/O chains per second Reads Blocks: The number of 4kB disk blocks read per second Writes Chain: The number of write I/O chains per second Writes Blocks: The number of 4kB disk blocks written per second Disk Reads Replaced: The number of reads that would have gone to disk that were replaced by the cache per second
I think you can do this, but you have to dig deeper into the rich collection of stats netapp now provides. Check out the following list of counters.
stats list counters ext_cache_obj Counters for object name: ext_cache_obj type blocks size usage accesses disk_reads_replaced hit hit_normal_lev0 hit_metadata_file hit_directory hit_indirect total_metadata_hits miss miss_metadata_file miss_directory miss_indirect hit_percent inserts inserts_normal_lev0 inserts_metadata_file inserts_directory inserts_indirect evicts evicts_ref readio_solitary readio_chains readio_blocks readio_max_in_flight readio_avg_chainlength readio_avg_latency writeio_solitary writeio_chains writeio_blocks writeio_max_in_flight writeio_avg_chainlength writeio_avg_latency invalidates
And you can go into "priv set diag" mode to see a whole lot more. All these counters can be processed into a preset like what you are looking at now "/etc/stats/preset/flexscale-access". You can look at it, it's a fairly simple list of counters to watch, and headers to use (and a few other things). You can make your own using the same format, and examine the stats_preset man page for the details on making your own.
I don't see exactly what you are looking for like eviction or invalidation by timer, but you can watch how much of the cache is being used, and possibly infer some data based on that. (stats show -i 1 ext_cache_obj:ec0:usage for example).
It's not exactly what you are looking for, but I hope it helps at least point you in the right direction.
-Blake
On Mon, Sep 19, 2011 at 12:46 PM, Robert McDermott rmcdermo@fhcrc.org wrote:
I'm trying to analyze the performance of a Flash Cache (PAMII) module and can't seem to find a detailed definition of the "inval" and "evict" metrics. All the documentation simply states the obvious with the very unhelpful "blocks evicted/invalidated from cache per second". Both eviction and invalidation are removal of blocks from the cache, but what I want to know is why are they removed. Essentially what causes an eviction verses an invalidation? Is a block "evicted" because it was never used and a timer expired? Is a block "invalidated" when the cache is full and a useful block is thrown out to make room for a new block?
Thanks,
-Robert
>stats show -p flexscale-access
Cache Reads Writes Disk Reads Usage Hit Meta Miss Hit Evict Inval Insert Chain Blocks Chain Blocks Replaced % /s /s /s % /s /s /s /s /s /s /s /s 0 0 0 0 0 0 0 0 0 0 0 0 0
Here are the definitions for those counters: Cache Usage: How much data is currently stored in the module(s) Hit: The 4kB disk block cache hit per second Meta: The 4kB metadata disk block cache hit per second Miss: The 4kB disk block cache missed per second Hit %: The percentage of total hit/miss Evict: The 4kB disk blocks evicted from the cache per second Inval: The 4kB disk blocks invalidated from the cache per second Insert: The 4kB disk blocks inserted into the cache per second Reads Chain: The number of read I/O chains per second Reads Blocks: The number of 4kB disk blocks read per second Writes Chain: The number of write I/O chains per second Writes Blocks: The number of 4kB disk blocks written per second Disk Reads Replaced: The number of reads that would have gone to disk that were replaced by the cache per second _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
On Mon, Sep 19, 2011 at 3:37 PM, Blake Golliher thelastman@gmail.com wrote:
. (stats show -i 1 ext_cache_obj:ec0:usage for example).
Nitpicking on this example, isn't this a stat for % full? That is, on a system which is warmed up, it should be at 100% at all times? I found this useful for what I needed to do with DFM and flash cache:
A system warmed up, may NOT be at 100% at all times. It dependent on the relative workload presented to the system.
PAM can be 50% full..and be 'warmed up'.
This starts to get into the subject of the math/history/etc of how long unreferenced blocks are kept in the pool.
On Mon, Sep 19, 2011 at 1:48 PM, Eugene Vilensky evilensky@gmail.comwrote:
On Mon, Sep 19, 2011 at 3:37 PM, Blake Golliher thelastman@gmail.com wrote:
. (stats show -i 1 ext_cache_obj:ec0:usage for example).
Nitpicking on this example, isn't this a stat for % full? That is, on a system which is warmed up, it should be at 100% at all times? I found this useful for what I needed to do with DFM and flash cache:
http://communities.netapp.com/thread/8461 _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
If your workload fills the cache, it should be. If your workload churns or is smaller then the amount of cache, it could not. Either way, it speaks to if block are ejected/invalidated due to size constraints of the cache.
Maybe netapp should expose some new counters for reasons why a block was invalidated, or ejected.
-Blake
On Mon, Sep 19, 2011 at 1:48 PM, Eugene Vilensky evilensky@gmail.com wrote:
On Mon, Sep 19, 2011 at 3:37 PM, Blake Golliher thelastman@gmail.com wrote:
. (stats show -i 1 ext_cache_obj:ec0:usage for example).
Nitpicking on this example, isn't this a stat for % full? That is, on a system which is warmed up, it should be at 100% at all times? I found this useful for what I needed to do with DFM and flash cache:
http://communities.netapp.com/thread/8461 _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
If your workload fills the cache, it should be. If your workload churns or is smaller then the amount of cache, it could not. Either way, it speaks to if block are ejected/invalidated due to size constraints of the cache.
Maybe netapp should expose some new counters for reasons why a block was invalidated, or ejected.
-Blake
Just speculating here...
To me "evict" indicates the removal of a valid block of data, probably to make room for a "fresher" block.
I think "invalidate" indicates that the block of data is no longer valid, perhaps because the corresponding disk block has been freed due to file deletion, etc.
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
My understanding is that "evicted" means valid blocks removed from PAM (i.e. to free space) while "invalidated" means blocks that were changed (i.e. by client write) so copy is no more actual.
--- With best regards
Andrey Borzenkov Senior system engineer Service operations
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Robert McDermott Sent: Monday, September 19, 2011 11:46 PM To: toasters@teaparty.net Subject: Flash Cache statistics question: Invalidate vs Evict?
I'm trying to analyze the performance of a Flash Cache (PAMII) module and can't seem to find a detailed definition of the "inval" and "evict" metrics. All the documentation simply states the obvious with the very unhelpful "blocks evicted/invalidated from cache per second". Both eviction and invalidation are removal of blocks from the cache, but what I want to know is why are they removed. Essentially what causes an eviction verses an invalidation? Is a block "evicted" because it was never used and a timer expired? Is a block "invalidated" when the cache is full and a useful block is thrown out to make room for a new block?
Thanks,
-Robert
stats show -p flexscale-access
Cache Reads Writes Disk Reads Usage Hit Meta Miss Hit Evict Inval Insert Chain Blocks Chain Blocks Replaced % /s /s /s % /s /s /s /s /s /s /s /s 0 0 0 0 0 0 0 0 0 0 0 0 0
Here are the definitions for those counters: Cache Usage: How much data is currently stored in the module(s) Hit: The 4kB disk block cache hit per second Meta: The 4kB metadata disk block cache hit per second Miss: The 4kB disk block cache missed per second Hit %: The percentage of total hit/miss Evict: The 4kB disk blocks evicted from the cache per second Inval: The 4kB disk blocks invalidated from the cache per second Insert: The 4kB disk blocks inserted into the cache per second Reads Chain: The number of read I/O chains per second Reads Blocks: The number of 4kB disk blocks read per second Writes Chain: The number of write I/O chains per second Writes Blocks: The number of 4kB disk blocks written per second Disk Reads Replaced: The number of reads that would have gone to disk that were replaced by the cache per second _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters