I'm currently contemplating caching on my filers. Specifically - setting those 3 flags on a flexscale cache appropriately.
I _think_ given the workload I'm putting through one of my set of filers, enabling lopri_blocks is beneficial
When I do, a 'stats show -p flexscale-access' gives me more disk reads replaced, and a better cache hit ratio.
However I'm also looking at wafl:wafl:read_io_type and seeing that mostly - my cache module is only servicing about 10% of the IOPs going through my system anyway. Of the remainder, it's a pretty even split between disk/cache.
And with 'lopri_blocks' set to on, my cache is churning data faster - but with it off, hit rate drops to <10%.
I'm wondering if anyone can offer suggestions as to a good way to compare the two states - filer side. Are there any metrics I _should_ be looking at, that I'm not?
Primarily looking at: - total IOPs (there's only one volume on this filer) - response time. - Disk utilisation (definitely increases when lopri_blocks is off). - cache stats as above.
CP types - mostly this system is doing log_full CPs, and is getting a fairly steady stream of RW IOPs. Most IOs are NFS getattr and lookups though, which are mostly metadata. Of 100K iops, it's about 90% non-RW and 5% read, 5% write.
I think I'm right in saying that these metadata IOs are going to be fairly small and thus served out of RAM (mostly) anyway.
I'm also (re) reading TR-3832 - Flash Cache best practice. This is a little more vague than I'd like on figuring out whether 'lopri_blocks' will actually be useful.
I _think_ they are based on a better hit rate, and a better reads-replaced rate. Any read replaced is one that's served from a faster tier, and is therefore 'better' objectively, right?
Edward Rolison wrote:
I _think_ given the workload I'm putting through one of my set of filers, enabling lopri_blocks is beneficial
When I do, a 'stats show -p flexscale-access' gives me more disk reads replaced, and a better cache hit ratio.
Those two are pretty much the things you should be looking at. I.e. if Disk Read Repl and Cahce Hit Ration goes up, then that's what you want as a setting
However I'm also looking at wafl:wafl:read_io_type and seeing that mostly - my cache module is only servicing about 10% of the IOPs going through my system anyway. Of the remainder, it's a pretty even split between disk/cache.
Add more FlashCache and this will change ;-)
And with 'lopri_blocks' set to on, my cache is churning data faster - but with it off, hit rate drops to <10%.
The faster churning doesn't really matter, if you get more disk reads replaced then AOK. Run with lopri_blocks on.
Primarily looking at:
- total IOPs (there's only one volume on this filer)
- response time.
- Disk utilisation (definitely increases when lopri_blocks is off).
- cache stats as above.
You're good. That's my take on this. There's nothing else you need to look at, of all this is bejhaving the way you indicate here, you're good. It's when it's behaving badly, when it really shouldn't (i.e. protocol latency goes up)that the fun starts...
CP types - mostly this system is doing log_full CPs, and is getting a fairly steady stream of RW IOPs. Most IOs are NFS getattr and lookups though, which are mostly metadata. Of 100K iops, it's about 90% non-RW and 5% read, 5% write.
This is pretty much what my workload is. And don't look for any more metrics than you have described here.
I think I'm right in saying that these metadata IOs are going to be fairly small and thus served out of RAM (mostly) anyway.
Yes. If you don't have enough RAM buy a bigger filer. And add more FlashCache, that's almost always goood (there's a limit of course, when money spent isn't really giving you any more due to too little RAM in the machine.)
I'm also (re) reading TR-3832 - Flash Cache best practice. This is a little more vague than I'd like on figuring out whether 'lopri_blocks' will actually be useful.
I _think_ they are based on a better hit rate, and a better reads-replaced rate. Any read replaced is one that's served from a faster tier, and is therefore 'better' objectively, right?
Yes.
/M
If you have operations manager (DFM, or whatever it’s called at the moment) you can graph the stats over time. I did this for a group of our filers to see what the stats looked like before/after the change was made. Specifically I graphed:
Cache hit % Hit, hit_percent & miss Disk Reads Replaced Insert/Evicts/Invalidates
I let those run for a couple of weeks to get a baseline and then enabled lopri_blocks and basically watched what happened. For us (NFS, heavy metadata workloads) I saw the ‘churn’ of the cache go up but we also started getting much higher hit rates.
--rdp
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Edward Rolison Sent: Tuesday, March 17, 2015 12:54 PM To: toasters@teaparty.net Subject: Flexscale lopri_blocks
I'm currently contemplating caching on my filers. Specifically - setting those 3 flags on a flexscale cache appropriately.
I _think_ given the workload I'm putting through one of my set of filers, enabling lopri_blocks is beneficial
When I do, a 'stats show -p flexscale-access' gives me more disk reads replaced, and a better cache hit ratio.
However I'm also looking at wafl:wafl:read_io_type and seeing that mostly - my cache module is only servicing about 10% of the IOPs going through my system anyway. Of the remainder, it's a pretty even split between disk/cache.
And with 'lopri_blocks' set to on, my cache is churning data faster - but with it off, hit rate drops to <10%.
I'm wondering if anyone can offer suggestions as to a good way to compare the two states - filer side. Are there any metrics I _should_ be looking at, that I'm not?
Primarily looking at: - total IOPs (there's only one volume on this filer) - response time. - Disk utilisation (definitely increases when lopri_blocks is off). - cache stats as above.
CP types - mostly this system is doing log_full CPs, and is getting a fairly steady stream of RW IOPs. Most IOs are NFS getattr and lookups though, which are mostly metadata. Of 100K iops, it's about 90% non-RW and 5% read, 5% write.
I think I'm right in saying that these metadata IOs are going to be fairly small and thus served out of RAM (mostly) anyway.
I'm also (re) reading TR-3832 - Flash Cache best practice. This is a little more vague than I'd like on figuring out whether 'lopri_blocks' will actually be useful.
I _think_ they are based on a better hit rate, and a better reads-replaced rate. Any read replaced is one that's served from a faster tier, and is therefore 'better' objectively, right?
Richard Payne wrote:
Cache hit % Hit, hit_percent & miss Disk Reads Replaced Insert/Evicts/Invalidates
High Insert and Invalidate = high churn thrugh the FlashCache.
It's implemented as an 8-way Set Associative Cache underneath the WAFL buffer cache. So Evict rate is not really that interesting, but it's of course related to the size of your FlashCache vs the working set size. The latter is very very difficult to determine, it's almost impossible for deadly creatures to find that out
I let those run for a couple of weeks to get a baseline and then enabled lopri_blocks and basically watched what happened. For us (NFS, heavy metadata workloads) I saw the ‘churn’ of the cache go up but we also started getting much higher hit rates.
I agree completely. Higher Disk Read Replaced + higher Hit Rate = better. Better on those with lopri_on? Then run it like that. Workloads may change over time though, so sometimes it can make sense to switch between in intervals between lopri = on and off -- if you can be bothered
And of course: get more FlashCache if you can (it costs, I know)... With more, lopri_blocks will be more effective (of course)
/M
-
Edward Rolison -
Michael Bergman -
Payne, Richard