Some time ago I wrote a tool which I call arc_summary. It is a kstat based PERL application that runs some calculations and presents you with a pretty report of ZFS ARC. The idea is to help you interpret the data more appropriately. Lets look at some output:
benr@quadra ~$ ./arc_summary.pl
System Memory:
Physical RAM: 4083 MB
Free Memory : 130 MB
LotsFree: 63 MB
ZFS Tunables (/etc/system):
ARC Size:
Current Size: 1530 MB (arcsize)
Target Size (Adaptive): 1555 MB (c)
Min Size (Hard Limit): 382 MB (zfs_arc_min)
Max Size (Hard Limit): 3062 MB (zfs_arc_max)
ARC Size Breakdown:
Most Recently Used Cache Size: 100% 1555 MB (p)
Most Frequently Used Cache Size: 0% 0 MB (c-p)
ARC Efficency:
Cache Access Total: 2090556
Cache Hit Ratio: 71% 1493620 [Defined State for buffer]
Cache Miss Ratio: 28% 596936 [Undefined State for Buffer]
REAL Hit Ratio: 70% 1472533 [MRU/MFU Hits Only]
Data Demand Efficiency: 98%
Data Prefetch Efficiency: 1%
CACHE HITS BY CACHE LIST:
Anon: 0% 12854 [ New Customer, First Cache Hit ]
Most Recently Used: 55% 825782 (mru) [ Return Customer ]
Most Frequently Used: 43% 646751 (mfu) [ Frequent Customer ]
Most Recently Used Ghost: 0% 3619 (mru_ghost) [ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 4614 (mfu_ghost) [ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 60% 900081
Prefetch Data: 0% 9547
Demand Metadata: 38% 572127
Prefetch Metadata: 0% 11865
CACHE MISSES BY DATA TYPE:
Demand Data: 2% 12318
Prefetch Data: 87% 524549
Demand Metadata: 9% 55222
Prefetch Metadata: 0% 4847
---------------------------------------------
First notice that you do not need to be root to run the report. I ran this on my personal workstation, an Intel QuadCore with 4GB of RAM.
So looking at the report, first I output some general memory data for you, then output any ZFS tunable you placed in /etc/system (this will not utilize MDB, but may one day).
Then we look at the ARC Sizing data. The “Current Size” is how large the ARC really is, whereas the “Target Size” is a constantly changing number, like a stock price, of what the ARC thinks it should be. I also include the Min and Max hard sizes for ARC, these are tunable.
The ARC Size Breakdown shows the division of the cache between “Most Recently Used” (MRU) and “Most Frequently Used”. More on this later. When you see parenthesis in the output those are the variable names, for those using DTrace or getting into the code.
I wrote arc_summary to answer the question “Why is my ARC so damned big!?!” The ARC Efficiency section helps you determine where or not your getting bang for your buck. ZFS ARC stats presented by Kstat provide several breakdowns of cache hits, so this section really runs computation on those values to present them in a sane manner.
In the report above, we have a cache hit ratio of 70%. The “Cache Hit Ratio” and “REAL Hit Ratio” differ in that ZFS considers an Anon buffer as a hit… but its not really, so I remove those for the “REAL” ratio.
We then breakdown Demand vs Prefetch hits. ZFS does aggressively caches data via prefetch; in the example above, only 1% of hits are a result of prefetched data already being in the cache, 98% of it was hit in the cache because it was explicitly asked for. So in this case, pre-fetch isn’t helping me.
Perhaps the most interesting information here is the “Cache Hits by Cache List”. ARC maintains multiple cache lists, including:
- Most Recently Used: Cache hit once.
- Most Frequently Used: Cache hit multiple times.
- Most Recently Used Ghost: Objects that were in MRU cache, but removed to save space.
- Most Frequently Used Ghost: Same but for MFU.
These Ghosts lists are magic. If you get a lot of hits to the ghost lists, it means that ARC is WAY too small and that you desperately need either more RAM or an L2 ARC device (likely, SSD). Please note, if you are considering investing in L2 ARC, check this FIRST.
So by looking at the spread between MRU and MFU hits, we get an idea of how much the cache is rolling over…. 55% of hits are MRU (1 hit), whereas 43% are MFU (multiple hits), so, not bad. The higher MFU is, the better, but this all depends on your workload. Workstations will tend toward MRU, but on one of my servers, for instance, the ARC was 99% MFU.
Looking on we can see what type of requests are hits or misses. Notice that cache hits are 60% demand (explicitly requested) data blocks, and 38% demand metadata…. prefetch isn’t helping on this box.
Its all really pretty self explanatory, if its not, let me know and I’ll do an ARC expose, but I tried to create the output report to be pretty intuitive.
Please note, this report is cumulative since boot. It should compliment the arcstats tool. arcstat can tell you want is happening, arc_summary can tell you what has been happening. Any serious ZFS deployment should have both of these bad boys around in your toolbag.
On a closing note… I hope this tool helps you appreciate the innovation of the ARC. Designed at IBM (IBM ARC page) and then implemented and improved for ZFS, ARC is elegant, powerful and extremely efficient! I hope my tool can help you better appreciate this amazing innovation in some small way.
