Understanding ZFS: Disk Space Discrepancies

Here’s a good ZFS Trivia question to bewilder your friends and common question: Why do the two following outputs for the same ZFS pool disagree?

# zfs list zones
NAME    USED  AVAIL  REFER  MOUNTPOINT
zones   634G   185G    69K  /zones

# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
zones                   832G    634G    198G    76%  ONLINE     -

If you add up USED and AVAIL from the zfs list output you get 819GB. So why does zfs list say we have 819GB but zpool list says we have a 832GB pool?

This is a question I have tried to answer in the past using zdb, quite unsuccesfully. I found that answer while digging through the ZFS code (dsl_pool.c):

    408 uint64_t
    409 dsl_pool_adjustedsize(dsl_pool_t *dp, boolean_t netfree)
    410 {
    411         uint64_t space, resv;
    412
    413         /*
    414          * Reserve about 1.6% (1/64), or at least 32MB, for allocation
    415          * efficiency.
    416          * XXX The intent log is not accounted for, so it must fit
    417          * within this slop.
    418          *
    419          * If we're trying to assess whether it's OK to do a free,
    420          * cut the reservation in half to allow forward progress
    421          * (e.g. make it possible to rm(1) files from a full pool).
    422          */
    423         space = spa_get_dspace(dp->dp_spa);
    424         resv = MAX(space >> 6, SPA_MINDEVSIZE >> 1);
    425         if (netfree)
    426                 resv >>= 1;
    427
    428         return (space - resv);
    429 }

So 1/64th of the pool is reserved? It makes sense, a copy-on-write filesystem is in trouble if it truly hits 100% Used. But do those numbers fit?

1/64th of ZPool:     832G * 0.016 = 13.3GB
Output Discrepency:  832G - 819GB = 13GB

Right on the money.

Therefore, let it be known! Above the capacity lost to whatever RAID scheme you choose, you will loose 1/64th of the pool to reserve. Based on prior experiments, I can verify definitively, when you hit 100% according to zfs list you can not write any more data. Therefore, never report or monitor ZFS capacity based on zpool list! As a point of interest, df also invokes this adjustedsize function and therefore outputs correct numbers. If your monitoring system tracks disk capacity based on df no change is needed.

29 Responses to “Understanding ZFS: Disk Space Discrepancies”

  1. Matt Simmons says:

    Very nice find. I’ve seen similar effects on different filesystems, and it seems that each filesystem has its own reason for not always accurately reporting disk space. Good to know the underlying reason, and another benefit of open source software.

  2. marcelo says:

    I used to monitor space with zpool until 2w ago… then it said 97% full and still a few GBs available but zfs list reported less than half of that was free. WTF?! Now it makes sense: 3% free – 1.6% reserved. Thanks ben.

  3. Mark Harrison says:

    Zpool free is also misleading reporting free space with raidz, so it’s looking like zfs list (and calculating the percentage used/free manually) is the way to go in all cases. Interestingly, the zpool man page mentions reserving space ‘for internal accounting’, but doesn’t go into any more detail, not mention how much is reserved.

  4. Thta was an inspiring post,

    its a very informative article, keep posting, and dont drop the good work…

    Thanks

  5. [url=http://www.inuggshopping.com]UGG BOOTS[/url] Discount40%-50% off
    free shipping
    Discount40%-50% off

  6. New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

  7. adidas shoes says:

    The aim of Adidas shoes is to make the very best adidas superstar shoes at an affordable price.cheap Adidas shoes of running sneakers is about a lifestyle. Always making style and innovative advances, cheap Adidas shoes have the fashionable running shoes you will love.From inexpensive adidas running shoes to competitive track shoes, the Adidas superstar sneakers offer great value.

    http://www.adidas-cheap.com/

  8. puma shoes says:

    To recommend a wonderful puma store http://www.pumashoesale.com , where you can find the perfect puma ferrari shoes. Nothing is impossible, come on, to get the best puma sneakers, enjoy shopping.

  9. wholesale says:

    It was a very nice idea! Just wanna say thank you for the information you have shared.
    http://www.trademic.com

  10. It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. [url=http://www.mydiscountjordanshoes.com/air-jordan-xv-c-79.html] air-jordan-xv

    [/url]

  11. Buy Nike Air Max 90 Shoes just $45-55 USD inhttp://www.iofferitems.com, 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

  12. dentist says:

    Great read. I enjoyed reading your post and I like your take on the issue. Thanks.

  13. dentist says:

    Great read. I enjoyed reading your post and I like your take on the issue. Thanks.
    http://www.cosmeticdentistryguide.co.uk/

  14. ed hardy says:

    Forever 21, the US clothes retailer popular for selling fashion-forward clothing at low prices, has courted controversy by expanding into maternity wear. The brand, loved by

    youngsters and teenagers, has launched Love 21, a range of cheap and stylish maternity clothing including jeans for as little as US $8.

    http://www.ed-hardy.cc/ed-hardy-clothing.html ed hardy clothes
    http://www.ed-hardy.cc/ed-hardy-caps.html ed hardy hat

  15. Tony p90x says:

    Your article are great, will visit very soon. there is a site: http://www.p90xwork.com

  16. Good post, thanks for sharing, I enjoy it very much! There is a site: http://www.cheapinsanitydvds.com/

  17. AVI to iPad says:

    http://www.avitoipadconverter.com AVI to iPad Converter is just the most suitable tool for iPad which let iPad user freely convert various video or audio files to iPad just with simple clicks.
    http://www.magicdvdtoipad.com

  18. Couldn’t be written any better. Reading this post reminds me of my old room mate! He always kept talking about this Michael Jordan Shoes. I will forward this article to him. Pretty sure he will have a good read. Thanks for sharing!Welcome First time jumped here on your site, found on Bing. It really helps to have sombody to talk to other then my divorced parents Cheap Jordan Shoes. Thanks again.

  19. Your article are great, will visit very soon. there is a site: http://www.nikesshox.com

  20. Buildings are expensive and not everyone can buy it. Nevertheless, home loans was invented to aid different people in such kind of cases.

  21. If you want to work in favor of some still buy into the barefoot running are not beautiful face Faibufingazu Vibram things,Vibram Five Fingers Mens store them, applicable.Our, but we guarantee to purchase large If a discount.Please to give. To see some action!

  22. We are traveling by train. Out timberland hiking boots windows, we drink in the passing scene of cars on nearby highways, of children timberland shoe company waving at a crossing, of cattle grazing on a distant timberland boots hillside, of smoke pouring from a power plant, of row upon row of corn and wheat, of flatlands and timberland wheat shoes valleys, of mountains and rolling classic 3 eye timberland boat hillsides, of city skylines and village halls.But uppermost in our black timberland boots minds is the final destination. http://www.timberland4you.co.uk/sitemap.html

  23. amylin says:

    wonderfull!!I have read your article,and i like your artical so much!!!thank you for your sharing!!! I will bookmark your website
    http://www.orderandtrade.com

  24. I read books for a while but now I changed it to great blogs, your place is one of the examples why I began to pay attention to internet resources. Good luck!