ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool

I’ve been working on building better and better ZFS replication tools for use at Joyent, but it often gets complex and frustrating because, although replication in ZFS is very simplistic, managing all the snapshots, retentions, and mountains of error checking and handling, on top of reporting and stats collection, is a nightmare.

So, just to relax I wrote a fun simple replication tool I call “Zynk”. It’s pathetically simple (read: elegant) and fun. As the comment says, if something breaks, its a pita to clean up, but otherwise should work well when set in motion. The intention is to run from cron every 30-600 seconds or so, but be aware that you should do the first run manually, because thats gonna take some time… the incrementals afterwards should be able to run in less than whatever frequency via cron you set.

#!/bin/bash
## ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool
## Form: zynk local/dataset root@remote.host destination/dataset

# Please note: The reason this is so simple is because there is no error checking, reporting, or cleanup.
#               In the event that something goes wonkey, you'll manually need to fix the snapshots and
#               modify or remote the /var/run/zynk datafile which contains the most recent snapshot name.
# Furthermore, this absolutely relies on the GNU version of 'date' in order to get epoch time
# Before using, make sure you've distributed your SSH key to the remote host and can ssh without password.

if [ ! $3 ]
then
        echo "Usage: zynk local/dataset root@remote.host destination/dataset"
        echo "WARNING: The destination is the full path for the remote dataset, not the prefix dataset stub."
        exit
fi

DATE=`date +%s`
if [ $DATE == "%s" ]
then
        echo "Must use GNU Date, please install and modify script."
        exit
fi

if [ -e /var/run/zynk ]
then
        # Datafile is found, creating incr.
        echo "Incremental started at `date`"
        zfs snapshot ${1}@${DATE}
        zfs send  -i  ${1}@`cat /var/run/zynk` ${1}@${DATE} | ssh ${2} zfs recv -F ${3}
        zfs destroy ${1}@`cat /var/run/zynk`
        ssh ${2} zfs destroy ${3}@`cat /var/run/zynk`
        echo ${DATE} > /var/run/zynk
        echo "Incremental complete at `date`"
else
        # Datafile not found, creating full.
        echo "Full started at `date`"
        zfs snapshot ${1}@${DATE}
        zfs send     ${1}@${DATE} | ssh ${2} zfs recv ${3}
        echo ${DATE} > /var/run/zynk
        echo "Full completed at `date`"
fi

Here it is in action:

root@quadra ~$ rm /var/run/zynk
root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Full started at Mon Nov 17 13:44:28 PST 2008
Full completed at Mon Nov 17 13:44:28 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:44:58 PST 2008
Incremental complete at Mon Nov 17 13:44:58 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:45:01 PST 2008
Incremental complete at Mon Nov 17 13:45:02 PST 2008

root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr
Incremental started at Mon Nov 17 13:45:19 PST 2008
Incremental complete at Mon Nov 17 13:45:20 PST 2008

root@quadra ~$ zfs list -r data/home/tamr backup/zynk/tamr
NAME                          USED  AVAIL  REFER  MOUNTPOINT
backup/zynk/tamr             2.45M   296G  2.45M  /backup/zynk/tamr
backup/zynk/tamr@1226958319      0      -  2.45M  -
data/home/tamr               2.47M   196M  2.45M  /data/home/tamr
data/home/tamr@1226958319        0      -  2.45M  -

Whats important to note is that it only maintains a single snapshot on either source or destination, so you don’t consume a bunch of additional space or have to worry about screwing up quotas.

This isn’t intended so much as a “real” tool, but something you can play around with and hopefully excite the mind about some new fun applications. Add error checking, add retention, add reporting, re-implement in a new language. Have fun…. drink Zima. :)

For a thorough discussion of ZFS Replication, see my post from a couple weeks ago: Understanding ZFS: Replication, Archive and Backup.

45 Responses to “ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool”

  1. Tim says:

    Ben,

    Love your posts. Only wish now was for Joyent to do the same.

    I miss the joys of about a year ago and the activity on the Joyent blog.

  2. jason says:

    Couldn’t you add to your if statement on the date the following if the date = ‘%s” then

    DATE=”`perl -e “print(time);”`”

    which would give you the same thing as the gnu date +%s, with out having to install something that might not be allowed in some environments.

  3. benr. says:

    Tim: I can write in the Joyent blog…. but then there wouldn’t be content here. Besides, most Joyent customers aren’t interested in my sort of content because I’m writting for admins with admin a level above what you get in a happy, cozzy, easy to use environment like Joyent Accelerators.

    Jason: Sure, but then your invoking PERL for something you don’t need PERL for. The point here isn’t to be clever or complete, its to simplistically demonstrate the concepts. Go forth and amaze. :)

  4. Ron says:

    The “retard” reference is offensive to those who are in care of mentally disabled people. Great post, otherwise.

  5. benr. says:

    Ron: Fine, Changed. I have not interest in offending the mentally disabled, or those around them.

  6. Zima No More says:

    I was with you until you said “Drink Zima”– since it has been discontinued by Coors as of October 2008.

  7. Simon says:

    Ben, this is great stuff. Exactly what I need at home to backup my Ultra20.

  8. benr. says:

    Seriously, Zima is gone? I used to love it, like 10 years ago, but Coors is killing my joke. Damn.

  9. saidi says:

    bon moi je mapelle saidi mohamed je ve joué dans un équipe de foot ball

  10. Hi, It was good reading your post! I’ve been hearing about John Beck’s program these days. My colleague also purchased the system to use it as a side business work.

  11. Great post! Hope to be better. Better means more features.
    good post,I think so!
    Thanks for your information, i have read it, very good!
    Bing is a really overlord!! support Bing~~
    This is great news. Best of luck for the future and keep up the good work.

  12. Today, they are perceived as status symbols and extremely popular among the young and old, and even celebrities.

  13. air max says:

    The newest version of air max 90 are available now.Welcome to see our special nike air max 2009 and air max. They are your must-have nike air max 2010|air max 97|cheap air max.

  14. thank you for sharing the article.
    The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
    Free download supported. http://www.videoconverterformac.com

  15. It’s very nice! I love what you wrote.
    I think we can make friends.

  16. Good post, thanks for sharing, I enjoy it very much! There is a site: http://www.cheapinsanitydvds.com/

  17. Your post is awesome, but why not take a look at our site: http://www.lovemypursemall.com

  18. cool stuff says:

    very cool article ,thanks for sharing the article!like my cool stuff .very useful.
    uCoolStuff is the leading China wholesaler for http://www.ucoolstuff.com cool stuff , http://www.ucoolstuff.com cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

  19. Once we get there, so many wonderful dreams will come true and the pieces of our black nike air max shoes lives will fit together like a completed jigsaw puzzle. http://www.sellnikeairmax.com/

  20. Once we get there, so many wonderful dreams will come true and the pieces of our black nike air max shoes lives will fit together like a completed jigsaw puzzle. http://www.sellnikeairmax.com/

  21. cheap ghd says:

    Many thanks for sharing this, very useful!
    Nice GHD, and you can get what you paid. GHD hair straightener

    is really good!http://www.ghdhairsonsale.com

  22. http://www.buylacosteclothing.com/lacoste-shoes-lacoste-observe-shoes-c-14_26.html
    Great! I really love how it is easy on my eyes and also the data are well written. I am wondering how I could be notified whenever a new post has been made. I have subscribed to your rss feed which must do the trick! Have a nice day!

  23. I cherished what you have got performed right here. The style is stylish, your penned articles elegant. Nonetheless, you might have bought an edginess to what that you are delivering the following.

    http://www.buylacosteclothing.com/lacoste-shoes-lacoste-protect-lase-shoes-c-14_30.html

  24. http://www.pumaferraris.com/men-s-speed-cat-big-c-47
    I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!

  25. http://www.pumaferraris.com/women-s-puma-basket-brights-c-31
    Sick unquestionably arrive back once more for very much way more in situation you protect this up. Dont wipe out hope if not as well a lot of men and women see your perspective, know you may have acquired a fan suitable the following who values what you’ve obtained to say along together with the way you’ve introduced yourself. Excellent on you!

  26. Advantageously, the post is really the freshest on this valuable topic. I concur with your conclusions and will thirstily look forward to your next updates. Just saying thanks will not just be adequate, for the exceptional clarity in your writing. I will right away grab your rss feed to stay abreast of any updates. Genuine work and much success in your business enterprize!

  27. I can see that you might be an expert at your field! I am launching a web site soon, and your info will be incredibly useful for me..!.! Thank you for all your help and wishing you all the success in your business.

    http://www.buylacosteclothing.com/lacoste-shoes-lacoste-protect-lase-shoes-c-14_30.html

  28. http://www.buylacosteclothing.com/lacoste-shoes-c-14.html
    Only want to say your article is brilliant. The lucidity in your post is simply impressive and i can assume you are an expert on this field. Well with your permission allow me to grab your rss feed to keep up to date with incoming post. Thanks a million and please keep up the a uthentic work.

  29. I can see that you might be an expert at your field! I am launching a web site soon, and your info will be incredibly useful for me..!.! Thank you for all your help and wishing you all the success in your business.

  30. moncler kid says:

    Just saying thanks will not just be adequate, for the exceptional clarity in your writing. I will right away grab your rss feed to stay abreast of any updates. Genuine work and much success in your business enterprize!

  31. Can I just say what a elimination toward come across a big name who essentially knows what theyre talking regarding on the world wide web.

  32. I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!

  33. Can I just say what a elimination toward come across a big name who essentially knows what theyre talking regarding on the world wide web.

  34. I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!

  35. Well with your permission allow me to grab your rss feed to keep up to date with future post. Thanks a million and please keep up the fabulous work.