I’ve been working on building better and better ZFS replication tools for use at Joyent, but it often gets complex and frustrating because, although replication in ZFS is very simplistic, managing all the snapshots, retentions, and mountains of error checking and handling, on top of reporting and stats collection, is a nightmare.
So, just to relax I wrote a fun simple replication tool I call “Zynk”. It’s pathetically simple (read: elegant) and fun. As the comment says, if something breaks, its a pita to clean up, but otherwise should work well when set in motion. The intention is to run from cron every 30-600 seconds or so, but be aware that you should do the first run manually, because thats gonna take some time… the incrementals afterwards should be able to run in less than whatever frequency via cron you set.
#!/bin/bash
## ZYNK: The Zuper Zimple ZFS Sync (Replication) Tool
## Form: zynk local/dataset root@remote.host destination/dataset
# Please note: The reason this is so simple is because there is no error checking, reporting, or cleanup.
# In the event that something goes wonkey, you'll manually need to fix the snapshots and
# modify or remote the /var/run/zynk datafile which contains the most recent snapshot name.
# Furthermore, this absolutely relies on the GNU version of 'date' in order to get epoch time
# Before using, make sure you've distributed your SSH key to the remote host and can ssh without password.
if [ ! $3 ]
then
echo "Usage: zynk local/dataset root@remote.host destination/dataset"
echo "WARNING: The destination is the full path for the remote dataset, not the prefix dataset stub."
exit
fi
DATE=`date +%s`
if [ $DATE == "%s" ]
then
echo "Must use GNU Date, please install and modify script."
exit
fi
if [ -e /var/run/zynk ]
then
# Datafile is found, creating incr.
echo "Incremental started at `date`"
zfs snapshot ${1}@${DATE}
zfs send -i ${1}@`cat /var/run/zynk` ${1}@${DATE} | ssh ${2} zfs recv -F ${3}
zfs destroy ${1}@`cat /var/run/zynk`
ssh ${2} zfs destroy ${3}@`cat /var/run/zynk`
echo ${DATE} > /var/run/zynk
echo "Incremental complete at `date`"
else
# Datafile not found, creating full.
echo "Full started at `date`"
zfs snapshot ${1}@${DATE}
zfs send ${1}@${DATE} | ssh ${2} zfs recv ${3}
echo ${DATE} > /var/run/zynk
echo "Full completed at `date`"
fi
Here it is in action:
root@quadra ~$ rm /var/run/zynk root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr Full started at Mon Nov 17 13:44:28 PST 2008 Full completed at Mon Nov 17 13:44:28 PST 2008 root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr Incremental started at Mon Nov 17 13:44:58 PST 2008 Incremental complete at Mon Nov 17 13:44:58 PST 2008 root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr Incremental started at Mon Nov 17 13:45:01 PST 2008 Incremental complete at Mon Nov 17 13:45:02 PST 2008 root@quadra ~$ ./zynk data/home/tamr root@localhost backup/zynk/tamr Incremental started at Mon Nov 17 13:45:19 PST 2008 Incremental complete at Mon Nov 17 13:45:20 PST 2008 root@quadra ~$ zfs list -r data/home/tamr backup/zynk/tamr NAME USED AVAIL REFER MOUNTPOINT backup/zynk/tamr 2.45M 296G 2.45M /backup/zynk/tamr backup/zynk/tamr@1226958319 0 - 2.45M - data/home/tamr 2.47M 196M 2.45M /data/home/tamr data/home/tamr@1226958319 0 - 2.45M -
Whats important to note is that it only maintains a single snapshot on either source or destination, so you don’t consume a bunch of additional space or have to worry about screwing up quotas.
This isn’t intended so much as a “real” tool, but something you can play around with and hopefully excite the mind about some new fun applications. Add error checking, add retention, add reporting, re-implement in a new language. Have fun…. drink Zima. 
For a thorough discussion of ZFS Replication, see my post from a couple weeks ago: Understanding ZFS: Replication, Archive and Backup.
Ben,
Love your posts. Only wish now was for Joyent to do the same.
I miss the joys of about a year ago and the activity on the Joyent blog.
Couldn’t you add to your if statement on the date the following if the date = ‘%s” then
DATE=”`perl -e “print(time);”`”
which would give you the same thing as the gnu date +%s, with out having to install something that might not be allowed in some environments.
Tim: I can write in the Joyent blog…. but then there wouldn’t be content here. Besides, most Joyent customers aren’t interested in my sort of content because I’m writting for admins with admin a level above what you get in a happy, cozzy, easy to use environment like Joyent Accelerators.
Jason: Sure, but then your invoking PERL for something you don’t need PERL for. The point here isn’t to be clever or complete, its to simplistically demonstrate the concepts. Go forth and amaze.
The “retard” reference is offensive to those who are in care of mentally disabled people. Great post, otherwise.
Ron: Fine, Changed. I have not interest in offending the mentally disabled, or those around them.
I was with you until you said “Drink Zima”– since it has been discontinued by Coors as of October 2008.
Ben, this is great stuff. Exactly what I need at home to backup my Ultra20.
Seriously, Zima is gone? I used to love it, like 10 years ago, but Coors is killing my joke. Damn.
bon moi je mapelle saidi mohamed je ve joué dans un équipe de foot ball
Hi, It was good reading your post! I’ve been hearing about John Beck’s program these days. My colleague also purchased the system to use it as a side business work.
Thank you very much!
Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.
links of london
thanks
good read thanks
Today, they are perceived as status symbols and extremely popular among the young and old, and even celebrities.
good
http://www.trademic.com/product/index.tm.keyword=AF%20shirts&bid=.html
I love http://www.shoppingmbtshoes.com
The newest version of air max 90 are available now.Welcome to see our special nike air max 2009 and air max. They are your must-have nike air max 2010|air max 97|cheap air max.
thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. http://www.videoconverterformac.com
http://www.air jordan shoes cheap.com/
http://www.vibramfivefinger.us/ vibram five fingers
It’s very nice! I love what you wrote.
I think we can make friends.
Good post, thanks for sharing, I enjoy it very much! There is a site: http://www.cheapinsanitydvds.com/
Your post is awesome, but why not take a look at our site: http://www.lovemypursemall.com
NFL jerseys wholesale
http://www.jerseycaptain.com/index.asp?showpage=productlist&cid=237
Cheap jerseys sale
http://www.jerseycaptain.com/index.asp?showpage=productlist&cid=269 with best quality
very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for http://www.ucoolstuff.com cool stuff , http://www.ucoolstuff.com cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you
Once we get there, so many wonderful dreams will come true and the pieces of our black nike air max shoes lives will fit together like a completed jigsaw puzzle. http://www.sellnikeairmax.com/
Once we get there, so many wonderful dreams will come true and the pieces of our black nike air max shoes lives will fit together like a completed jigsaw puzzle. http://www.sellnikeairmax.com/
Many thanks for sharing this, very useful!
Nice GHD, and you can get what you paid. GHD hair straightener
is really good!http://www.ghdhairsonsale.com
ugg boots
http://www.buylacosteclothing.com/lacoste-shoes-lacoste-observe-shoes-c-14_26.html
Great! I really love how it is easy on my eyes and also the data are well written. I am wondering how I could be notified whenever a new post has been made. I have subscribed to your rss feed which must do the trick! Have a nice day!
I cherished what you have got performed right here. The style is stylish, your penned articles elegant. Nonetheless, you might have bought an edginess to what that you are delivering the following.
http://www.buylacosteclothing.com/lacoste-shoes-lacoste-protect-lase-shoes-c-14_30.html
http://www.pumaferraris.com/men-s-speed-cat-big-c-47
I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!
http://www.pumaferraris.com/women-s-puma-basket-brights-c-31
Sick unquestionably arrive back once more for very much way more in situation you protect this up. Dont wipe out hope if not as well a lot of men and women see your perspective, know you may have acquired a fan suitable the following who values what you’ve obtained to say along together with the way you’ve introduced yourself. Excellent on you!
Advantageously, the post is really the freshest on this valuable topic. I concur with your conclusions and will thirstily look forward to your next updates. Just saying thanks will not just be adequate, for the exceptional clarity in your writing. I will right away grab your rss feed to stay abreast of any updates. Genuine work and much success in your business enterprize!
I can see that you might be an expert at your field! I am launching a web site soon, and your info will be incredibly useful for me..!.! Thank you for all your help and wishing you all the success in your business.
http://www.buylacosteclothing.com/lacoste-shoes-lacoste-protect-lase-shoes-c-14_30.html
http://www.buylacosteclothing.com/lacoste-shoes-c-14.html
Only want to say your article is brilliant. The lucidity in your post is simply impressive and i can assume you are an expert on this field. Well with your permission allow me to grab your rss feed to keep up to date with incoming post. Thanks a million and please keep up the a uthentic work.
I can see that you might be an expert at your field! I am launching a web site soon, and your info will be incredibly useful for me..!.! Thank you for all your help and wishing you all the success in your business.
Just saying thanks will not just be adequate, for the exceptional clarity in your writing. I will right away grab your rss feed to stay abreast of any updates. Genuine work and much success in your business enterprize!
Can I just say what a elimination toward come across a big name who essentially knows what theyre talking regarding on the world wide web.
I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!
Can I just say what a elimination toward come across a big name who essentially knows what theyre talking regarding on the world wide web.
I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business endeavors!
Well with your permission allow me to grab your rss feed to keep up to date with future post. Thanks a million and please keep up the fabulous work.