ZFS Backup & Recovery using Hadoop HDFS

Posted on April 18, 2011

Hadoop HDFS has essentially become the de facto standard in cluster file systems.  I’m theory I’m a big fan of Lustre; I say “theory” because it never got ported to Solaris, despite the fact that Sun bought Lustre.  But thats a different story.  HDFS is extremely portable and well supported by a thriving community who are doing anything you can image with it.

Consider a large cluster of production nodes.  They almost certainly have unused disk space, and its probly pretty fast disk.  Wouldn’t it be nice if we could aggregate all that disk together for backups?  With HDFS we can.  Setting up HDFS is pretty well documented (I can do it for Solaris users if there is a demand, but its pretty clear), so see how easy it is to get ZFS backups in and out of HDFS.

Here is a DFS report from my HDFS test setup (single node):

root@newton hadoop$ hadoop dfsadmin -report
Safe mode is ON
Configured Capacity: 2942589468672 (2.68 TB)
Present Capacity: 1019145412608 (949.15 GB)
DFS Remaining: 1017809015808 (947.91 GB)
DFS Used: 1336396800 (1.24 GB)
DFS Used%: 0.13%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

Datanodes available: 1 (1 total, 0 dead)

Decommission Status : Normal
Configured Capacity: 2942589468672 (2.68 TB)
DFS Used: 1336396800 (1.24 GB)
Non DFS Used: 1923444056064 (1.75 TB)
DFS Remaining: 1017809015808(947.91 GB)
DFS Used%: 0.05%
DFS Remaining%: 34.59%
Last contact: Mon Apr 18 14:18:30 PDT 2011


So HDFS is ready to rock. Now lets create a ZFS dataset and populate it with some data:

root@newton ~$ zfs create -o mountpoint=/backup_test quadra/backup_test
root@newton ~$ cp *.pdf /backup_test/


Now, lets create a directory within HDFS to put our backups in:

root@newton ~$ hadoop fs -mkdir /zfs_backups
root@newton ~$ hadoop fs -ls /
Found 11 items
drwxrwxrwx   - root supergroup          0 2011-03-05 00:07 /hypertable
drwxr-xr-x   - root supergroup          0 2011-03-02 12:27 /system
drwxr-xr-x   - root supergroup          0 2011-04-18 14:22 /zfs_backups


Ready to rock. So lets actually do the backup. We’re going to create a snapshot and the zfs send it to the stdin of “hadoop fs -put”. Once we’ve done that, we’ll delete our origonal ZFS dataset:

root@newton ~$ zfs snapshot quadra/backup_test@051811
root@newton ~$ zfs send quadra/backup_test@051811 | hadoop \
> fs -put - /zfs_backups/backup_test.051811.zdump
root@newton ~$
root@newton ~$ zfs destroy -r quadra/backup_test
root@newton ~$ zfs list quadra/backup_test
cannot open 'quadra/backup_test': dataset does not exist


OK, so the ZFS snapshot has been stored as a file within HDFS and we destroyed our dataset. Now, lets recover it using the reverse proceedure:

root@newton ~$ hadoop fs -get /zfs_backups/backup_test.051811.zdump \
>  - | zfs recv -d quadra
root@newton ~$ zfs list -r quadra/backup_test
NAME                        USED  AVAIL  REFER  MOUNTPOINT
quadra/backup_test         32.3M   316G  32.3M  /quadra/backup_test
quadra/backup_test@051811      0      -  32.3M  -


Notice that we lost our properties during the receive, lets fix that and check that our files are back:

root@newton ~$ zfs set mountpoint=/backup_test quadra/backup_test
root@newton ~$ ls -l /backup_test/
total 33076
-rw-r--r-- 1 root root   330028 2011-04-18 14:20 Deployment_Guide_for_HP_ProLiant_Servers.pdf
-rw-r--r-- 1 root root    88378 2011-04-18 14:20 GeekBench-Receipt.pdf
-rw-r--r-- 1 root root   101243 2011-04-18 14:20 HP_ProLiant_Health_Monitor_User_Guide.pdf
-rw-r--r-- 1 root root    90844 2011-04-18 14:20 HP_ProLiant_Support_Pack_User_Guide_861.pdf
-rw-r--r-- 1 root root   123419 2011-04-18 14:20 inthebeginning.pdf
-rw-r--r-- 1 root root 21337122 2011-04-18 14:20 Jurans Quality Handbook.pdf
-rw-r--r-- 1 root root  1338119 2011-04-18 14:20 perc-technical-guidebook.pdf
-rw-r--r-- 1 root root  2401352 2011-04-18 14:20 PowerEdgeR510_Technical_Guidebook[1].pdf
-rw-r--r-- 1 root root  5923274 2011-04-18 14:20 R710-HOM.pdf
-rw-r--r-- 1 root root  1103929 2011-04-18 14:20 server-poweredge-r710-tech-guidebook.pdf
-rw-r--r-- 1 root root   504059 2011-04-18 14:20 TheQualityTrilogy.pdf


Please note that the reason the dataset properties were not retained was because I’m using an old ZPool (Version 18). If your running a newer pool version (check with “zpool get all pool_name”) the properties will go with the backup stream.

So there you go. Backup and recovery using ZFS send/recv to and from HDFS. Straightforward and easy to implement.