I’ve been thinking a lot about low end storage tonight. I mean ultra-low end. It comes to mind for 2 reasons. First, I’m really sick of the “whitebox linux mafia”, as I call them, bashing me over the head for prefering enterprise storage solutions. These are the folks that say NetApp or Sun sucks because they can build a 20TB storage box for the price of a ham sandwich and they generally talk about companies like HDS and EMC in the same way that people discuss the US Government on Coast-to-Coast. Secondly, I’m wondering what someone who really needs to milk every last penny out of their solution, such as a hobbiest, small organization, church, etc, is to do when considering storage.

And so, to figure out just how cheap we can get with our storage we first need to figure out what the cost of disks alone would be. To do that I gathered the price range of disks from NewEgg for 400G, 500G, and 750G drives. The range is from the least expensive disk to the most expensive disk in each capacity:

• 400GB Low: Western Digital Caviar SE16 WD4000KD 400GB 7200RPM: $140
• 400GB High: Seagate Barracuda 7200.10 ST3400620AS 400GB 7200RPM: $205
• 500GB Low: Western Digital Caviar SE16 WD5000KS 500GB 7200RPM: $195
• 500GB High: Maxtor MaXLine Pro 500 7H500F0 500GB 7200RPM: $310
• 750GB (Only one drive): Seagate Barracuda 7200.10 ST3750640AS 750GB 7200RPM: $400

The ranges consisted of 10 400GB drives, 11 500GB drives, and only 2 750GB drives, but the 750GB drives were both the same disk just in both PATA and SATA versions. These drives were both PATA and SATA.

So thats our price range. When we then take that range and break it down by cost-per-gig we see this:

• 400GB Low: .35/gig
• 400GB Hi: .51/gig
• 500GB Low: .39/gig
• 500GB Hi: .62/gig
• 750GB: .53/gig

And so that got me wondering how that cost compares to some enterprise-grade midrange storage solutions, and so I worked the following numbers based on the cost found online (actual cost will be lower through a VAR):

• Sun 3511 SATA Array: $18,495.00 for 1250GB: $14.80/gig
• Sun 3511 SATA Array: $36,995.00 for 6000GB: $6.16/gig
• Sun X4500 SATA: $32,995.00 for 12TB: $2.74/gig
• Sun X4500 SATA: $69,995.00 for 24TB: $2.91/gig
• Apple Xserve RAID: $5,999.00 for 1TB: $6.00/gig
• Apple Xserve RAID: $8,499.00 for 3.5TB: $2.43/gig
• Apple Xserve RAID: $12,999.00 for 7TB: $1.86/gig
• StoreVault S500: $7,758 for 2TB: $3.88/gig
• StoreVault S500: $12,000 for 4TB: $3.00/gig

And so clearly in these cases (all the solutions above are SATA, or in the case of Xraid, PATA) you can consider the cost per-gig above the cost of the drive as what we’re paying for the technology behind the solution. The feature premium we’ll call it.

So then, our goal is to create our own whitebox (build-it-yourself) storage solution that lowers the feature premium as much as possible.

I started by looking for a rack-mount chassis that could accomidate a large number of disks. This proved harder than I expected because most of these chasis are ugly as sh*t (yes, style is important) or I just couldn’t find pricing information. As I was browsing around I found something that really excited me: The V-Storm Series from iStar. It sexy and has some kick ass features. The V2-M8 in particular excites me, boasting 8 3.5″ hot-swap slots in 2U, plus thin-line DVD drive, plus an additional internal 3.5″ slot and an option for a 500W redundant power supply. A damned nice case. In fact, the V Series line ranges from 4 disks in 1.3U up to 40 disks in 8U, with everything inbetween. Wow! I just wish I could find pricing.

And so this gets me thinking… suppose I could get one of these cases cheap and build my own little mini-Thumper. The most SATA heads I’ve seen on a motherboard is 8, so we’ll pretend that we’re buying a 2U 8 disk enclosure.. how much are those disks alone going to cost? So I run the numbers for each capacity range and the figure out what the usable capacity will be for both RAID5 (or RAIDZ) and RAID6 (Dual Parity RAID5/RAIDZ):

                        RAW     RAID5   RAID6           Cost
        8x 400GB=       3.2TB   2.8TB   2.4TB           $1,120
        8x 500GB=       4TB     3.5TB   3TB             $1,552  
        8x 750GB=       6TB     5.2TB   4.5TB           $3,200

And so based on this cost, I’m going to consider that 500GB drives make for the “sweet spot”. So for about 3.5TB RAID5, I’m looking at spending around $1,600.

At this point I run into the “barebones server” market. This market is dominated by Tyan and SuperMicro. Barebones typically meaning case, power supply, and motherboard. And so I find an interesting offering from Tyan, the Tyan Transport TA26. This box offers 8 SATA/SAS hot-swap disks, power supply (redundant power supply model for about $300 more), and 2 socket Opteron motherboard in 2U for around $1,200. This setup requires that you buy a Tyan M9000-10 SO-DIMM, which sets you back another $100. And so, with this box we could have our 3.5TB usable for $2,900, leaving just procs and memory to buy.

And so, how low can we go? $3,500 is looks pretty doable for 3.5TB of RAID5 storage, which puts at a buck a gig, or a “feature premium” of 61 cents.

But… what other options are there?

An interesting option is the External SATA (eSATA) arrays from Norco Technologies. On NewEgg I found the NORCO DS-1220 3U 12-bay Hot-swap Rackmount eSATA RAID Hard Drive Storage Subsystem for $849. Nocro also has some interesting Firewire Array offerings.

And even still, another way to go is with a really realy low end offering thats a little more similar to what we expect in the enterprise range. The Promise VTrak line of hardware RAID subsystems for instance. The VTrak is available in Fibre Channel, SCSI, and iSCSI versions… but its the iSCSI that interests me. The Promise VTrak M200i is an 8 disk iSCSI array with redundant hot-swap this and that, for $3,400 (no drives). if we put in our cheap 500GB SATA drives that comes to $4,952. Thats pretty dirt cheap. Other models are available to accomidate more drives, such as the M500i which, with 15 500GB drives, would give you 7.5TB raw for $7,325.

Performance? Well… thats something that is hard to tell looking at these specs alone. But, I look at it like this: if your doing a search like this your obviously concerned more about capacity and price than performance. In a setup like these above if I got 40MB/s I’d consider myself really happy.

Clearly there is more digging and research to be done, but what I’m seeing here is that while its definately possible to get pretty decent storage capacities into the $1/GB range, its pretty difficult to go below it. Considering the fabeled Coraid solution, the SR1520 EtherDrive has 15 slots can costs $3,995, which means that if we use the same 500GB cheapo’s we’ve been using for our comparisons above the total cost is $6,920 for 7.5TB, or 91 cents a gig, but with some important tradeoffs to consider.

But… I want to emphasis that this dirt cheap storage is made possible by one thing in particular: OpenSolaris. Thanks to the work being done in OpenSolaris, such as the iSCSI Target Project, iSNS Server Project, ZFS, and everything else Nevada has to offer, decent solutions on less-than-decent gear are possible.