The Blog of Ben Rockwood

Cheap Ass Storage: How Low Can You Go

28 Aug '06 - 04:11 by benr

I've been thinking a lot about low end storage tonight. I mean ultra-low end. It comes to mind for 2 reasons. First, I'm really sick of the "whitebox linux mafia", as I call them, bashing me over the head for prefering enterprise storage solutions. These are the folks that say NetApp or Sun sucks because they can build a 20TB storage box for the price of a ham sandwich and they generally talk about companies like HDS and EMC in the same way that people discuss the US Government on Coast-to-Coast. Secondly, I'm wondering what someone who really needs to milk every last penny out of their solution, such as a hobbiest, small organization, church, etc, is to do when considering storage.

And so, to figure out just how cheap we can get with our storage we first need to figure out what the cost of disks alone would be. To do that I gathered the price range of disks from NewEgg for 400G, 500G, and 750G drives. The range is from the least expensive disk to the most expensive disk in each capacity:

400GB Low: Western Digital Caviar SE16 WD4000KD 400GB 7200RPM: $140
400GB High: Seagate Barracuda 7200.10 ST3400620AS 400GB 7200RPM: $205
500GB Low: Western Digital Caviar SE16 WD5000KS 500GB 7200RPM: $195
500GB High: Maxtor MaXLine Pro 500 7H500F0 500GB 7200RPM: $310
750GB (Only one drive): Seagate Barracuda 7200.10 ST3750640AS 750GB 7200RPM: $400

The ranges consisted of 10 400GB drives, 11 500GB drives, and only 2 750GB drives, but the 750GB drives were both the same disk just in both PATA and SATA versions. These drives were both PATA and SATA.

So thats our price range. When we then take that range and break it down by cost-per-gig we see this:

400GB Low: .35/gig
400GB Hi: .51/gig
500GB Low: .39/gig
500GB Hi: .62/gig
750GB: .53/gig

And so that got me wondering how that cost compares to some enterprise-grade midrange storage solutions, and so I worked the following numbers based on the cost found online (actual cost will be lower through a VAR):

Sun 3511 SATA Array: $18,495.00 for 1250GB: $14.80/gig
Sun 3511 SATA Array: $36,995.00 for 6000GB: $6.16/gig
Sun X4500 SATA: $32,995.00 for 12TB: $2.74/gig
Sun X4500 SATA: $69,995.00 for 24TB: $2.91/gig
Apple Xserve RAID: $5,999.00 for 1TB: $6.00/gig
Apple Xserve RAID: $8,499.00 for 3.5TB: $2.43/gig
Apple Xserve RAID: $12,999.00 for 7TB: $1.86/gig
StoreVault S500: $7,758 for 2TB: $3.88/gig
StoreVault S500: $12,000 for 4TB: $3.00/gig

And so clearly in these cases (all the solutions above are SATA, or in the case of Xraid, PATA) you can consider the cost per-gig above the cost of the drive as what we're paying for the technology behind the solution. The feature premium we'll call it.

So then, our goal is to create our own whitebox (build-it-yourself) storage solution that lowers the feature premium as much as possible.

I started by looking for a rack-mount chassis that could accomidate a large number of disks. This proved harder than I expected because most of these chasis are ugly as sh*t (yes, style is important) or I just couldn't find pricing information. As I was browsing around I found something that really excited me: The V-Storm Series from iStar. It sexy and has some kick ass features. The V2-M8 in particular excites me, boasting 8 3.5" hot-swap slots in 2U, plus thin-line DVD drive, plus an additional internal 3.5" slot and an option for a 500W redundant power supply. A damned nice case. In fact, the V Series line ranges from 4 disks in 1.3U up to 40 disks in 8U, with everything inbetween. Wow! I just wish I could find pricing.

And so this gets me thinking... suppose I could get one of these cases cheap and build my own little mini-Thumper. The most SATA heads I've seen on a motherboard is 8, so we'll pretend that we're buying a 2U 8 disk enclosure.. how much are those disks alone going to cost? So I run the numbers for each capacity range and the figure out what the usable capacity will be for both RAID5 (or RAIDZ) and RAID6 (Dual Parity RAID5/RAIDZ):

                        RAW     RAID5   RAID6           Cost
        8x 400GB=       3.2TB   2.8TB   2.4TB           $1,120
        8x 500GB=       4TB     3.5TB   3TB             $1,552  
        8x 750GB=       6TB     5.2TB   4.5TB           $3,200

And so based on this cost, I'm going to consider that 500GB drives make for the "sweet spot". So for about 3.5TB RAID5, I'm looking at spending around $1,600.

At this point I run into the "barebones server" market. This market is dominated by Tyan and SuperMicro. Barebones typically meaning case, power supply, and motherboard. And so I find an interesting offering from Tyan, the Tyan Transport TA26. This box offers 8 SATA/SAS hot-swap disks, power supply (redundant power supply model for about $300 more), and 2 socket Opteron motherboard in 2U for around $1,200. This setup requires that you buy a Tyan M9000-10 SO-DIMM, which sets you back another $100. And so, with this box we could have our 3.5TB usable for $2,900, leaving just procs and memory to buy.

And so, how low can we go? $3,500 is looks pretty doable for 3.5TB of RAID5 storage, which puts at a buck a gig, or a "feature premium" of 61 cents.

But... what other options are there?

An interesting option is the External SATA (eSATA) arrays from Norco Technologies. On NewEgg I found the NORCO DS-1220 3U 12-bay Hot-swap Rackmount eSATA RAID Hard Drive Storage Subsystem for $849. Nocro also has some interesting Firewire Array offerings.

And even still, another way to go is with a really realy low end offering thats a little more similar to what we expect in the enterprise range. The Promise VTrak line of hardware RAID subsystems for instance. The VTrak is available in Fibre Channel, SCSI, and iSCSI versions... but its the iSCSI that interests me. The Promise VTrak M200i is an 8 disk iSCSI array with redundant hot-swap this and that, for $3,400 (no drives). if we put in our cheap 500GB SATA drives that comes to $4,952. Thats pretty dirt cheap. Other models are available to accomidate more drives, such as the M500i which, with 15 500GB drives, would give you 7.5TB raw for $7,325.

Performance? Well... thats something that is hard to tell looking at these specs alone. But, I look at it like this: if your doing a search like this your obviously concerned more about capacity and price than performance. In a setup like these above if I got 40MB/s I'd consider myself really happy.

Clearly there is more digging and research to be done, but what I'm seeing here is that while its definately possible to get pretty decent storage capacities into the $1/GB range, its pretty difficult to go below it. Considering the fabeled Coraid solution, the SR1520 EtherDrive has 15 slots can costs $3,995, which means that if we use the same 500GB cheapo's we've been using for our comparisons above the total cost is $6,920 for 7.5TB, or 91 cents a gig, but with some important tradeoffs to consider.

But... I want to emphasis that this dirt cheap storage is made possible by one thing in particular: OpenSolaris. Thanks to the work being done in OpenSolaris, such as the iSCSI Target Project, iSNS Server Project, ZFS, and everything else Nevada has to offer, decent solutions on less-than-decent gear are possible.

- - C O M M E N T S - -
Nice work! I to occasionaly like to to swim in the low-end, especially after seeing how much we pay EMC fo maintenance. :(

Thanks for the link to iStar. Looks interesting.

Infortrend has some good looking low-end FC SAN boxes. I understand Sun rebadges a unit or two, but I don’t know which (I’m not a big Sun follower.)

jenkins - 28 August '06 - 14:01

How much of this is possible with Linux (say RH 4.0/2.6)

sri - 28 August '06 - 15:24

Sri: Linux has advantage in really only one important way: better driver support for hardware raid (pci) cards. LVM/EVMS/Linux RAID are difficult to manage and personally I don’t trust them. VxVM is available for Linux but its day is dying. And so if your using Linux for your storage server you definately want to have a hardware RAID card… under ZFS you don’t need it, and in some respects, don’t want it (software Dual Parity RAID5, vs hardware RAID5). That gap will close in the future thanks to the acceptance of OpenSolaris but it’ll be alittle while.

Filesystems like XFS and JFS are very good, but don’t provide the features that you get in ZFS, such as snapshotting, cloning, replication, etc.

The otherside of the coin is iSCSI, which I won’t get into.

If you want to compete against OpenSolaris and ZFS with Linux, you can do it, but its gonna cost you money for commercial software. Besides just the cost, its not all nicely integrated like you’d find in OpenSolaris.

benr - 28 August '06 - 17:51

I called iStar about the V Storm cases… the V2-M8 case won’t be available untill Sept, and they won’t give me pricing on it untill then. The V3-M16 (3U, 16 Drives) with a 600W Power Supply will set you back $1,099. I haven’t gotten details on the SATA backplane yet.

benr - 28 August '06 - 17:53

Would it be possible to have a barebones image of OpenSolaris bootable through PXE? By that I mean a configured operating system, not just the installer. Then you could just have it mount whatever hard disks are installed in the system.

arbnpx - 28 August '06 - 19:50

Surprised nobody mentioned Apple X-Serve RAID. I personally don’t have experience with it, but it seems to be working well for many people. I will NEVER AGAIN touch cheapo SuperMicro-type enclosures without embedded management, I have wasted too many late nights on those.

The X-serve has builtin web-based management and notifications, multiple 2GB FC interfaces, and comes in under $2/GB. It’s cheaper than buying new SCSI disks for your old E450. Next time my boss asks me what kind of SAN we can get for less than $15K I’ll ask to test drive one of these.

wsanders (URL) - 29 August '06 - 18:39

Benr: you can compete against ZFS using Linux. Check out Lustre from ClusterFS. It scales to much larger and faster systems than ZFS will ever hope to achieve, mostly because ZFS is an instance of a giant file server exported over NFS, whereas Lustre is an actual distributed filesystem. Lustre lacks some handy management features of other systems, but it comes with big time performance.

jwb (Email) - 29 August '06 - 21:17

I just ordered two 320G WD drives from NewEgg for my Linux box for a total of $189. It currently has two 120G drives configured as RAID1. I may need to remove one of them since I won’t have enough drive bays & power for 4 drives. I’ll keep the 120G as / and move /home to the two 320s configured as RAID1.

MikeC (Email) (URL) - 29 August '06 - 21:55

Thanks (as always) Ben for doing a wonderful job in researching/presenting various topics. Thanks also for responding to my earlier question.

sri - 30 August '06 - 20:40

benr: What do you think of the stuff that Adaptec sells, ie. the Snap Server range and SANbloc gear?

www.snapsolutions.com

I have no experience with them as yet, but am keen to get my hands on an eval of the new range units.
They definitely seem to be cheap enough and run GuardianOS.

elixxir - 31 August '06 - 07:40

Hello, this is my first post, hope not to write silly things!

These days I made the same kind of research for low price storage: I need a storage unit for a 2cpu Sun Fire X2200 M2 that I’m planning to buy (it will be used for numerical works for my thesis in astrophysics). Here follow some thoughts:

First of all, on new Sun amd64 boxes with PciE, external SCSI-to-SATA RAID arrays seem to be ruled out, at least with Solaris 10: there are no “Solaris ready” PciE SCSI hba in the Solaris 10 HCL (I know the Atto UL5D is supported by Linux, but maybe it won’t fit in 1U servers with low profile PciE slots?). Hope this will change in near future…

Also looked for an external (rackmount) box filled with SATA drives, and the drives connected to a SATA RAID card inside the server. The Norco case is very interesting, but it uses “multi lane” eSata connectors instead of faster Infiniband ones… And you’ll need a SATA RAID card with “external” connector: again no PciE cards with external connectors…

Then I found an interesting Linux distro called OpenFiler (a NAS system with a web magement console) that should supports iSCSI software targets, and network link aggregation . Maybe this kind of solution could provide a bit more from the famous 40MB/s that should be expected from a NAS with a GigE ? Anyone tried this?

Other nice rackmount chassis: from Chenbro [[http://www.chenbro.com]] .See the very beautiful RM32100 , it’s a 3U chassis with 16 3.5” hot swap trays (SATA or SCSI); it should be great to build DAS arrays. Check also RM31212B, a 3U chassis with 12 hot swap trays, plus optical and floppy drives; seen a lot of Chenbro chassis on ebay, with interesting prices.

A final comment for a “do it yourself low price storage”.
You can find on ebay and other online stores the “GATEWAY 840 WAHOO SCSI RAID CONTROLLER” which is a self contained SCSI-to-SATA RAID controller, for less than 100$. The only problem is that this kind of module is connected to SATA disks with a big backplane connector… and you don’t have the schematics of course! But this could be a great project for electronics hobbist :-)

Many thanks for your great site.
Cheers,

Emiliano

emiliano (Email) - 03 September '06 - 22:16

So, I have tried out that Norco beast with 500GB drives & zfs and so far I must have something wrong somewhere, cause performance is horrid, less than 1MB/s in most cases of write

Stuka - 17 November '06 - 16:59

Check this out:
[[http://www.dvwarehouse.com/R8-XP-p-323..]]

ben g - 19 February '07 - 12:58

The Atto UL5D uses a LSI chipset that works with the Solaris mpt driver if you add the appropriate device alias by running the following command:

update_drv -a -i “pci117c,30” mpt

There is a bug open to add Solaris support for the UL5D.
[[http://bugs.opensolaris.org/bugdatabas..]]

Martin (Email) (URL) - 19 March '08 - 00:25

shop zithromax welcome

zithromax (Email) (URL) - 15 May '08 - 05:52

ultram

ultram (Email) (URL) - 18 May '08 - 23:35

earshot

earshot (Email) (URL) - 24 May '08 - 15:50

stromkern

stromkern (Email) (URL) - 25 May '08 - 19:57

If you do not wish to receive similar messages please inform us on it by mail ban.site[dog]gmail.com

Google (Email) (URL) - 07 July '08 - 11:03

helps international dallas schoolbags with sharks talladega newspaper amanda burrell dden screen printed t shirt
clinique song pine wreaths bermuda five star hotels sicilia doro restaurant college boys pissing

nanny_[!2] (Email) (URL) - 11 October '08 - 11:43

open alcoholic beverage las vegas mcdonnell douglas sp80 aussie bites cookie recipe connecticut huskies cheerleaders washington post and express
nascarraces nashville indiana little nashville opry wicker basket with pedestal southern living receips reader for bernard telsey

nanny_[!2] (Email) (URL) - 11 October '08 - 18:43

racing bellhousing professional writing services for cover letters neptune municipal va hospital in gainesville fl i needs a love lyrics
1965 philly 76ers w. h. plummer black wicked lady sheraton hotel anaheim cooper klipper lawn mowerscompany

nanny_[!2] (Email) (URL) - 12 October '08 - 21:19

Hi there!
My first post at this great blog!
I wanna show u my dayly updated blog: Black Amateur Fuck Video
Have a nice day!
BB!

P.S. if you don’t want to see this message please write me to no.ads08@gmail.com with subject “NO ADS” and URL of your forum
Thank you for cooperation!

KurmanAhlabm (Email) (URL) - 09 November '08 - 08:24

Very interesting article: “Cheap Ass Storage: How Low Can You Go”

Chris (Email) (URL) - 15 January '09 - 18:06

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 02:39

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 01:30

Great post! Hope to be better. Better means more features.
good post,I think so!

links of london (Email) (URL) - 17 November '09 - 04:54

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 07:03

links of london links of london charm

links of london (Email) (URL) - 12 January '10 - 05:37

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:41

When people are stuck with essay thesis writing, thence I would recommend to buy essay writing at some professional paper writing service in such situation.

KATIEKw21 (Email) - 13 February '10 - 12:19

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:35

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:55

A good idea! [[http://www.offerreplicawatches.com.The..]] sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:37

hp nx6100 battery [[http://www.adapterlist.com/hp/nx6100.h..]]

laptop battery (Email) (URL) - 14 May '10 - 09:33

Apple a1185 black Battery [[http://www.globallaptopbattery.co.uk/a..]]

laptop batteries (Email) (URL) - 14 May '10 - 09:49

All people deserve good life time and personal loans or just term loan will make it better. Just because people’s freedom bases on money state.

home loans (Email) (URL) - 13 June '10 - 19:36

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:45

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:19

The athletic shoes which makes using this technology may the very good local constable convoy mobilization body, Air Max 2009.
[[http://www.allhotshoes.com/]]

air max shoes (URL) - 09 July '10 - 05:26