I’ve been thinking a lot about low end storage tonight. I mean ultra-low end. It comes to mind for 2 reasons. First, I’m really sick of the “whitebox linux mafia”, as I call them, bashing me over the head for prefering enterprise storage solutions. These are the folks that say NetApp or Sun sucks because they can build a 20TB storage box for the price of a ham sandwich and they generally talk about companies like HDS and EMC in the same way that people discuss the US Government on Coast-to-Coast. Secondly, I’m wondering what someone who really needs to milk every last penny out of their solution, such as a hobbiest, small organization, church, etc, is to do when considering storage.
And so, to figure out just how cheap we can get with our storage we first need to figure out what the cost of disks alone would be. To do that I gathered the price range of disks from NewEgg for 400G, 500G, and 750G drives. The range is from the least expensive disk to the most expensive disk in each capacity:
- 400GB Low: Western Digital Caviar SE16 WD4000KD 400GB 7200RPM: $140
- 400GB High: Seagate Barracuda 7200.10 ST3400620AS 400GB 7200RPM: $205
- 500GB Low: Western Digital Caviar SE16 WD5000KS 500GB 7200RPM: $195
- 500GB High: Maxtor MaXLine Pro 500 7H500F0 500GB 7200RPM: $310
- 750GB (Only one drive): Seagate Barracuda 7200.10 ST3750640AS 750GB 7200RPM: $400
The ranges consisted of 10 400GB drives, 11 500GB drives, and only 2 750GB drives, but the 750GB drives were both the same disk just in both PATA and SATA versions. These drives were both PATA and SATA.
So thats our price range. When we then take that range and break it down by cost-per-gig we see this:
- 400GB Low: .35/gig
- 400GB Hi: .51/gig
- 500GB Low: .39/gig
- 500GB Hi: .62/gig
- 750GB: .53/gig
And so that got me wondering how that cost compares to some enterprise-grade midrange storage solutions, and so I worked the following numbers based on the cost found online (actual cost will be lower through a VAR):
- Sun 3511 SATA Array: $18,495.00 for 1250GB: $14.80/gig
- Sun 3511 SATA Array: $36,995.00 for 6000GB: $6.16/gig
- Sun X4500 SATA: $32,995.00 for 12TB: $2.74/gig
- Sun X4500 SATA: $69,995.00 for 24TB: $2.91/gig
- Apple Xserve RAID: $5,999.00 for 1TB: $6.00/gig
- Apple Xserve RAID: $8,499.00 for 3.5TB: $2.43/gig
- Apple Xserve RAID: $12,999.00 for 7TB: $1.86/gig
- StoreVault S500: $7,758 for 2TB: $3.88/gig
- StoreVault S500: $12,000 for 4TB: $3.00/gig
And so clearly in these cases (all the solutions above are SATA, or in the case of Xraid, PATA) you can consider the cost per-gig above the cost of the drive as what we’re paying for the technology behind the solution. The feature premium we’ll call it.
So then, our goal is to create our own whitebox (build-it-yourself) storage solution that lowers the feature premium as much as possible.
I started by looking for a rack-mount chassis that could accomidate a large number of disks. This proved harder than I expected because most of these chasis are ugly as sh*t (yes, style is important) or I just couldn’t find pricing information. As I was browsing around I found something that really excited me: The V-Storm Series from iStar. It sexy and has some kick ass features. The V2-M8 in particular excites me, boasting 8 3.5″ hot-swap slots in 2U, plus thin-line DVD drive, plus an additional internal 3.5″ slot and an option for a 500W redundant power supply. A damned nice case. In fact, the V Series line ranges from 4 disks in 1.3U up to 40 disks in 8U, with everything inbetween. Wow! I just wish I could find pricing.
And so this gets me thinking… suppose I could get one of these cases cheap and build my own little mini-Thumper. The most SATA heads I’ve seen on a motherboard is 8, so we’ll pretend that we’re buying a 2U 8 disk enclosure.. how much are those disks alone going to cost? So I run the numbers for each capacity range and the figure out what the usable capacity will be for both RAID5 (or RAIDZ) and RAID6 (Dual Parity RAID5/RAIDZ):
RAW RAID5 RAID6 Cost
8x 400GB= 3.2TB 2.8TB 2.4TB $1,120
8x 500GB= 4TB 3.5TB 3TB $1,552
8x 750GB= 6TB 5.2TB 4.5TB $3,200
And so based on this cost, I’m going to consider that 500GB drives make for the “sweet spot”. So for about 3.5TB RAID5, I’m looking at spending around $1,600.
At this point I run into the “barebones server” market. This market is dominated by Tyan and SuperMicro. Barebones typically meaning case, power supply, and motherboard. And so I find an interesting offering from Tyan, the Tyan Transport TA26. This box offers 8 SATA/SAS hot-swap disks, power supply (redundant power supply model for about $300 more), and 2 socket Opteron motherboard in 2U for around $1,200. This setup requires that you buy a Tyan M9000-10 SO-DIMM, which sets you back another $100. And so, with this box we could have our 3.5TB usable for $2,900, leaving just procs and memory to buy.
And so, how low can we go? $3,500 is looks pretty doable for 3.5TB of RAID5 storage, which puts at a buck a gig, or a “feature premium” of 61 cents.
But… what other options are there?
An interesting option is the External SATA (eSATA) arrays from Norco Technologies. On NewEgg I found the NORCO DS-1220 3U 12-bay Hot-swap Rackmount eSATA RAID Hard Drive Storage Subsystem for $849. Nocro also has some interesting Firewire Array offerings.
And even still, another way to go is with a really realy low end offering thats a little more similar to what we expect in the enterprise range. The Promise VTrak line of hardware RAID subsystems for instance. The VTrak is available in Fibre Channel, SCSI, and iSCSI versions… but its the iSCSI that interests me. The Promise VTrak M200i is an 8 disk iSCSI array with redundant hot-swap this and that, for $3,400 (no drives). if we put in our cheap 500GB SATA drives that comes to $4,952. Thats pretty dirt cheap. Other models are available to accomidate more drives, such as the M500i which, with 15 500GB drives, would give you 7.5TB raw for $7,325.
Performance? Well… thats something that is hard to tell looking at these specs alone. But, I look at it like this: if your doing a search like this your obviously concerned more about capacity and price than performance. In a setup like these above if I got 40MB/s I’d consider myself really happy.
Clearly there is more digging and research to be done, but what I’m seeing here is that while its definately possible to get pretty decent storage capacities into the $1/GB range, its pretty difficult to go below it. Considering the fabeled Coraid solution, the SR1520 EtherDrive has 15 slots can costs $3,995, which means that if we use the same 500GB cheapo’s we’ve been using for our comparisons above the total cost is $6,920 for 7.5TB, or 91 cents a gig, but with some important tradeoffs to consider.
But… I want to emphasis that this dirt cheap storage is made possible by one thing in particular: OpenSolaris. Thanks to the work being done in OpenSolaris, such as the iSCSI Target Project, iSNS Server Project, ZFS, and everything else Nevada has to offer, decent solutions on less-than-decent gear are possible.
Nice work! I to occasionaly like to to swim in the low-end, especially after seeing how much we pay EMC fo maintenance.
Thanks for the link to iStar. Looks interesting.
Infortrend has some good looking low-end FC SAN boxes. I understand Sun rebadges a unit or two, but I don’t know which (I’m not a big Sun follower.)
How much of this is possible with Linux (say RH 4.0/2.6)
Sri: Linux has advantage in really only one important way: better driver support for hardware raid (pci) cards. LVM/EVMS/Linux RAID are difficult to manage and personally I don’t trust them. VxVM is available for Linux but its day is dying. And so if your using Linux for your storage server you definately want to have a hardware RAID card… under ZFS you don’t need it, and in some respects, don’t want it (software Dual Parity RAID5, vs hardware RAID5). That gap will close in the future thanks to the acceptance of OpenSolaris but it’ll be alittle while.
Filesystems like XFS and JFS are very good, but don’t provide the features that you get in ZFS, such as snapshotting, cloning, replication, etc.
The otherside of the coin is iSCSI, which I won’t get into.
If you want to compete against OpenSolaris and ZFS with Linux, you can do it, but its gonna cost you money for commercial software. Besides just the cost, its not all nicely integrated like you’d find in OpenSolaris.
I called iStar about the V Storm cases… the V2-M8 case won’t be available untill Sept, and they won’t give me pricing on it untill then. The V3-M16 (3U, 16 Drives) with a 600W Power Supply will set you back $1,099. I haven’t gotten details on the SATA backplane yet.
Would it be possible to have a barebones image of OpenSolaris bootable through PXE? By that I mean a configured operating system, not just the installer. Then you could just have it mount whatever hard disks are installed in the system.
Surprised nobody mentioned Apple X-Serve RAID. I personally don’t have experience with it, but it seems to be working well for many people. I will NEVER AGAIN touch cheapo SuperMicro-type enclosures without embedded management, I have wasted too many late nights on those.
The X-serve has builtin web-based management and notifications, multiple 2GB FC interfaces, and comes in under $2/GB. It’s cheaper than buying new SCSI disks for your old E450. Next time my boss asks me what kind of SAN we can get for less than $15K I’ll ask to test drive one of these.
Benr: you can compete against ZFS using Linux. Check out Lustre from ClusterFS. It scales to much larger and faster systems than ZFS will ever hope to achieve, mostly because ZFS is an instance of a giant file server exported over NFS, whereas Lustre is an actual distributed filesystem. Lustre lacks some handy management features of other systems, but it comes with big time performance.
I just ordered two 320G WD drives from NewEgg for my Linux box for a total of $189. It currently has two 120G drives configured as RAID1. I may need to remove one of them since I won’t have enough drive bays & power for 4 drives. I’ll keep the 120G as / and move /home to the two 320s configured as RAID1.
Thanks (as always) Ben for doing a wonderful job in researching/presenting various topics. Thanks also for responding to my earlier question.
benr: What do you think of the stuff that Adaptec sells, ie. the Snap Server range and SANbloc gear?
http://www.snapsolutions.com
I have no experience with them as yet, but am keen to get my hands on an eval of the new range units.
They definitely seem to be cheap enough and run GuardianOS.
Hello, this is my first post, hope not to write silly things!
These days I made the same kind of research for low price storage: I need a storage unit for a 2cpu Sun Fire X2200 M2 that I’m planning to buy (it will be used for numerical works for my thesis in astrophysics). Here follow some thoughts:
First of all, on new Sun amd64 boxes with PciE, external SCSI-to-SATA RAID arrays seem to be ruled out, at least with Solaris 10: there are no “Solaris ready” PciE SCSI hba in the Solaris 10 HCL (I know the Atto UL5D is supported by Linux, but maybe it won’t fit in 1U servers with low profile PciE slots?). Hope this will change in near future…
Also looked for an external (rackmount) box filled with SATA drives, and the drives connected to a SATA RAID card inside the server. The Norco case is very interesting, but it uses “multi lane” eSata connectors instead of faster Infiniband ones… And you’ll need a SATA RAID card with “external” connector: again no PciE cards with external connectors…
Then I found an interesting Linux distro called OpenFiler (a NAS system with a web magement console) that should supports iSCSI software targets, and network link aggregation . Maybe this kind of solution could provide a bit more from the famous 40MB/s that should be expected from a NAS with a GigE ? Anyone tried this?
Other nice rackmount chassis: from Chenbro http://www.chenbro.com .See the very beautiful RM32100 , it’s a 3U chassis with 16 3.5″ hot swap trays (SATA or SCSI); it should be great to build DAS arrays. Check also RM31212B, a 3U chassis with 12 hot swap trays, plus optical and floppy drives; seen a lot of Chenbro chassis on ebay, with interesting prices.
A final comment for a “do it yourself low price storage”.
You can find on ebay and other online stores the “GATEWAY 840 WAHOO SCSI RAID CONTROLLER” which is a self contained SCSI-to-SATA RAID controller, for less than 100$. The only problem is that this kind of module is connected to SATA disks with a big backplane connector… and you don’t have the schematics of course! But this could be a great project for electronics hobbist
Many thanks for your great site.
Cheers,
Emiliano
So, I have tried out that Norco beast with 500GB drives & zfs and so far I must have something wrong somewhere, cause performance is horrid, less than 1MB/s in most cases of write
Check this out:
http://www.dvwarehouse.com/R8-XP-p-32374.html
The Atto UL5D uses a LSI chipset that works with the Solaris mpt driver if you add the appropriate device alias by running the following command:
update_drv -a -i “pci117c,30″ mpt
There is a bug open to add Solaris support for the UL5D.
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6672889
shop zithromax welcome
ultram
earshot
stromkern
If you do not wish to receive similar messages please inform us on it by mail ban.site[dog]gmail.com
http://index1.arolew.com rel=”nofollow”>helps international dallas http://index2.arolew.com rel=”nofollow”>schoolbags with sharks http://index3.arolew.com rel=”nofollow”>talladega newspaper amanda burrell http://index4.arolew.com rel=”nofollow”>dden http://index5.arolew.com rel=”nofollow”>screen printed t shirt
http://index1.dupolk.com rel=”nofollow”>clinique song http://index3.dupolk.com rel=”nofollow”>pine wreaths http://index2.dupolk.com rel=”nofollow”>bermuda five star hotels http://index5.dupolk.com rel=”nofollow”>sicilia doro restaurant http://index4.dupolk.com rel=”nofollow”>college boys pissing
http://index1.dwerts.com rel=”nofollow”>open alcoholic beverage las vegas http://index2.dwerts.com rel=”nofollow”>mcdonnell douglas sp80 http://index3.dwerts.com rel=”nofollow”>aussie bites cookie recipe http://index4.dwerts.com rel=”nofollow”>connecticut huskies cheerleaders http://index5.dwerts.com rel=”nofollow”>washington post and express
http://index2.terewf.com rel=”nofollow”>nascarraces http://index1.terewf.com rel=”nofollow”>nashville indiana little nashville opry http://index5.terewf.com rel=”nofollow”>wicker basket with pedestal http://index4.terewf.com rel=”nofollow”>southern living receips http://index3.terewf.com rel=”nofollow”>reader for bernard telsey
http://index1.sabrob.com rel=”nofollow”>racing bellhousing http://index2.sabrob.com rel=”nofollow”>professional writing services for cover letters http://index3.sabrob.com rel=”nofollow”>neptune municipal http://index4.sabrob.com rel=”nofollow”>va hospital in gainesville fl http://index5.sabrob.com rel=”nofollow”>i needs a love lyrics
http://index3.naloga.com rel=”nofollow”>1965 philly 76ers http://index2.naloga.com rel=”nofollow”>w. h. plummer http://index4.naloga.com rel=”nofollow”>black wicked lady http://index5.naloga.com rel=”nofollow”>sheraton hotel anaheim http://index1.naloga.com rel=”nofollow”>cooper klipper lawn mowerscompany
Hi there!
My first post at this great blog!
I wanna show u my dayly updated blog: http://www.megautbe.cn rel=”nofollow”>Black Amateur Fuck Video
Have a nice day!
BB!
P.S. if you don’t want to see this message please write me to no.ads08@gmail.com with subject “NO ADS” and URL of your forum
Thank you for cooperation!
Very interesting article: “Cheap Ass Storage: How Low Can You Go”
Thank you very much!
Great post! Hope to be better. Better means more features.
good post,I think so!
links of london links of london charm
When people are stuck with essay thesis writing, thence I would recommend to buy essay writing at some professional paper writing service in such situation.
P90x .It really is not expensive if you factor in the cost
of a gym membership,P90x workout . The cost for P90X is
about three months of a paid gym membership but you get to
keep the program foreverP90x . You can try many of the
online sites, but it will be the same as buying from the
company or a Beachbody Coach. Make sure you are getting
original DVD’s. People are selling copies all over. The
problem is how long will they last, P90x workout ,and you
truly need the exercise and nutrition guide to even follow
the program. You can go to any site or you can go to
http://www.p90xmall.com and click on products. P90x dvd You can
order directly from the site,P90x dvd.
hp nx6100 battery http://www.adapterlist.com/hp/nx6100.htm
Apple a1185 black Battery http://www.globallaptopbattery.co.uk/apple/a1185-black.htm
All people deserve good life time and personal loans or just term loan will make it better. Just because people’s freedom bases on money state.
http://www.mbtshoeslatest.com
http://www.nikeairmaxshoe.com
excellent article , I added you to my http://www.china-wholesale-directory.com Top China Wholesalers category.. thanks for sharing the article!
very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for http://www.ucoolstuff.com cool stuff http://www.ucoolstuff.com cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you
i really appreciate what you’re doing here. very interesting site.
http://www.timberlandbootsale.co.uk
you have a very talented and skilled writting. i had a great time reading your comments.
http://www.timberlandboots4sale.co.uk
00zoe
MBT shoes http://www.mbtshoes4sale.co.uk
Timberland boots http://www.timberlandbootsale.co.uk
Hello,
This really doesn’t seem like a good idea to me at all. The server cabinet comes in all manner of shapes and sizes and represents a considerable range of products. Thanks a lot…
Moving Home? At Smart Storage Boxes we offer a wide selection of moving boxes, packs, bubble wraps, foam protectors, loose fills,tissue papers with tapes and markers to meet your requirements. For details log on to http://smartstorageboxes.co.uk