Guide: unRAID Server – Part 1 The Journey to unRAID

by Jim Metcalf on May 18, 2011 · 21 comments

in Guides

This is part one of a three part series of articles serving as a review and basic install walk-through for the storage server unRAID. It attempts to explain the tradeoffs and install details of the popular storage alternative to Windows Home Server (WHS). This guide is something like “I wish I had known this before I started.”


My home theater evolved in a piecemeal fashion and I found unRAID to be a simple and a low cost storage solution to my fragmented and burgeoning media collection. But simple ain’t easy and the tradeoffs might convince you that the new WHS is a better option. WHS and unRAID can also work in concert – many people use an old PC and unRAID as a backup for their WHS. This allows for performance gains and cost savings while having a complete backup of your media on a separate system.

I like to tell my wife that my reasons for ripping our DVDs were practical. Between a DVD player in the minivan and a hardwood floor under the family DVD player, the typical lifespan of a Blues Clues DVD was less than a month. An ISO backup of a new DVD for the kids was more an economic decision than one of convenience or media distribution. Unfortunately, the price of storing and streaming our entire DVD collection was still too expensive a few years ago. However, once the price of a 2TB consumer hard drive fell below $100, it became feasible to use an existing workstation as a storage server for my entire library of ISOs. I was also eager to cut the cord on our expensive cable providers and more storage was the key. Basically, I appealed to the hoary old chestnut: Honey, all of this stuff from Newegg is actually going to save us money.

IN THE BEGINNING, it was easy enough to shove hard drives into my beefy Windows workstation and create simple shares for movies. I played them over a wired network to an HTPC running Windows Media Center. This worked well until there was a proliferation of shares (Movies1,2,3,4) and I could no longer work in Photoshop while a movie was being served from my workstation. I had no data protection in the event that a hard drive failed and the thought of re-ripping and correcting metadata was daunting. The final reason for building a storage solution came with the purchase of a Dune Smart streamer and MyMovies software.

A single metadata aggregator and a single storage server were somewhat of a holy grail for me. After I installed and configured unRAID with MyMovies, I now run MyMovies on my main ripping workstation and my HTPC is a client for this metadata server. I also use MyMovies as the metadata aggregator for my Dune Smart H1. With unRAID, I have the freedom to try things like NFS for XBMC on Linux that were difficult on WHS.

A “Here Be Dragons” warning is in order because the subtle differences among storage technologies are argued ad nauseam among people (usually IT professionals) with strong opinions. I kept my focus on the cheapest way to store everything in a central location with some amount of data protection. My first thought was WHS, but my principle goal was achieving the biggest bang for buck. The problems with WHS mostly involved cost and the decision to abandon data duplication on the recent version. If I wanted data protection in WHS, I would have to go with the older version of WHS. Even if I decided to go with WHS for data protection, its mirroring technology meant that I would need twice the disk space for a given ISO or mkv. Even at low disk prices, these tradeoffs seemed too much for me.

Other products that I investigated also had tradeoffs between reliability, complexity, and cost. For example, hardware RAID is certainly faster than most software RAID, but it comes with vendor lock in. Many RAID solutions require same-sized disks that must be purchased at the beginning of the build. There are also other software RAID products (such as FlexRAID) that offer great data protection for free, but they don’t have a system for presenting a unified share of all my drives or the ability to off load work from my workstation. Still others are industrial strength solutions like ZFS and FreeNAS that can be tailored to the home server, but lack some of the ease of unRAID.

unRAID is basically a specialized Slackware (Linux) server that servers up Just a Bunch of Disks (JBOD) with a parity drive. This protects against a drive failure while allowing any sized disks (as long as the parity disk is the same size or larger). unRAID fills the disk with individual files rather than “striping” the data among disks. This means that in the event of multiple disk failures, the data on the surviving disks is recoverable.

unRAID hits my sweet spot for a pure media storage server for these reasons:

  • Modest hardware requirements
  • Unified shares with sophisticated data abstraction schemes
  • Able to easily serve three HD streams over gigabit network with green drives
  • Mature product with good support and documentation
  • Ability to use existing hard drives
  • Data protection (RAID)
  • Power efficiency by spinning down drives that aren’t in use
  • NFS server for Dunes and XBMC
  • Try before you buy with up to 3 (2 data + 1 parity) disks
  • Ability to grow the storage to 40TB usable with a mix of drive sizes
  • Ability to backup family workstations to an unRAID server and to a cloud vendor (CrashPlan)
  • Hardware flexibility – all components are easily replaced – even the motherboard

But this comes with some tradeoffs:

  • You should be comfortable with simple command line syntax for the setup
  • Patience is a requirement
  • Write performance is good, but not great (~30-35 MB/s) on green drives
  • Maximum of one parity drive
  • Not suited for critical data

unRAID can be bittersweet because you can achieve Rock Solid® and dead simple operation, but you have to spend some time understanding how to properly set it up. However, setting up unRAID is more a matter of patience than of complexity.

In part two of this series I’ll be discussing hardware requirements, preparation steps, and walking through the unRAID installation process.


Article by

Hello, I'm Jim. I reside in Athens Georgia with my wife and four kids. For a salary and on occasional late nights, I manage technologies which include VMWare, Active Directory, and NetApp. For fun and necessity, I build systems that deliver the largest bang for a given dollar which include unRaid, Dune streamers, and a movie screen made from a counter top.


{ 19 comments }

Damian May 18, 2011 at 7:52 am

Awesome writeup Jim, looking forward to part 2 and 3 of your writeup. I have definitely been interested in unRAID (among other non WHS solutions) but the learning curve has detracted me a little (and I am sure the same goes for others as well)

Jim May 18, 2011 at 12:05 pm

Thanks Damian.

It is far less complex than most of the guides that you have put together. I would say it’s about as complex as MyMovies and not nearly as complex as hacking a Dune or setting up MPC-HC with bitstreaming.

Damian May 18, 2011 at 12:06 pm

Yeah but most of my guides I make up :-)

Rick May 18, 2011 at 9:26 am

Great article. I came to the same conclusion a few months ago about what to do with media storage and now have about 6TB on my UNRAID while I run WHS2011 for PC backups, more daily use, etc. Still learning the ins and outs of tweaking UNRAID so looking forward to your next two articles to glean some further information of your experiences.

Luke Foust May 18, 2011 at 10:37 am

I am in the same boat. I have out grown my WHS machine and am looking to go with a longer term solution. Unraid is currently the path I am leaning towards so I look forward to reading more about your experiences.

Also, I would love to hear you expand on why you think unraid is “Not suited for critical data”

Thanks!

Jim May 18, 2011 at 11:27 am

Luke,

I should preface this by saying that any RAID system is not a substitute for backups (by definition, backups are offsite and is the reason I love CrashPlan).

The reason for the assertion is less related to any unRAID shortcomings, but the probability that you will encounter two Unrecoverable Errors on a large array (say 12) of consumer green drives. If you look at the published data on error rates for these drives, it becomes a matter of mathematics.* Error rates are based on the number of bits written, and as these drives reach higher densities, there is a significant (but small) chance that you will have errors on more than one disk. (I can 100% guarantee that your critical data will be on these disks.)

I keep critical data on mirrored and smaller enterprise drives with an offsite backup.

* A really good treatment of the subject can be found here:
http://dsp.ucsd.edu/~jfmurray/publications/Hughes2004.pdf

Ron May 18, 2011 at 3:22 pm

I moved to unRAID also and love it, though im also running a WHS 2011 server. Love both but for different reasons.
Moved from WHS v1 to unRAID about 5 months ago and it was pretty painless. Setup the server for future growth so its has 20 slots but i havent filled them all yet. But moving all my data from WHS v1 was pretty easy.
Started out with 4 HDDs, got unRAID up and running then one by one mounted each of my 4 WHS HDDs and coppied all the data to each unRAID HDD then created a Split Level 0 and pretty much had a duplicate of my WHS DriveExt.
Then just precleared the remaing HDDs, added them to the array and sorted out my files.
Of course being mostly 2TB drives this process took all week but couldnt have been any easier to transition.

Damian May 18, 2011 at 3:29 pm

Part of my concern with unRAID is we already have two commenters here saying they are running both unRAID and WHS. For me the whole purpose is two have just one server running, not multiple servers to carry out different roles. Backups seems to be the key area missing which is why WHS is still getting used in this case. I don’t know much about unRAID, I wonder if this is something the developers could look to implement?

Jim May 18, 2011 at 4:47 pm

Damian,

WHS has a tremendous feature set and unRAID will never be the clear winner in terms of functionality. It’s only because Windows fails utterly in the critical area of storage that is making users rethink WHS. Is this feature set worth that pain? I came to the conclusion that they did not. (And I’m writing this from TechEd and I generally love Microsoft products!) Most WHS features are simply “nice” or could be done better with different software given my requirements.

Your specific example of backups is a touchy subject. I would argue that WHS are not backups per se, rather, they are local copies with versioning. This is a great feature for WHS, however I use something like CrashPlan for critical data and occasionally back up system images to unRAID. This is less convenient, but more robust in my opinion. So that is a wash at best for me.

After speaking with some Microsoft guys at TechEd, I’m convinced that MS is pulling back from large media storage for the home. This is a representative quote from the Windows Home Server Blog:

“…But maybe it’s time to take a hard look at what data you’re hording on your server. Do you really need to be a digital hoarder and save everything?”

Can you ever imagine Microsoft saying this to a soon-to-be-former corporate customer?

Alex Kuretz May 18, 2011 at 10:21 pm

After speaking with some Microsoft guys at TechEd, I’m convinced that MS is pulling back from large media storage for the home. This is a representative quote from the Windows Home Server Blog:

“…But maybe it’s time to take a hard look at what data you’re hording on your server. Do you really need to be a digital hoarder and save everything?”

Can you ever imagine Microsoft saying this to a soon-to-be-former corporate customer?

Your comments are certainly accurate and representative of the opinion that Microsoft is communicating, but I would challenge you to consider if it is a valid opinion. The quote you are referencing is from the Small Business Server team that happens to have an ‘H’ in front of their name representing “Home” that they simply choose to ignore.

I am of the belief that large media storage in the home is a reality for many years to come, and I think you are completely justify my belief with this series of articles.

Jim May 19, 2011 at 5:20 am

Alex,

I was dumbfounded when the MS guys sat there telling me that supporting large storage for the home is just not a profitable segment and not worth investment. They feel that it is just too complex and expensive for the average user.

MS is completely enamored with streaming from the cloud and making Windows embedded the focus of WMC. I think that this is a completely invalid assumption. I am certainly not an expert in this market segment, but in my estimation, OEM developers at TechEd were more interested in creating devices (DLNA servers mostly) that would stream to Windows rather than creating more platforms that could play these streams.

jericko May 19, 2011 at 9:26 am

Jim

Great article, look forward to part 2 and 3. I am currently running a 16TB WHS v1 to stream all my BluRay rips to my various PopcornHours. While happy with WHS V1, V2 is not a option and I have been on the lookout for something to replace WHS v1 in the future.

JohnBick May 26, 2011 at 7:15 pm

First let me say I have NOT (yet) tried UNRAID and may, therefore, not understand some things relative to “parity” in this context. As I understand it there is one parity disk and it has to be the size of the largest disk. This alone implies to me that errors can be DETECTED but not CORRECTED. And you imply that, Jim, when you say that your other disks and the data on them will be intact. I have two concerns:

(1) Detection of an error is not correction. Parity, if this is what it is, is not sufficient to correct an error. It is, of course, best to know your data is corrupt but I want it recoverable. RAID does that (different ways depending on the RAID level) and WHS does it with duplication.

(2) By the time you have 20 2-TB drives in your system is there really room for the parity on that single drive?

As I say, I have not tried it and have only investigated it casually. But I have not yet seen answers to these two concerns that encourage me to go this way.

Would love to be corrected!

Jim Metcalf May 26, 2011 at 8:45 pm

John,

Congratulation, you are indeed a rare bird if you love to be corrected on the Internet ; )

You do ask a great question, because parity can become a complex issue, but it is based on a very simple idea.

Parity can be thought of as a simple addition operation that must always sum to even (or odd) number. Say you have 3 data drives that can store a single bit (0 or 1). Furthermore, in this system there is a parity drive that also stores a single bit. In our scheme, the data bits are written to the first three drives and then a parity bit is calculated such that the sum of ALL bits (3 data bits plus the parity bit) are equal to an even number.

To flesh it out a little more, say the values written to our data disk are {0,0,1}. The OS would do a quick operation and determine that the parity drive should have a 1 written in order to sum to an even amount.

0+0+1 (+1 written by the parity operation) = EVEN

If an error occurs in our system and the computer can’t read the data on drive #2, then we know it must have been 0. This is how the system recovers data. The parity doesn’t really discover or detect errors per se.

To answer to your numbered questions above:

1.) If unRAID detects an error, it corrects it. Parity (assuming it is healthy) IS sufficient to correct an error (as displayed above). In fact, the mechanism that unRAID uses to recover data is very similar to WHS.

Basically, in both WHS and unRAID if a read error occurs, it is reported to the Operating System. At this point, the parity (on unRAID) or the mirror (on WHS) is consulted for the information. The only difference is that unRAID consults multiple disks for this information, while WHS consults a single disk. (A better discussion of parity can be found here: http://en.wikipedia.org/wiki/Parity_bit)

2. “20 2-TB drives… room for the parity on that single drive?” There is plenty of room for parity on a single drive (assuming that the drive is as large as the largest data drive because the system must have enough room to store the value to make it even). Usually the parity disk is one of the first disks you install, however, you could certainly install the parity drive as your 21st drive if you go with the Pro version.

Jim Metcalf May 26, 2011 at 9:09 pm

John,

I would further add that unRAID calculates parity on JBOD (just filling a bunch of disks) rather than a traditional RAID scheme (say RAID 5) that stripes drives.

This comes with benefits and costs. One benefit is that you can recover at least some data from a two-disk failure. In RAID5, a two-disk failure generally means that you have just lost ALL of your data. Also, RAID5 is less energy efficient because you must spin all of the disks in order to write or read. unRAID only spins two disks on an arbitrary write, and one on a read.

The cost for this is speed. RAID5 can write or read many disks at once which can dramatically increase performance. However, you don’t need nearly that amount of performance when streaming media. In fact, reading from a sing green drive is enough to serve 3 HD streams.

System design is about finding the right trade-offs. For many reasons, unRAID is the perfect media storage system because its strengths are highly desired and its weakness are not relevant.

Alex October 25, 2011 at 8:57 pm

I guess this is a good enough article for those who have no idea about storage solutions. Or maybe it won’t be simple enough for them.

I looked around for a long time. I had RAID5, RAID6 and I had issues with them almost loosing my data. RAIDs were designed to be performant and highly available storage solutions. They are not what an average person needs at home. I find it extremely funny reading or hearing people talk about speed of RAID all while they have a network between their file server and client computer. It’s not like they have thousands of clients to warrant the need of the speed that some RAID configurations provide.

unRAID is alright, but it’s got it’s problems. Then you brush on FlexRAID and dismiss it, while I believe that this is the best solution for a home user, especially when one needs a way to store and protect a media library. I’ve settled on FlexRAID and I think it’s got huge advantages over other solutions. One of them being the fact that it works with any combination of drives/folders/files and doesn’t need specially formatted drives that become useless once your controller card is dead or your software mulfunctions. All you have is “normal” drives that can function independently. And even if you have 2 or 3 simultaneus drive failures you lose only those drives unlike RAIDs. Plus, FlexRaid now supports new levels of parity protection – you can have multiple parity drives thus increasing the level of protection. And, the last note – I doubt that many people need real-time parity calculation on their ‘file servers’ at home – the data there changes rarely and parity can be calculated on demand or on schedule, which will be sufficient for 99.999% of users, I would guess.

unRaid is unnecesserily complex and demanding. Plus, it’s not free.

Jim Metcalf October 26, 2011 at 7:09 am

Hi Alex,

Good to hear that FlexRAID has worked out for you and your needs.

I would not disagree with some criticism of unRaid, namely that it takes some amount of time to figure out. However, its complexity fades into the noise when compared to the time and effort needed to build an HTPC or XBMC rig that perfectly bitstreams every single HD format with MadVR and a low WAF, but I digress…

I disagree that I brushed off Flexraid. It did not meet two of my requirements:

1.) Off load tasks and management away from my desktop. With Flexraid, if you don’t have a dedicated box, you must share resources and management with an existing workstation. I do a lot of work on my existing workstation that served as my FlexRAID test box. FlexRAID must live in the same sandbox where I constantly tweak/install-uninstall software/update the OS/dual boot/change hw/update drivers/etc… When you combine this with power management, this was just too much of a burden for my requirements.

2.) UnRaid supports native NFS. With my primary Win7 box, a bolt-on NFS server needs to be purchased for my Dunes. Plus, my UnRAID box sips power unlike my workstation and I really have not touched it since setting it up.

I’d like to disagree with your assertion that unRaid requires “specially formatted drives that become useless once your controller card is dead or your software malfunctions.” I’m not sure what you mean by useless, but it certainly does not mean “unusable.” In fact, you can power down an unRAID server and mount any of its data drives in any other Linux box and have immediate access to all of you files without any effort. The same goes for a crashed system. Another nice feature is that in the event of a catastrophic hw failure, you can simply pull out the USB stick and the surviving drives and immediately boot the entire OS on any supported hardware. Try that with Windows ;).

In addition, “if you have 2 or 3 simultaneous drive failures you lose only those drives” is equally true for unRAID.

I’m not going to deny that FlexRAID can be a perfect solution for many users here, however, it did not meet my requirements as well as unRAID.

Mike January 4, 2012 at 11:31 am

For those hosting less than 500 HD movies, I would like to propose a different solution. It’s basically “mirroring” with offsite backup.

I have 3 computers in my house, each with a single (4TB) hard drive. I use GoodSync, which automatically mirrors the data on all 3 computers and keeps everything sync’d up. On the main computer, I installed BackBlaze for $5/month unlimited online backup.

Advantages:
1.) Can sustain multiple disk failures
2.) All files are always locally accessible to the computer (no network connection needed)
3.) When I save new files on my computer, they automatically get mirrored and backed up in the offsite cloud
4.) If a thief steals my computers, I have my data
5.) If my house floods, I have my data
6.) If my house burns down, I have my data
7.) Pretty affordable when compared to multiple drives, spare computers, and the power to run all that
8.) No single point of failure

Cons:
1.) Limited/inefficient storage (4TB max)
2.) Must always purchase the largest/slowest hard drives on the market as your storage needs grow, instead of adding more disks
3.) Not as much fun as having a cool RAID-X storage server

Mike January 4, 2012 at 12:30 pm

In my previous post (above), I forgot to mention the cost savings of NOT having a separate storage/NAS server in terms of power savings. Just having a single internal disk on the computer that you’re already using means hundreds of watts of savings. Could be $10 or more per month.

Comments are closed, visit the forums to continue the discussion.

{ 2 trackbacks }

Previous post:

Next post: