Hi guys, I'm thinking of finally taking the plunge and ripping my entire collection to a central server. Let's start with what's already at hand: HP MediaSmart ex490, Cat 6 Network and the Home Cinema (obviously), approx 400 Blurays (and growing) and around 1600 DVDs.

What I'm looking for is a large external storage box to connect to the MediaSmart in order to store the disks. The largest (non-enterprise class) boxes I've seen are 8-port boxes with a maximum capacity of 24TB max, and that not assuming the use of 3TB disks, not considering the possible incompatibilities between WHS v1 with 3TB disks. but the way I see it, it's just not feasible to store that size of a collection on a single storage box, is it?

The other obvious solution is to compress or convert to MKV, but this is something I'd rather not do since it would create additional problems (not being able to view the extras, etc).

I'm open to almost all ideas (replacing the ex490 with another device, upgrading to Vail, etc) as long as I can come up with a reasonable, working solution. The only requirement would be that the storage box would have to be silent (same as ex490 or similar) and should offer some form of redundancy (RAID 5 would suffice), since re-ripping my collection in case of H/W failure would be rather time-consuming I suppose.

As far as the media player is concerned, I'm currently in between Popcorn's A210, C200 or the Boxee.

I'm looking forward to your ideas; thanks for your time.

Aside from the need for drives you really need to play with some test files to see what works best for you. I have come to like MKV and use makeMKV to rip all my DVD and Blu-Rays but before figuring that out I spent a couple months playing with various formats streaming both in the house and on the internet. I also played with a wide range of programs mostly playing with streaming my content online. I have used ORB, SubSonic (Great for Music but also does video), and lately I have been using with great success Remote Potato. One issue you will have to work around in your EX490 is its horsepower and working within its means to achieve the desired effect. For this reason I urge you to experiment with a few Movies and then see how everything comes together.

As for storage solution and you will use quite abit of hard drives looking at the size of your collection. The eSATA port on your EX490 will support a 5 bay enclosure using the USB 2 ports you can connect 8 bay enclosures. There has already been much written about these types of enclosures and their use with the various HP MSS units that may help you to determine if thats a soultion for you or not. To actually rip a collection of that size will be extremely time consuming and so it would be wise to have Duplication turned on and thus the requirement for more drives. Figuring out the need for capacity will be determined by the Video Format you chose then you can average it out accross your collection and then you will get the big picture of what you will need and then you can cost out the enclosures and drives to see which path you will take.

Honestly the one factor that puts a big hit on the CPU is transcoding and so you may want to upgrade the CPU in your EX490 to better handle the task at hand. For now you just need to determine the format you want and experiment with the stream be it in your home or online. Streaming online is a much bigger task and you may want to address the CPU to help boost streaming performance. Now this is not my area of expertise but I am speaking from my own personal experience and there is something to be said about that. I know nothing about HD Audio bitstreaming or some of the finer details of such things. I am getting old and just to be able to see a clear picture and hear sound is plenty enough for me, heck I do not even use any of my surround equipement anymore.

You do have lots of options before you but please test out somethings, review the needs and then cost out the approach to see what works best for you. I do everything using Windows Home Servers and I no longer attempt to stream anyting using my EX470's even the one with a BE2300 but I do have a ProLiant running WHS v1 that streams well and I have a WHS2011 Test Server and am in the process of a DIY Build called Highland which you can see in my tag line. Take a look at Remote Potato and see if it apeals to you and no pun intended there.

The task you are about to take on is fun, enjoyable and when done right its great to be able to stream your media in home, on the internet, on your cell phone you name it it can be done and streaming Blu-Rays is cool too.

_________________
Exploring the possibilities!

Migrated from WHS2011 to WS2012E: HIGHLANDER

~ Norco 4220 Enclosure
~ Gigabyte GA-990FXA-UD5
~ AMD Phenom II X4 995 3.2 GHz
~ 8 GB Corsair Vengeance DDR3 1600 (PC3 12800)
~ 3 Supermicro AOC-SASLP-MV8

Thanks for your reply Comp1962. I did some additional research and just by the sheer size of my collection, I believe that I will have to make the concession of ripping everything to MKV.

As far as the CPU of my ex490 is concerned, this was the weakest link of the setup from day one, so I should have already mentioned that I have replaced the stock CPU with an E8600 (if I remember correctly). I was also thinking of performing all ripping on my workstation and just copy the created mkv file to the server when created, so the MSS would just have to be powerful enough to stream the HD content to a network media player without stuttering and freezes. Regardless of the CPU horsepower however, I am starting to think that the ex490 with WHS1 or Vail will NOT suit my needs. Given the fact that the only form of data redundancy offered is duplication (WHS1) or Raid1 (Vail), I simply cannot afford to buy twice as many disks in order to store my entire collection. Raid 5 would be a different thing though...

I then started thinking about the HP Proliant MicroServer, where in theory, you can install a SmartArray P410 Array controller for Raid5 on for the four (or 5, if you use the dvd slot as well) internal drives. For external storage, a viable solution could be the Sans Digital TowerRAID TR8M-BP. However this solution is not scalable. The MicroServer's e-sata is not port multiplier aware, meaning that when you fill up, you have nowhere else to go... Buying 3TB disks (apart from extremely expensive), is not supported by either the Sans Digital box or the MicroServer (due to Vail and Small Business Server limitations).

Can anyone suggest a redundant and scalable configuration based on any kind of box or OS?

The 2TB limitation is in the OS itself but there are work arounds to utilize drives larger than 3TB. While a RAID Solution looks good the thing is to do it properly is expensive and to do it improperly you can expect drives to drop out. Utilizing WHS v1 and its DE offers up decent but slow duplication and has its issues but its time tested and does work. In the WHS2011 front some are using Raid others are testing Drive Bender and Stablebit Drive Pooling software. I am testing Stablebit right now and its friendly and flexible but I have never used Drive Bender so I can not comment there.

MKV formats offer no real space savings in that the format does not compress the file as does AVI, MP4 etc. The big difference is in the quality of the stream. Again I do urge you to take a couple of your favorite movies and rip them into different formats and just test them to see which works best for you then you will not have ripping remorse months later. I use to take the subtitles out and spent 3 months ripping my collection. My gf at the time never said a word to me while I was doing this but after it was all said and done she asked where the words were, Chinese for Subtitles. I said no one reads those things so I took them out. She said put them back. So before you begin make certain you have everything covered.

The HP Microserver has a couple expansion slots I believe and you can use those to install eSATA Cards to support external enclosures. The only issue I see that I eluded to earlier is the CPU and if your server is going to be doing Transcoding it will need a decent processor to accomplish the task at hand. If your media extender does all the transcoding then your server only needs to deliver the media. If you want to stream online then your server will be doing the transcoding. So depending on what your going to ask of your sever current of future may or may not require a workhorse CPU. With the task your about to undertake you do have the oppertunity to test the waters with what you have and see if it will work for you or not and if not then of course you can continue to explore other options which your doing now. Because your movie collection is large you will need to calculate how much storage capacity you will need but now I am sounding like a broken record but you will be amazed at how quickly you can eat up drives.

Once you figure out the space requirements then you will be better able to determine the type of enclosure you would be after. Having one enclosure to house everything is nice but can be costly. The largest single enclosure I have seen and there are posts on the site about it is a 30 bay enclosure which is self contained and on wheels to boot. I can point you to the links when I have more time after work if you like. Then there are the 20 and 24 bay 4U Norco Rack Mount Enclosures and then there is another Norco Enclosure just for housing drives but it too is a 4U rack mount enclosure. Again I will be more than happy to point you to all those enclosures later tonight when I come home from work.

_________________
Exploring the possibilities!

Migrated from WHS2011 to WS2012E: HIGHLANDER

~ Norco 4220 Enclosure
~ Gigabyte GA-990FXA-UD5
~ AMD Phenom II X4 995 3.2 GHz
~ 8 GB Corsair Vengeance DDR3 1600 (PC3 12800)
~ 3 Supermicro AOC-SASLP-MV8

I definitely see what you mean with the first step being the selected file format and what to keep/discard during the ripping process. If disk space were not an issue, I would definitely go with ISO or BDMV structure. However, given the size of my collection, MKV is the only way to go for me. I realize that MKV is merely a container and can offer no real savings if nothing is removed from a disk, but removing the extra audio streams, subtitles and extras, I count on saving approx 20-25% on the size of each BD. Compressing movies to AVI or any other video format is not an option for me; the way I see it, I started my BD collection to upgrade my picture quality and I have no intention of compressing a film just to save on disk space, hence the decision to go with MKV.

Oh, I forgot to mention that I'm not interested in online streaming, just local, home streaming via gigabit ethernet would be more than enough for me. I would really appreciate it if you could provide me with any info you have available on boxes. I wouldn't consider rack mounting enclosures but anything else would be great. Ideally speaking, I'm looking for a box or combination of boxes capable of storing approx 30TB with expandability in mind!

Thanks again for all your time and advise.

I said that too - at one point in my life. After seeing what could be done (at work, on the phone, on a wi-fi plane) - I changed my mind about internet streaming.

There are two things I always say (1) Exploring the Possibilities and (2) Never Say Never. However not everyone is interested in streaming their content online but if you do it just once and find yourself addicted to it then it becomes an entirely new ball game with new challenges to tackle. If you can maintain a minimalist approach then great and you may be better off for it but if you configure or build something defined and limited to just a few specifics and not allow for some growth then you end up upgrading.

_________________
Exploring the possibilities!

Migrated from WHS2011 to WS2012E: HIGHLANDER

~ Norco 4220 Enclosure
~ Gigabyte GA-990FXA-UD5
~ AMD Phenom II X4 995 3.2 GHz
~ 8 GB Corsair Vengeance DDR3 1600 (PC3 12800)
~ 3 Supermicro AOC-SASLP-MV8

I understand where your coming from cutter - but from real life experience - I've found that compressing into h264 with handbrake reduces a 40gb file to 6 gb. The compression is always variable - you never know what your going to get in final file size (there's alot of dependencies on final size like picture granularity, the compression profile (high, baseline, etc), final dimensions, etc.).

I have a 58" samsung plasma tv with a self-bult htpc networked to my WHS. On the WHS - I have over 430 movies at 1.41 terabytes and 1142 tv show files (across 16 different tv shows) at 581 gb. My blu-rays are all 1080p and still look incredibly awesome on the plasma tv.

Remember - compression isn't about reducing the dimensions of the picture (why convert from 1080p to 420p? that'd be horrible). Instead - compression programs like Handbrake reduce the 40gb to a more manageable value by smartly removing unneeded, unseen, artifacts within the pixels of the video or changing the way the pixels are displayed by playback devices that support the compression methodology (I hope I said this correctly - if not - please feel free to correct me).

My biggest movie to date (compressed) is godfather - part II at 13 gb. I use a high profile preset inside of handbrake (which uses all the tricks in the ffmpgeg book to reduce file sizes), includes two audio tracks (AC3 passthrough and AAC mixdown 2 channel), and I always burn in the forced subtitles (this is always hard to figure out where forced subtitles may be...)

Reducing each movie to approx 6GB is definitely very attractive TedC. In fact, it's so attractive that I'm tempted to hit the purchase button on a MicroServer or similar box, purchase 4 2TB disks and start ripping during the weekend! But I know what's going to happen next. In a few months time, just when I'm about to finish the rip and compression of approximately 2000 disks, I'll be looking at the latest 10 or 12-bay enclosure with integrated RAID5 controller, fully supporting 3TB disks and thinking whether I should re-rip everything in lossless MKV...

The more research I do, the more I think that if I'm going to do this right, and more importantly, if I'm going to do this once, the best solution would be to wait for a few months hoping that a box more suitable to my needs will appear... or maybe I'm just tired from all the searching these past few days and just need to clear my head and continue researching!

Actually the best thing for you to do is look at building your own server which can handle your needs now and grow with you over time. You can get some large enclosures and add drive cages as you expand or you can go with a Norco enclosure which has the backplanes and trays. One forum member has two large Norco Enclosures do not remember if they are 20 or 24 bay enclosures another has a 30 bay enclosure. My new server has 20 bays which I hope will carry me for a couple years.

My media collection is not as extensive as yours or others but it continues to grow and so just having the ability to keep adding drives and the ability to daisy chain other enclosures is a nice option to have. Chosing the components to best suit your needs does require abit of thought but you can somewhat calculate your storage needs based on the collection you have to get a ball park figure on how many drives you will need. If you use a RAID Solution I recommend you take the time to do it right or you may have drives dropping out of your arrays. I am contemplating a RAID Solution but the cost to do it properly is out of my price range so I am hoping Stablebit Drive Pool will work. I am testing it now in a test server I built last year and so far its working well and looks very promising for my WHS2011 servers.

If you have not done so already take a look in the DIY Build Forums and read about what others have built and their thought processes which should provide you with some really good ideas. There are some very impressive builds there that are just well worth looking at.

_________________
Exploring the possibilities!

Migrated from WHS2011 to WS2012E: HIGHLANDER

~ Norco 4220 Enclosure
~ Gigabyte GA-990FXA-UD5
~ AMD Phenom II X4 995 3.2 GHz
~ 8 GB Corsair Vengeance DDR3 1600 (PC3 12800)
~ 3 Supermicro AOC-SASLP-MV8

Keep in mind there is no right answer when it comes to encoding/compression. There are just too many variables (individual's equipment/setup, eyes, ears, etc...). You just need to find what works best for you. Here is my approach that I finally settled on:

- Custom Built WHS - Shawshank - currently at 16 HDDs for 26TB
- All Blu-ray movies are ripped to a main movie BDMV folder rip using Clown_BD
- All DVDs movies (very few these days) are ripped to a main movie mkv with MakeMKV
- All TV Show Blu-Rays/DVDs are ripped and encoded to individual episodes using Handbrake or Clown_BD

So now the question, why preserve lossless on the movies yet encode tv shows. Well, for two reasons:

(1) I have over 2,400 TV Show episodes. About half of them I record via TV Tuner so they are already no longer lossless. For the DVD/Blu-ray boxsets a single TV Series could be in excess of 100 episodes, and keeping lossless would take up an extremely large amount of space
(2) PQ or AQ is not as important for me with TV shows as it is for movies, in particular for the amount of space I would need for TV shows as mentioned above.

For movies I have found it hit or miss with the quality when encoding. Some movies look great, others you can easily see artifacts, especially in dark scenes. Also, just as important as picture quality is audio quality, and I want to make sure I don't make any compromises there. Ultimately staying with lossless movies for me keeps me future proof (who knows, maybe down the road I will build a home theater with a massive 100in+ screen, and in an instance like that a compressed movie could be more obvious).

Now once again keep in mind, I am just talking specific to my setup and needs. Ultimately you should choose what works best for your setup and gives you the most flexibility.

_________________
Cheers,
Damian

DIY WHS - 12.5TB and growing, too many gadgets to list

Did you read the MediaSmartServer blog today?
Blog - http://www.adigitalhomeblog.com
"Like a midget at a urinal, stay on your toes!" - Lieutenant Frank Drebin

Do the math, are you kidding, you don't want to compress, and have duplication.
400 Blurays (and growing) and around 1600 DVDs

Let us know how you pull this off.

I rip my dvd's to mpg which shrinks them by 3/4 but still at a gig a pop it adds up fast and yes, the quality is not great.

Sounds like you need stuff that netflicks has

good luck

_________________
[X510 CPU/RAM/All 2 TB Red's] [X510 CPU/Sync'ed Backup/All 2 TB Hitachi's]
[X510 /2 in box]
[X710 LIAN LI DIY W2016E] Stablebit Drivepool [EX-503 LIAN LI 5-bay USB3 External]
Fear the Cloud