It is currently Thu Jun 04, 2020 8:15 pm

All times are UTC - 7 hours [ DST ]

Recent News:



Post new topic Reply to topic  [ 329 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16, 17 ... 22  Next
Author Message
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 12:46 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10711
Location: Fort Collins, Colorado
Thanks: 623
Thanked: 922 times in 877 posts
cavediver wrote:
Disk 15 filled up with BDBB backups from my MSS awhile ago. I reported to Yakuza the PP1 behavior with the BDBB backups, but could never figure out why all of the BDBB backups weren't being spread across the drive pool. Disks 0 & 5 filled up when I copied over all of my 12TB video share from the MSS to Andromeda. When I was copying over 5 or 6 files at a time, the files were being spread evenly across the drive pool. I got impatient and started selecting and copying 30 - 40 files at a time, letting it copy all day while I was at the office. I noticed after the first couple of these that all of the files were being copied to one drive exhibiting PP1 drive extender behavior. I went back to smaller batches and the system again began spreading each file across the storage pool drives.


The only idea I have is that perhaps WHS is trying to keep "like data" together in circumstances where files are copied over within a short period of time. One of the benefits of the PP1 method of storing data on was that, for example, the tracks of a CD that you copy would all reside on a single drive. In the new model the tracks of the CD would be scattered across the various drives as the files are copied to the drive with the most free space. Could DE be trying to store them together, thinking they are the same "batch" of files and would be better off residing on the same disk?

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins?


Top
 Profile  
Thanks  

Attention Guest: Remove this ad by Registering with the MediaSmartServer.net Forums. It's Free!
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 6:39 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
yakuza wrote:
The only idea I have is that perhaps WHS is trying to keep "like data" together in circumstances where files are copied over within a short period of time. One of the benefits of the PP1 method of storing data on was that, for example, the tracks of a CD that you copy would all reside on a single drive. In the new model the tracks of the CD would be scattered across the various drives as the files are copied to the drive with the most free space. Could DE be trying to store them together, thinking they are the same "batch" of files and would be better off residing on the same disk?


I considered that possibility. Actually, I remember now that the two 2TB drives that are full, filled up when I attached a drive that had supposedly failed in my MSS to Andromeda as a lettered drive and copied the video files to the video share in Andromeda. Even though the MSS thought the drive had failed, the data was all accessible and copied over without errors. I did ctrl-A and copied the complete contents of the video share on the drive drive, which was pretty much 1.5TB of video files. Copying internally was extremely fast, but I was not pleased with the DE behavior, so I went back to copying over the network and the PP1 DE behavior stopped.


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 10:59 am 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
I believe that the Drive Extender manages files primarily by folder and attempts to keep as much of a folder on a single disk as possible. As I recall it will fill beyond that 90% to make that happen. Now, that is my recollection from the early days (beta, initial release, PP1) but I have not tried to follow it in detail since then.

Could that be what is going on here?

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 11:19 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
JohnBick wrote:
I believe that the Drive Extender manages files primarily by folder and attempts to keep as much of a folder on a single disk as possible. As I recall it will fill beyond that 90% to make that happen. Now, that is my recollection from the early days (beta, initial release, PP1) but I have not tried to follow it in detail since then.

Could that be what is going on here?


No! Since PP2, DE places all new files copied to the WHS storage pool to the drive with the MOST amount of free space. So, in a system with multiple drives, each new file should always go to a different drive than the file copied previously. This keeps the storage balanced as a percentage of free or used space between all of the drives in the storage pool. However, the data or files already existing on a drive stay there and do not move without some sort of external manipulation, such as using drive balancer or removing and replacing drives. If you look at the screenshot I provided above you will see 5-2TB drives with only 5% used space. These 5 drives were added just before I turned on duplication and most of the duplicated files landed on them. All new files copied to Andromeda will land on one of these 5 drives until they reach 14%, then the data will be spread over 6 drives and so on until the data spread reaches a balance with all of the other drives. All this is assuming no further DE anomolies.

I'll be receiving about 10 or so new blu-ray's today and tomorrow from Amazon which will be copied over to Andromeda. I'll post some new screenshots this weekend to show you how DE spreads the data.


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 11:31 am 
Offline
Moderator
Moderator
User avatar

Joined: Wed Jul 16, 2008 4:23 am
Posts: 3838
Location: Long Island, New York
Thanks: 94
Thanked: 243 times in 236 posts
cavediver wrote:
I'll be receiving about 10 or so new blu-ray's today and tomorrow from Amazon which will be copied over to Andromeda. I'll post some new screenshots this weekend to show you how DE spreads the data.


OK, so I have to ask, what Blu Rays? I just got Harry Potter yesterday and will probably get Public Enemies in the next few days.

_________________
Cheers,
Damian

DIY WHS - 12.5TB and growing, too many gadgets to list

Did you read the MediaSmartServer blog today?
Blog - http://www.adigitalhomeblog.com
"Like a midget at a urinal, stay on your toes!" - Lieutenant Frank Drebin


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 11:50 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
dbone1026 wrote:
cavediver wrote:
I'll be receiving about 10 or so new blu-ray's today and tomorrow from Amazon which will be copied over to Andromeda. I'll post some new screenshots this weekend to show you how DE spreads the data.


OK, so I have to ask, what Blu Rays? I just got Harry Potter yesterday and will probably get Public Enemies in the next few days.


Harry Potter, Masters of Horror Season 1 (4 disks), Star Trek, Terminator, Godzilla, and a couple more that I haven't decided on yet, (I'm weighing my options on which to purchase now and which ones to wait and see if they go on sale for Xmas).


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 5:29 pm 
Offline
Moderator
Moderator
User avatar

Joined: Wed Jul 16, 2008 4:23 am
Posts: 3838
Location: Long Island, New York
Thanks: 94
Thanked: 243 times in 236 posts
cavediver wrote:
Harry Potter, Masters of Horror Season 1 (4 disks), Star Trek, Terminator, Godzilla, and a couple more that I haven't decided on yet, (I'm weighing my options on which to purchase now and which ones to wait and see if they go on sale for Xmas).


OK, sorry for taking this thread off topic, after this post I will stop. I watched Star Trek Blu Ray this weekend. Very good movie, but the sound was absolutely awesome (TrueHD bitstreamed from my PS3 to my onkyo 906 AVR). A lot of people talk about the video quality of Blu Rays, but often overlooked is the superior audio quality that I find versus the same SD DVD.

_________________
Cheers,
Damian

DIY WHS - 12.5TB and growing, too many gadgets to list

Did you read the MediaSmartServer blog today?
Blog - http://www.adigitalhomeblog.com
"Like a midget at a urinal, stay on your toes!" - Lieutenant Frank Drebin


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 5:53 pm 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
I'm completely spoiled now by 1080p blu-ray + DTS HD-MA and Dolby TrueHD audio.


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Wed Dec 09, 2009 9:31 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
cavediver wrote:
JohnBick wrote:
I believe that the Drive Extender manages files primarily by folder and attempts to keep as much of a folder on a single disk as possible. As I recall it will fill beyond that 90% to make that happen. Now, that is my recollection from the early days (beta, initial release, PP1) but I have not tried to follow it in detail since then.

Could that be what is going on here?


No! Since PP2, DE places all new files copied to the WHS storage pool to the drive with the MOST amount of free space. So, in a system with multiple drives, each new file should always go to a different drive than the file copied previously. This keeps the storage balanced as a percentage of free or used space between all of the drives in the storage pool. However, the data or files already existing on a drive stay there and do not move without some sort of external manipulation, such as using drive balancer or removing and replacing drives. If you look at the screenshot I provided above you will see 5-2TB drives with only 5% used space. These 5 drives were added just before I turned on duplication and most of the duplicated files landed on them. All new files copied to Andromeda will land on one of these 5 drives until they reach 14%, then the data will be spread over 6 drives and so on until the data spread reaches a balance with all of the other drives. All this is assuming no further DE anomolies.

I'll be receiving about 10 or so new blu-ray's today and tomorrow from Amazon which will be copied over to Andromeda. I'll post some new screenshots this weekend to show you how DE spreads the data.

I am not so certain it is strictly by file as opposed to FOLDER/file...

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Sat Dec 12, 2009 10:28 pm 
Offline
Newbie
Newbie

Joined: Sat Dec 12, 2009 10:25 pm
Posts: 6
Thanks: 0
Thanked: 0 time in 0 post
Just out of curiosity: are you encoding your blu-rays to .mkv? Or are you avoiding stripping/compression and replicating the image? Additionally, how many movies is 12TB worth?


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Sat Dec 12, 2009 11:03 pm 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
RemoteDoc wrote:
Just out of curiosity: are you encoding your blu-rays to .mkv? Or are you avoiding stripping/compression and replicating the image? Additionally, how many movies is 12TB worth?


I do not encode anything to mkv. I personally don't understand the fascination with mkv files. All of my blu-rays are either 1:1 m2ts files or m2ts files made from seemlessly branched blu-rays with ToNMT. My current video share is 12.5 TB which consists of ~ 400 blu-ray movies and 13 TV series, some with multiple seasons each which contain 10-15 episodes per season and a bunch of SD DVD iso files.


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Mon Dec 14, 2009 2:44 am 
Offline
Newbie
Newbie

Joined: Sat Dec 12, 2009 10:25 pm
Posts: 6
Thanks: 0
Thanked: 0 time in 0 post
cavediver wrote:
I do not encode anything to mkv. I personally don't understand the fascination with mkv files. All of my blu-rays are either 1:1 m2ts files or m2ts files made from seemlessly branched blu-rays with ToNMT. My current video share is 12.5 TB which consists of ~ 400 blu-ray movies and 13 TV series, some with multiple seasons each which contain 10-15 episodes per season and a bunch of SD DVD iso files.


Thanks Cavediver, answered all my questions.


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Mon Dec 14, 2009 5:53 am 
Offline
Moderator
Moderator
User avatar

Joined: Wed Jul 16, 2008 4:23 am
Posts: 3838
Location: Long Island, New York
Thanks: 94
Thanked: 243 times in 236 posts
RemoteDoc wrote:
Just out of curiosity: are you encoding your blu-rays to .mkv? Or are you avoiding stripping/compression and replicating the image? Additionally, how many movies is 12TB worth?


if you want to put into mkv but keep the original quality what many people do is they use eac3to and convert the hd audio to FLAC. This allows you to get the full lossless audio without having to worry about whether or not your AVR can handle. I prefer the same method as Cavediver though, I think he uses ToNMT where I use Clown_BD, but end result is still the same

_________________
Cheers,
Damian

DIY WHS - 12.5TB and growing, too many gadgets to list

Did you read the MediaSmartServer blog today?
Blog - http://www.adigitalhomeblog.com
"Like a midget at a urinal, stay on your toes!" - Lieutenant Frank Drebin


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Thu Dec 24, 2009 7:57 am 
Offline
.5TB storage
.5TB storage

Joined: Tue Dec 22, 2009 9:04 am
Posts: 11
Thanks: 4
Thanked: 0 time in 0 post
This is sweet!

I want to attempt something similar (but probably on a bit smaller scale to start). How loud is the Norco case? I live in a condo and plan to keep it in the den (has no door).

Could I swap out the fans to make it almost completely quiet?


Top
 Profile  
Thanks  
 Post subject: Re: Andromeda
PostPosted: Thu Dec 24, 2009 8:25 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
scrabble wrote:
This is sweet!

I want to attempt something similar (but probably on a bit smaller scale to start). How loud is the Norco case? I live in a condo and plan to keep it in the den (has no door).

Could I swap out the fans to make it almost completely quiet?


With the 120mm fans, Andromeda is quieter than my MSS + the three external enclosures. Andromeda, now at 34 TB, has 19 drives in it including the SSD OS drive. Andromeda lives in my office about 6 feet away from my desk and the sound is audible but not too much. I could make it completely silent with lower RPM fans, but then my drive temps would increase. I've found a good balance of sound level and drive temps that I'm happy with.


Top
 Profile  
Thanks  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 329 posts ]  Go to page Previous  1 ... 11, 12, 13, 14, 15, 16, 17 ... 22  Next

All times are UTC - 7 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group