MediaSmartServer.net
http://www.mediasmartserver.net/forums/

FYI, Red storage drive failure without any Smart warning
http://www.mediasmartserver.net/forums/viewtopic.php?f=1&t=14621		 Page 1 of 1
Author:  Gardian [ Sun Feb 25, 2018 9:42 am ]
Post subject:  FYI, Red storage drive failure without any Smart warning
Long read, but might be of value.
Hello all, I thought I would recount what happened yesterday, what I found and how I fixed it and how it's going today.
I have the X-510 same hardware as the EX-490 with WHS V1 for system. 2tb red for os drive and two 2tb reds for storage drives. Both storage drives were full and I was getting ready to add the forth 2 tb red soon.

First off what happened, I had a red 2tb storage drive fail (with No Smart warning) yesterday morning. This drive was 6.5 years old. I believe it was one of the first I had purchased.

How it started was I noticed that the console was grayed out on my laptop yesterday morning and I could not remote to the server. I checked the rack and sure enough the server was in a far right light blinking blue state, All drive lights on.
I tried to soft shut it down, no joy, so had to hard kill it by holding the button on the back for 4-5 seconds. Waited a bit and fired it back up, now same light blinking but no hard drive lights. I thought wow this is strange because I had No Smart warnings.

So Time to pull out the debug board and cable and see what was stopping it. It was time for its yearly deep cleaning anyway.
So got the server out of the rack and on the table, disassembled it and attached the cable, routed it up the back inside and replaced the motherboard and power supply after blasting everything with canned air and wiping everything down, then all the drives, 3 total, all red 2tb.
I knew two drives were full and the os drive was half full, I was getting ready to add a new drive sometime soon.
Attached the debug adapter dragged out a monitor/keyboard/mouse/cords and fired It up, saw the win 2003 startup logo then it went dark and then nothing.
So, not good, I pulled both storage drives and tried again, walla, the server boots right up so I know it is a bad storage drive. But Smart never said anything (which is why I wanted to write this) so I plugged in one of the storage drives and it booted ok, Great, So I plug in the other storage drive and bang, nothing.
I figured it was the storage drive, but took the os drive to another PC and ran a full disk check on it just to make sure, tried booting again, nothing.
So I pulled the bad storage drive and attached it to another pc via usb dock and the system asks if I wanted to format. Dang, it is dead. I figure what the heck, I have two backups of all my data so I click yes to the format and I get an error.
It’s dead.
I looked at the date 2012 and smart said it had 6.15 years of run time, so pretty sure it was the first red I purchased.

So now I had to force a removal which took about an hour. I rebooted the server and I fine something I did not expect. Some of my biggest folders, it had turned off duplication and warned me there was only enough room for one copy of each file. I thought, OK at least maybe all the files are still present, that’s cool.

I add a new 2tb red drive and add to the pool, reboot and get a warning that the client pc backups need to be checked, I wasn’t surprised, clicked yes and find all are gone, no problem, they are all minimum loads all data stored on the server and I only keep the last couple of backups anyway just in case of ransom ware. It finishes and I reboot it again and before server 2003 boots it run a system disk check, I let it and then it boots up.

Long story short, all is becoming back to normal, I did lose some data but very little and most of it was stuff I didn’t really need anyway.
I turned off the sleep program and let it run all night and it is backing up all the clients this morning which will take all day as they are full backups.
The other drives in the system are 2016 so I figure I need to find some more reds on sale and pickup 3-4 more and due a recovery at least on the main os drive and then slowly rotate out the older ones in the next few years.
I will add the fourth drive just to give the system some breathing room.

I was surprised at how well this storage failure went, but am concerned that the red drive seemed to give no smart alert before complete failure.
I was also surprised at how the system kept 2 copies of the files on two drives and turned off duplication by itself to help preserve what files it had. I did not expect that. I figured at best I would have to do a big restore of data from my non-pooled drive at best. But really not that much, I think I will pull what little data lost from the second sync’ed server as it was the last time I backed up.

Hopefully this might help somebody, I could have done this without the debug board, but it would have been difficult at best. Being able to see was great!

I will let everything settle back in and check file status, maybe next week I will do a re-sync of data and then the week after that I will update my non-pooled drives.

Keep an eye on those drives when you’re cleaning, it looks like the reds go 5-6 years and can die without warning and as always, keep some kind of backup of the data you can’t lose outside the server.

Hope this helps, Good Luck and enjoy.
Author:  Ruben Rocha [ Sun Feb 25, 2018 10:53 am ]
Post subject:  Re: FYI, Red storage drive failure without any Smart warning
Probably the motor failed in the drive.
Author:  Gardian [ Mon Feb 26, 2018 8:48 pm ]
Post subject:  Re: FYI, Red storage drive failure without any Smart warning
I agree, I could hear:
click,click,click, buzz, click,click,click, buzz over and over.

Just to followup, lost no data that was in duplication only stuff that was not duplicated which I didn't really care about anyway. It's backed up on the other systems.

All is well now. All clients are backed up and system has been disk cleaned and rebooted.

Very nice I have to admit.
Page 1 of 1 All times are UTC - 7 hours [ DST ]
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/