I would be a LOT concerned... You should NEVER directly manipulate those files. This is exactly why HP and MS warn us about the use of the Remote Desktop!

Whether your client backups are messed up or not will depend on a lot of things (which DAT file it was in the sequence, for example) and whether running another backup completely restored things. I would very much suspect that your backup database is now corrupt. Feel free to wait for comments from others but I would not rely on it as my only backup at this point.

If you had backed it up using the BDBB Add-In (or the separately document4ed automated procedure) I would restore the backup from there.

If you have not backed it up I would reset it and start over. (WHEN would be a function of whether you have another backup scheme running and your tolerance for risk.)

EDIT: Even more concerned seeing your more recent post...

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

George, I noticed you have the backup job running at 6AM. Normally this should be fine, but do note that the backup service will continue to run after the end of the maintenance window if it needs to continue backing up a client that is in progress. Your script could then shut down the backup service and leave the backup DB in a weird state. Backup Cleanup on Sunday AM could take a while complete and cause this error to occur. Just a hypothesis.

Did a reboot of the server resolve the Backup Service not Running message in the Server Console? I've occasionally seen this message after doing a WHS BDBB backup, and a reboot always resolved it.

Mike, I recommend you stop the Backup Service, delete everything in D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} and then start the backup service. By only deleting the dat file, you've removed the index but left the data chunks behind, this will at best consume a ton of disk space and at worst could potentially cause a conflict. This should get you to a clean state with the backup DB.

Finally, George, it made it a bit difficult for me to read with all the Edits and Updates to existing posts, the chronology was lost so it was sometimes hard to tell what was old and what was new content. I personally find it easier to just read the thread in chronological order. For example, your first post on this failure has an Update that you are investigating a possible failed drive, and it's unclear to me from reading the rest of the discussion if that is resolved or not. Just some feedback that may be useful or may be promptly tossed out the window.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins?

Mike, the deleted file will be rebuilt when your PCs are backed up, I think. So you don't need to worry about the deleted file. JohnBick confirm or deny this if you know. If not overnight during the regular nighly backups, it will be "replaced" during backup/cleanup on Sunday morning.

Edit: Sorry John and Alex. Somehow I missed your previous posts about being concerned about deleting the backup file/cluster.

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

Alex, point taken on the confusing edits. From now own I will just do another post rather than updating an existing post. I can see how that would make the timing confusing.

I will move my daily batch file to 9PM from 6AM. I had chosen 6AM before because I noted that my backups on all PCs were always completed by 3AM. Since the batch file runs in under three hours even after backup/cleanup on Sundays, starting the batch file at PM should allow ample time for it to complete each day before the Midnight backup window starts. If you see any problems with this plan, please let me know.

As I recall, Alex. I did not reboot the server on Saturday when this error occured and then came back. But, honestly, I don't remember for sure. That's a good tip. If I see the condition again, I will reboot to see if that helps. I am not sure I will see the condition again, in the near term at least, becuause I "remarked out" the XCOPY and net stop/start/drive letter services part of the batch file.

However, I still need to decide what I am going to do about having my BDBB backed up daily. If we don't come up with a reason the XCOPY command (along with some of my other activites... like removing an external disk or sometghing like that) is causing the screenshot conditions to exist, I will have to think about that and decide what course of action I will take regarding BDBB backups.

Thanks for the feedback on chronology. To attempt to clarify, any post I made after June 16 at 1:03PM, six (of a total of eight drives in my datapool) drives that had backups on them had been chkdsk(ed) with chkdsk x: /f /r . I ran chkdsk on the remaining two drive a day later. After all that, chkdsk found no errors on any of the 8 disks.

Situation now:

1. Disk drives have no errors (all eight drives checked)
2. Screenshot conditions were not present for serveral days during which time I did NOT run the batch file.
3. This past Saturday I manually ran the batch file and watched it run in a DOS window on the server. After the XCOPY command completed, the server immediately shut down. When I repowered the server the screenshot conditions DID exist. I tried to fix them by running Repair on the BDBB. No help. As well as I can remember, I did NOT reboot these server. A few hours later the screenshot conditions were gone and backups were back to "normal".
4. The BDBB remains normal to this day (currently three days after Saturday)
5. They seem to be staying that way as long as I don't run the XCOPY command in the batch file.
6. The XCOPY Command in the batch file that is executed is when I don't have it "remarked out" is:

XCopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} G:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /D /E /V /C /I /G /H /R /K /X /Y >C:\Temp\External-Backups.log

Of course, the command in the batch file is not wrapped (It's on one line)

7. I suspect this error has something to do with my problem but I don't know for sure.

Here is the System Event Log error I am referring to:

Client Backup server failed at d:\wssg_src\whs_pp2\qhs\src\backup\util\datafile.cpp(890)

I have no idea what that means and I have not approached MS about it yet to find out more.

8. I believe I would be able to reproduce the error by running the batch file with the XCOPY command "turned on". If anyone would like me to try that, in the interest of getting to the bottom of this very odd sitation, I will.

Of course, another solution would be to just keep the XCOPY command remarked out and go to another method of backing up the BDBB, maybe your 1.0.1.6 version of the WHS BDBB add-in.

Thanks to everybody for trying to help
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

I have one additional piece of information that may be relevant to my current problem summarized in the previous post. I began running Perfect Disk 10 for WHS near the end of April. I emailed PD support giving them the error:

Client Backup server failed at d:\wssg_src\whs_pp2\qhs\src\backup\util\datafile.cpp(890)

They do not think my problem has anything to do with PD, but I am not completely convinced of that.
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

I have been using basically the same command line for a while now with NO problems:

XCopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} T:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /D /E /V /C /I /G /H /R /K /X /Y >"C:\WHS Backup Logs\External-Backups.log"

I also have been running PerfectDisk (several generations, currently "10") since before I tried the first automated backup.

I do NOT experience this problem and can NOT make it occur.

George: Does this occur if you use RoboCopy instead of XCOPY? And I thought you had stated before it also occurred at least once with BDBB? Or am I mis-remembering that?

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

That's an good idea, John. I have not tried RoboCopy to make a copy of the BDBB. I will replace the xcopy with the equivalent RoboCopy command line and let you know.

As to your second question, I was using WHS BDBB (Alex's first released version) before I started using the batch file you wrote and I modified. I never saw the same problem with WHS BDBB. What you are remembering is that I told you that I had tried a "round trip" create and then restore from WHS BDBB and one of those BDDB's (my oldest one) restored a corrupted backup database. When I tried a repair, it lost all the backups. I never saw the screenshots I posted above when I restored the DB... they were normal. The Health Monitor Shield in the Cosole just said I needed to Repair the backup database. When the other DB restore failed, I then restored from a more recent WHS BDBB and it worked fine. I told you I had not reported this to Alex because I was not certain that the older BDBB (the one that restored a BDBB in need of a Repair) was broken when the WHS BDBB was created. I will never know the answer to that. I just don't remember if I created the older backup DB when my BDBB was bad or not. However, it seems unlikely that I would make a backup when I knew the DB was "bad" but I would swear I did NOT do that.

Just to summarize, WHS BDBB never caused the subject screenshots to appear. It was another problem. Hope that makes sense.

This stuff and remembering what was at what stage when backups were made is getting complicated
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

I am running the batch file "as we speak". It's taking a lot longer with RoboCopy as you previously noticed, John. It appears to be nearly through processing the RoboCopy command now (after about 4 hours). Of course this particular backup is the longer "Sunday type" backup.

Based on the possibility I will be using the RoboCopy command to replace the XCOPY backup of the BDBB (fingers crossed it will not casue the screenshot problem), I have set my daily scheduled time back to 6PM.
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

The batch file finished about 4:55PM. The RoboCOPY command caused the same conditions to exist (the two screenshots DO appear). I think I may have an idea of something I have been doing wrong to cause this error but I am not sure. I want to document it here before I forget what happened or the screenshot conditions return to "normal" and then ask some questions to see if I am causing the problem myself.

1. Both screenshots showing the problem exist as of this minute (about 15 minutes afterf the batch file finished processing).
2. Specifically, the Computers and Backups Tab in the Console is showing "unknown" as the backup status of each home computer.
3. Also, the backup section of the Setting in the Console show Backup settings are not available".
4. The server did not crash this time (I wonder if that was coincidence on Saturday, last week)
5. There are no error conditions at the time the batch file finished processing in any of the Event Viewer Log Files.
6. I am "pretty sure" both screenshot conditions were present during the time the batch file was processing. I checked the Computers and Backup so I know that screenshot condition was present. I did not check the Backup section of the Setting screen so I don't know when that screenshot began being "not normal". I suspect these screens look like the problem is going on because the Backup Service is stopped before the BDBB backup starts. Can anyone confirm that?
7. The health monitor light on the server was red the entire time the batch file was running (because Backup service was off, I presume)
8. When the batch file finished the health monitor light on the server turned blue, indicating the Net Start Backup Service command was accepted, I presume)
9. I checked and both the backup service and the drive letter service are running on the server.
10. A log file of the backup was NOT produced (or it was deleted) because I don't see the backup log file with today’s date in the folder it should be stored in. I only see the backup log files numbered one through 8. File 9 is missing, indicating some of the commands were not processed. The number 8 backup log file is dated 6/21/09.
11. There is a file called index.4096.dat on the server but I believe that file might not have been copied to the external drive. I cannot confirm that because the log file for today (should be backup log file 9) is missing.

That is about all I know of the current conditions. I hope I did not leave anything out that is important.

Here is the Big Question. If it is not obvious why I was doing this I will explain in my next post (if necessary). I think this may have caused a problem but I am asking to make sure. During the batch file processing, I had the BDBB Backup folder open so I could see what files were there. I was watching the progress of the files in the BDBB folder being copied by the batch file. As I was watching the progress, I begin to think I had done that before (on Saturday) and I wondered if the backup folder being open when the bacth file was copying the files in it to the external drive and particularly when the last file was copied would cause a problem. I deliberately left the folder showing all the backup files even to the end. I wanted to see if that might cause the problem. Now, do you think that could be the problem? Am I creating the situation where these "unknown status" and Backup setting not available" conditions to exist?

I will monitor the screenshot conditions to see if they return to "normal" on their own and let you know if that happens.

Well I will just let you know right now. It is exactly one hour after the batch file processed and the screenshot conditions are back to normal.

So is this just a non-problem after all? Why is that happening? I wonder why the log files were not renamed correctly and I don't have a backup log for today’s backup? Still some questions unresolved!
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

Having the folder open shouldn't matter, as those are the source files to be copied to your external disk destination.

I think your Console may just be out of sync due to the backup service being down, this could also explain why it sometimes seems to magically fix itself after some time. Try doing a "Restart the Server Console", if that doesn't work, try restarting the server, and I suspect your problem will go away.

I think you just have too high of an attention to detail.

P.S. If you do decide to try BDBB again, shoot me a PM as I've got a new 1.0.1.9 that has a couple tweaks (nothing hugely significant, just changed the way I identify drives to be backup destinations).

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins?

My scheduled batch file ran at 6PM. Its almost 3 hours later and the screenshot conditions, that started just after the batch file ran still exist. If I had a need to restore a back in those three hours (and I bet it goes even longer until backups run after midnight this time) I would not be able to do so. I don't think you would say you pay too much attention to detail if it were your backups you needed to get to and could not.

I followed your advice and reset the home server console as well as rebooted the server. The condition is still there.

Thanks for letting me know about WHS BDBB 1.0.1.9. I will send the PM if I give up on getting to the bottom of this weird condition. Thanks for letting me know that.
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

George:

The red light during backup is normal. I have seen a (very) few times when that resets to blue but the backup service is actually NOT running and I am unable to actually access the backups without restarting it. Interestingly this has not prevented subsequent backups of clients or of the MSS. I am not positive as I have never really checked the times, but it could be this is an interim condition that is resolved at the next (hourly) "balancing".

The RoboCopy command lime may be saving files to a different directory. (That is actually what mine does.)

Check the actual backup for index.4096.dat -- is it there?

I thought Alex may have had a point with his comment that "your Console may just be out of sync due to the backup service being down". Next time I run into this I will try his suggestion myself, but it didn't solve your problem...

What error messages do you get if you attempt, after an hour or more, to access a backup? (Just open a client, pick a date and navigate to a file. No need to actually copy (restore) it.)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

Sorry George, poor attempt at humor.

What happens if you do a BDBB backup at this point when things look right in the console, does it show the same failure mode?

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins?

Good Morning gentlemen ... Retired early last night. Sorry about the delayed response. I will address your questions/comments in the order they were posted.

John: If you have time today, would you please run your script (btw, I interchange script/command file/batch file...can't seem to get away from the old DOS term "batch file" but I mean script or command file when I say that ) manually to see if the two screenshot conditions are present during the running of your script? If they are present, how long does it take for them to repair the server/console to "normal"? I am almost certain they will be present when processing on your machine as well. If they are not, that will tell me something about my cofiguration. The problem for me is there seems to be an inordinate amout of time (after the script is run containing a BDBB backup) that these "failure conditions" are present. As you say, the conditions may be present only unitl the next "balancing" is finished, but that should be an hour or less according to (as I understand their intent)the change MS implemented in PP1. Mine are often longer than that, as I saw last night when the script automatically ran. If that "window" extends until "after the next nightly backup is finished", with my previous custom of runninng the script at 6AM, my backups would be unavailable all day and well beyond after midnight. By changing the scheduled time as Alex recommended for anothe reason and by electing to run the script at 6PM, they would only be unavailable from 6PM to just after midnight on some occasions. Maybe I have just not done enough testing but my impression it the screenshot conditions are present after a script runs (either manual or scheduled) for sometimes as little time as 30 Minutes to as much as "not until the nightly backups have finished".

You are right. I copied your RoboCopy command (for backing up the BDBB) form your script in the first post. When I looked closely at it, I realized the log was going to another directory. Once I corrected that the logs are being handled properly.

Don't know why I did not think to check that Yes the index.4096.dat is there on the external drive so that not a problem; the BDBB database was copied successfully.

Correct. I am reconfirming that reseting the console and/or rebooting the server did not repair the screenshot conditions. I agree those were both good suggestions. Unfortuantly, neither repaired the conditions.

When the screenshot conditions exist, I cannot even view the backups for a given home computer. More spicifically, when the status of the backup is shown as "unknown" no amoutn of clicking on one of the PC that are being backed up will open the list of backups for it. Therefore, the backups are inaccessable. When the "unknown" status is repaired (after the hourly rebalancing has run or whatever is automatically reparing the status so it shows "backed up" instead of "unknown") the backups work "normally". I tested as you suggested and could select a file from a backup for restore... worked as it is supposed to. In either case ("unknown" or "backed up") there are no error messages. I looked at the event logs and found none there last night either. The server/console or whatever is "wrong" just magically repairs itself and then all is fine. I just don't like the idea of my backup being unavailable for a "period of time" but I may just have to live with it. You will see in my comments to Alex below, I will test WHS BDBB to see if the same screenshot conditions are present after a WHS BDBB backup.

Alex: I took your comment as humor and noted the afterwards. Mine was given back in humor with a wink.

Short answer to your quesiton is Yes. Long answer is any time I run a BDBB backup (with the script) wheather the screenshot conditions are present or not, I get the same screenshot conditions for "some" period of time after the script finishes processing.

Note my last comment to John above. When I read your quesiton about backing up BDBB, I think I misunderstood your quesiton. By my misunderestanding I got a idea to try. It would perhaps shed some light on the problem to use WHS BDBB to test and see if the screenshot conditions are present for "some" period of time after WHS BDBB has run. It could be a set of circumstances of my particular "configuration of the server" and testing wiht WHS BDBB may produce the same "failure mode". Of course this would take a bit of time as WHS BDBB is not incremental. If I do that do you want me to test with 1.0.1.9 or just try the one available from your forum post?

Thank you both for your assistince in nailing this issue down.
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)

I've been putting off doing that for some time now and I do apologize. (Not for the delay today, rather for the long delay since you posted it!)

YES, I see those errors when the script is running the backup database backup. Once it moves on to the shared folders the error clears immediately. Only tried it twice -- same each time.

So I tried a little experiment. I turned off the backup service without running the rest of the script and then looked for the same "problems" shown in your screenshots. All was normal on the Computers & Backup tab. In Settings-->Backup I immediately see the same error. Also unable to access the backups (as I would expect). I typed a couple lines here and looked at the Computers & Backup tab and the error is now present -- there is a delay there! Turned the backup service back on and after less than a minute my mail PC (wired in) was back to normal; about 15 seconds later the laptop (wireless) was back.

I believe the problem is that your system does not always quickly restore the backup service. I have definitely known the backup service to not start up immediately in the past, both using the script (XCOPY or RoboCopy) and using BDBB, so I do not think the occurrence is unusual -- but the length of your delays may be.

I do not know if this delay occurs in the EX48x or the LX195 or in DIY units, but I am beginning to think of it as a "bug".

WORKAROUND: Thurn it back on manually.

PARANOID WORKAROUND: Schedule the "turn on" script file to run a few minutes before your backup window.

Note: I have not yet seen a backup fail so maybe WHS turns it on just before the backups begin. (I really do not know. Maybe Alex can find out through his MVP contacts.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch