It is currently Tue Dec 11, 2018 8:51 pm

All times are UTC - 7 hours [ DST ]

Recent News:



Post new topic Reply to topic  [ 239 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 16  Next
Author Message
PostPosted: Wed May 13, 2009 10:40 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
I've certainly considered adding the ability schedule backups and do incremental backups. There's a couple of issues with this, though.

1. The problem with incremental backups is that most of the 4GB chunks are updated when Backup Cleanup runs over the weekend, so unless you had a utility that is capable of only updating changed blocks within the file then you'd frequently be copying the new changed 4GB chunks.

2. Related to point 1, since the backups are so large, without the ability to do incremental backups a scheduled backup mechanism isn't as useful since the backup can take hours and consume vast quantities of storage space. In my personal case, it would take several hours and only 2 or 3 backups would fill an attached 1TB drive. A possibility would be to replace the existing backup every time the scheduled job runs, but I'm not sure that solution would work for everyone.

3. I could utilize something like robocopy under the hood, however that would require manual user intervention to install Robocopy so that BDBB could use it, given that Robocopy is not redistributable from Microsoft.

Johns script (and your restore script) is basically what BDBB does, it stops the two needed services and then copies files, that's it. So your proposed script should in theory work just fine.

I hope that helps.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  

Attention Guest: Remove this ad by Registering with the MediaSmartServer.net Forums. It's Free!
PostPosted: Wed May 13, 2009 11:06 am 
Offline
Top Contributor
Top Contributor
User avatar

Joined: Tue Apr 01, 2008 5:33 pm
Posts: 902
Location: Florida
Thanks: 53
Thanked: 35 times in 35 posts
yakuza wrote:
I've certainly considered adding the ability schedule backups and do incremental backups. There's a couple of issues with this, though.

1. The problem with incremental backups is that most of the 4GB chunks are updated when Backup Cleanup runs over the weekend, so unless you had a utility that is capable of only updating changed blocks within the file then you'd frequently be copying the new changed 4GB chunks.

2. Related to point 1, since the backups are so large, without the ability to do incremental backups a scheduled backup mechanism isn't as useful since the backup can take hours and consume vast quantities of storage space. In my personal case, it would take several hours and only 2 or 3 backups would fill an attached 1TB drive. A possibility would be to replace the existing backup every time the scheduled job runs, but I'm not sure that solution would work for everyone.

3. I could utilize something like robocopy under the hood, however that would require manual user intervention to install Robocopy so that BDBB could use it, given that Robocopy is not redistributable from Microsoft.

Johns script (and your restore script) is basically what BDBB does, it stops the two needed services and then copies files, that's it. So your proposed script should in theory work just fine.

I hope that helps.


Yes, Alex. That does help. It gives me confidence to consider BDBB backups in the "restore script". It does leave me wondering about the registry stuff, however. It probably time for me to tell you of a problem I had with a BDBB Restore back in late March when I did a Factory Reset on my server. I have not said anything before because I am not sure it was not my fault that the Restore did not work as expected. What happend is when I restored the BDBB using the first Version of your BDBB Add-In, the BDBB was restored, but with errors. When I tried to repair the backup database using the WHS Console, all backups were lost on all machines. That is what I recall, however its possible that a few backups were ok for one machine. I am not really sure on this point.

Again, I did not report this to you before becuse it was not clear in my mind if that BDBB (that failed on restore) was not taken when I had a error in the Backup Database. I don't think that was true, because "why would I make a BDBB backup of a database that already had errors in it?" But I cannot rule that out and is why I have not mentioned it until now.

Just for clarification, I went to an older (about two months earlier) BDBB and it restored without errors and is the basis for my Backup database today. If course I have added daily backups to that. However, I did lose about 2 months of backup databases becauser the "newer" BDBB restore did not work.

One final point. Related to your futrue plans for the feature requests. When WHS v2 comes out it is my understanding it will use Server 2008 as the basis for WHS. Server 2008 has a copy of RoboCopy included so it would not have to be downloaded. As to today, except for the glitch you caught when I downloaded the ver 1.71 of RoboCopy (Not the current one as you pointed out-use the Resource Kit Toolbox instead that you pointed me to-XP010 is the correct ver of Robocopy for others who might be interested) I had no further problems getting it installed. I did place a copy of robocopy.exe in the c:\ root directory because batch files (scripts) will automatically look there for files referenced in the script.

My point? It would be well worth the effort (IMHO) for your add-in users to download RoboCopy and put the exe in the root directory of their servers to get incremental backups as an option for your very useful (already) add-in :D
George

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)


Top
 Profile  
Thanks  
PostPosted: Wed May 13, 2009 11:40 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
SeaRay33 wrote:
What happend is when I restored the BDBB using the first Version of your BDBB Add-In, the BDBB was restored, but with errors. When I tried to repair the backup database using the WHS Console, all backups were lost on all machines.

I've seen this occur when you attempt to interact with the DB before all updates are applied. Meaning, if you took a backup with BDBB, did a restore or recovery, and tried to interact with the DB before applying the PP1 and PP2 updates, the repair mechanism can delete all backups. I don't know if this is the cause of your failure, and it's a bit late to troubleshoot now, but if you see it again we can investigate. It's also possible that scenarios like this are why MS doesn't support a solution like BDBB and removed it from PP1. :-k

SeaRay33 wrote:
It would be well worth the effort (IMHO) for your add-in users to download RoboCopy and put the exe in the root directory of their servers to get incremental backups as an option for your very useful (already) add-in :D
George


We'll have to wait and see when Vail arrives, won't we? :) MS may deliver a feature that invalidates the need for BDBB, we just don't know yet. But as useful as having Robocopy on the server in its current form may be, I'm trying to follow a design principle of not requiring manual user intervention for my add-ins. This reduces my support load and provides a better user experience.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  
PostPosted: Wed May 13, 2009 11:51 am 
Offline
Top Contributor
Top Contributor
User avatar

Joined: Tue Apr 01, 2008 5:33 pm
Posts: 902
Location: Florida
Thanks: 53
Thanked: 35 times in 35 posts
yakuza wrote:
SeaRay33 wrote:
What happend is when I restored the BDBB using the first Version of your BDBB Add-In, the BDBB was restored, but with errors. When I tried to repair the backup database using the WHS Console, all backups were lost on all machines.

I've seen this occur when you attempt to interact with the DB before all updates are applied. Meaning, if you took a backup with BDBB, did a restore or recovery, and tried to interact with the DB before applying the PP1 and PP2 updates, the repair mechanism can delete all backups. I don't know if this is the cause of your failure, and it's a bit late to troubleshoot now, but if you see it again we can investigate. It's also possible that scenarios like this are why MS doesn't support a solution like BDBB and removed it from PP1. :-k

SeaRay33 wrote:
It would be well worth the effort (IMHO) for your add-in users to download RoboCopy and put the exe in the root directory of their servers to get incremental backups as an option for your very useful (already) add-in :D
George


We'll have to wait and see when Vail arrives, won't we? :) MS may deliver a feature that invalidates the need for BDBB, we just don't know yet. But as useful as having Robocopy on the server in its current form may be, I'm trying to follow a design principle of not requiring manual user intervention for my add-ins. This reduces my support load and provides a better user experience.


Point taken on the support and user experience. I have to agree with you.

Also agreed on the BDBB restore issue. I can not rule out that I did not try to repair the backup database before applying the PP1 update (PP2 was not out yet when I did the Factory Reset). I will be more careful about that if the situation arises again. If I run into problems, I'll be better about asking for help if the BDBB restore does not work as expected. Thanks!
George
P.S. Sure wish Vail would get here. Its like a kid waiting for Christmas :D

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)


Top
 Profile  
Thanks  
PostPosted: Wed May 13, 2009 2:25 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
ALEX: You want to avoid having to manually install RoboCopy -- how about just using XCOPY with the date option? I am finding that is very effective in this operation as the file dates are reliable -- AND the operation is faster with XCOPY than with RoboCopy!

(The two variations are documented in the Automated Server Backups thread.)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Thu May 14, 2009 1:19 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
I could do some benchmarking of File.Copy vs xcopy and see if one is faster. However given the way the files change during backup cleanup I'm not yet convinced that xcopy would buy much improvement. I'll keep it in mind...

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  
PostPosted: Thu May 14, 2009 2:37 am 
Offline
Top Contributor
Top Contributor
User avatar

Joined: Tue Apr 01, 2008 5:33 pm
Posts: 902
Location: Florida
Thanks: 53
Thanked: 35 times in 35 posts
yakuza wrote:
I could do some benchmarking of File.Copy vs xcopy and see if one is faster. However given the way the files change during backup cleanup I'm not yet convinced that xcopy would buy much improvement. I'll keep it in mind...

Alex, I might be wrong. But I think John is saying that using some kind of copy command (that looks at the dates of the 4GB backup chunks... he is suggesting XCOPY might be one such copy command) might allow your add-in to do incremental backups and and to do them on an schedule (say daily), i.e. the two feature request I was referring to in my post above. On Sundays because of backup/cleanup there would be no speed advantage as all blocks would need to be copied becase they all changed and Sunday would be a "long" day, but the six other days would be faster. The BDBB runs would be very large as you suggested each Sunday, but maybe the user could choose to "replace" the previous BDBB's instead of creating a "new" one. Each Sunday's BDBD would get only slightly larger (only because the PC backups themselves are growing due to more "days" being accumulated)

You reasons about support, etc. are still valid reasons if you don't want to do this. Again that is understood. But except for that (and I am guessing there may be other reasons you don't want to schedule daily and incremental BDBB runs that I don't know about) you could schedule your BDBB add-in and do it daily if the user so chose.

That is, John is talking about scheduled, incremental, daily backups of the BDBB, not speed as the primary user benefit.

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)


Top
 Profile  
Thanks  
PostPosted: Thu May 14, 2009 9:26 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
I understand the intent, I guess I didn't find as much value investing effort into a partial fix. I've got some ideas on this for a subsequent release, but am not actively working on this now.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 6:59 am 
Offline
Newbie
Newbie

Joined: Mon May 18, 2009 6:31 am
Posts: 2
Thanks: 0
Thanked: 0 time in 0 post
First Thank you for this Add in. I am a home user and not a technical professional. With that said I understand the importance of backing up. We have thousands of family pictures, videos etc...

I just starting using WHS and was very happy with the backup service. Like most people on these boards I was not happy with the restrictions around backing up the server backup itself.

Anyway I recently download Add-In version 1.0.1.6. I did my first backup last night and everything seemed to work okay.
Here is some information around my setup
1. Using HP Media Smart Server
2. Backing up to a 1tb WD external 1gb Network hard drive
3. On the “Server Storage” Tab within the WHS Console it says my PC backups are 109 GB in size.

Here is my question, I started the backup on Sunday night about 9:45 pm. When I checked it this morning it seems to still be running. The destination file is currently about 106GB and when I look at the WHS BDBB within the settings it says its 45% complete. While I don’t think it’s locked up (the Destination file has been increasing since I checked it this morning) it seems to be taking really long. I have been reading other post that mention backups that are a lot smaller and they seem to go a lot faster based on what they said in their posts. Does this time frame seem right to you (almost 11 ½ hours and still running)? Could this be caused by accessing this drive through my network? Its 1gb though….


Also can you let me know specifically which files I should see in the destination file once the backup is complete? Currently I see (REG folder, Commit.dat, control 4096.dat, data 4096.0.dat - data 4096.26.dat)

Thanks


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 7:59 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
cepp77,

Correct me if I'm wrong, but it appears from your post above that you are backing up to a network drive and not a drive attached to the MSS? If you are backing up the backup database via BDBB to a network drive, it could take awhile. In my tests, backups to an old Snap Server NAS took at least 3 times longer than backups to a backup drive attached to the MSS. When backing up to a network drive, there are a lot of factors that could affect the backup speed, including your router, switches, wired vs. wireless, the speed of the NAS, the ethernet adapters and settings in the system, etc.


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 9:07 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
cepp77, you are seeing the correct files and file structure, it sounds like things are progressing correctly if slowly.

As cavediver mentions, you're dependent on network factors. How fast is your switches/network cards/the network hard drive? If they aren't gigabit it's going to be quite slow.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 11:13 am 
Offline
Newbie
Newbie

Joined: Mon May 18, 2009 6:31 am
Posts: 2
Thanks: 0
Thanked: 0 time in 0 post
Guys thanks for the feedback. Shortly after my post the backup completed. I guess it took about 12 hours… As for my setup:
• Yes I am backing this up over my network to a shared drive connected directly into my router.
• DLink DIR-855 router with 1gb ports
• HP Media Smart Server with 1gb ports
• Western 1tb Digital World book (the drive I am using to back up WHS to) with 1gb port.

So it looks as though it completed successfully. I am almost sure that I have 1gb connectivity all the way through I guess it just takes that long through a network setup. Maybe I have some restrictions on my router that is slowing it down. Does anyone have an experience with how long it would take if I backed up 100gigs over usb or internally? Either way great Add in.


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 11:52 am 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
cepp77 wrote:
Guys thanks for the feedback. Shortly after my post the backup completed. I guess it took about 12 hours… As for my setup:
• Yes I am backing this up over my network to a shared drive connected directly into my router.
• DLink DIR-855 router with 1gb ports
• HP Media Smart Server with 1gb ports
• Western 1tb Digital World book (the drive I am using to back up WHS to) with 1gb port.

So it looks as though it completed successfully. I am almost sure that I have 1gb connectivity all the way through I guess it just takes that long through a network setup. Maybe I have some restrictions on my router that is slowing it down. Does anyone have an experience with how long it would take if I backed up 100gigs over usb or internally? Either way great Add in.


My backup database is a little over 110 GB. It takes about 1.5 hours to do a BDBB backup to a 500 GB drive attached to my MSS via USB. It takes about 1.5 hours to backup to a network shared esata drive attached to my main PC (gigabit network connection). It takes about 7 hours to backup to an old Snap Server NAS with raid 1 mirroring and a 10/100 network card.

Your shared drive may only have a 10/100 network adapter in it which would explain the slow transfer.


Top
 Profile  
Thanks  
PostPosted: Mon May 18, 2009 11:55 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10690
Location: Fort Collins, Colorado
Thanks: 622
Thanked: 915 times in 871 posts
cavediver wrote:

Your shared drive may only have a 10/100 network adapter in it which would explain the slow transfer.


This would be my suspicion as well.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins? If so, be sure to Rate them at WeGotServed.


Top
 Profile  
Thanks  
PostPosted: Thu May 21, 2009 10:23 am 
Offline
Newbie
Newbie

Joined: Thu May 21, 2009 10:05 am
Posts: 4
Location: Sweden
Thanks: 1
Thanked: 0 time in 0 post
Just installed the BDBB Add-In. Exactly what I have been looking for. Thank you! :)

While investigating what backup strategy will work best for me I enabled the duplicate WHS backups. It doesn't seem as if it started duplicating immediately. Is this something that will happen next Sunday? How do I know that the WHS backups have been duplicated? Will it show up in the pie chart in the server storage tab in WHSC?


Top
 Profile  
Thanks  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 239 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6 ... 16  Next

All times are UTC - 7 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group