It is currently Thu Mar 28, 2024 1:49 pm

All times are UTC - 7 hours [ DST ]

Recent News:



Post new topic Reply to topic  [ 307 posts ]  Go to page 1, 2, 3, 4, 5 ... 21  Next
Author Message
 Post subject: Automated Server Backups
PostPosted: Fri Jan 23, 2009 8:35 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
Would you like to have a process that will back up all your shared folders onto a WHS Unmanaged Disk (generally external) on a pre-arranged schedule of your choosing? Would you like to also copy the WHS backup database? If you are interested, keep reading...

To start you will need to install Robocopy, which can either be installed as part of the Windows 2003 Server resource kit, or by itself, from the link at the top of this article: http://technet.microsoft.com/en-gb/magazine
/2006.11.utilityspotlight.aspx . (The downloaded file is a self-extracting ZIP. Extract the files to a temporary location on your WHS server and then run the setup. RoboCopy will be installed. The article describes RoboCopy.

Then you will need to know the disk to which you wish to save the backups (E:, F:, etc). The script ass-u-me-s that the target drive is E: and needs to be modified in five places if that is to be changed (one early comment line and then for the two RoboCopy and two Xcopy command lines).

Copy the code below into a file to save in your WHS. I have named this Automatic.bat and saved in the root directory of the c: drive (C:\).

Added 19 Jan 10: You also need to add the target directory for the log files that are created (or modify the script to not create them -- NOT recommended). The default target directory to be created is C:\WHS Backup Logs\. Note that the first few executions WILL generate errors for the "rename" commands as prior logs do not exist. These may be safely ignored. (THANKS to "Lazar" for pointing these out!)

Test the code on your server before automating it, running it from a remote desktop. (Can either do this from a command prompt or by navigating to the file and double-clicking.) Be aware that on the first use the entire process will probably take a goodly number of hours. After the initial backup is complete nightly backups are MUCH faster. Initial use for me was around 20 hours, daily backups now take 30-40 minutes. (The Sunday backups do take considerably longer as the backup database is significantly reconfigured as part of the weekly cleanup; for me this is about 7 hours.) All of this is in the background and the server can be used as this is running although I would avoid having this overlap with the nightly backup window.

Each time the routine is run the RoboCopy progress is captured in a pair of log files, one for the backup database and one for the shared folders. Nine logs are kept for historical reference with the oldest being erased each time and the others renamed. They will be stored in the C:\ root directory.

AFTER the initial use of the routine and after you are comfortable with letting it run unattended, this can be scheduled to execute at intervals of your choice. (I run this 30 minutes after my nightly backup window is over, in the morning hours when there is little load on my swerver.) You can do the scheduling from Start --> Control Panel --> Scheduled Tasks --> Add Scheduled Task

The following code is the most current (as of 7 July 09) and will be updated as it changes. The code segment immediately following this is the original and other code in this thread may not be current.

Edited 18 Jan 10 to change the default target drive to E: to match the original.

Code:
REM Windows Home Server backup of backup folder and then shares folder

REM This assumes the target drive is the E-Drive (E:). Change targets appropriately if using a different drive.

REM This currently defaults to using XCOPY. If the use of RoboCopy is preferred comment out the XCOPY lines (2) and
REM remove the comments ("REM ") from the Robocopy command lines (2). Note that RoboCopy must be installed to use Robocopy!

REM A Critical Error will be present on the server when the Backup Database is being backed up as the Backup Service will be shut down.
REM (This may be safely ignored as it will be restarted at the completion of the database backup -- before the Shared Folders are copied.)


REM Log files will be created in C:\WHS Backup Logs\

REM (C) 2008 www.HomeAutomation-Direct.com - Modified by JohnBick December 2008, Jan 2009, May 2009
REM Released under GPL. No consequential loss or damages accepted.  Use at your own risk!
REM WHS will complain if the services are stopped for too long!

REM ------------------

REM Copy the WHS CLIENT BACKUP Database

REM Stop the Drive Letter Service
NET STOP PDL

REM Stop the Backup Service
NET STOP WHSBackup

REM Select one copy method OR the other below.

REM ROBOCopy the "backup" folder in Mirror Mode, Restartable, All attributes, No progress, 4 retries, 5 second wait between retries, Log to file, Console output
REM Robocopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} E:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /MIR /Z /COPYALL /NP /R:4 /W:5 /LOG:"C:\WHS Backup Logs\External-Backups.log" /TEE

REM Copy the "backup" folder
XCopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} E:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /D /E /V /C /I /G /H /R /K /X /Y >"C:\WHS Backup Logs\External-Backups.log"

REM Start the Backup Service
NET START WHSBackup

REM Start the Drive Letter Service
NET START PDL 

REM -------------------

REM Rename the log files to save last TEN (0-9)

erase  "C:\WHS Backup Logs\External-Backups-9.log"
rename "C:\WHS Backup Logs\External-Backups-8.log"  External-Backups-9.log
rename "C:\WHS Backup Logs\External-Backups-7.log"  External-Backups-8.log
rename "C:\WHS Backup Logs\External-Backups-6.log"  External-Backups-7.log
rename "C:\WHS Backup Logs\External-Backups-5.log"  External-Backups-6.log
rename "C:\WHS Backup Logs\External-Backups-4.log"  External-Backups-5.log
rename "C:\WHS Backup Logs\External-Backups-3.log"  External-Backups-4.log
rename "C:\WHS Backup Logs\External-Backups-2.log"  External-Backups-3.log
rename "C:\WHS Backup Logs\External-Backups-1.log"  External-Backups-2.log
rename "C:\WHS Backup Logs\External-Backups-0.log"  External-Backups-1.log
rename "C:\WHS Backup Logs\External-Backups.log"    External-Backups-0.log

REM Backup Database backed up, backup service has been restarted, and logs saved

REM -------------------

REM Backup the WHS SHARES folder

REM Select one copy method OR the other below.

REM Copy the “shares” folder in Mirror Mode, Restartable, All attributes, No progress, 4 retries, 5 second wait between retries, Log to file, Console output
REM Robocopy D:\shares E:\Shares /MIR /Z /COPYALL /NP /R:4 /W:5 /LOG:"C:\WHS Backup Logs\External-Shares.log" /TEE

REM Copy the WHS Shared Folders
XCopy D:\shares E:\Shares /D /E /V /C /I /G /H /R /K /X /Y >"C:\WHS Backup Logs\External-Shares.log"

REM -------------------

REM Rename the log files to save last TEN (0-9)

erase  "C:\WHS Backup Logs\External-Shares-9.log"
rename "C:\WHS Backup Logs\External-Shares-8.log"  External-Shares-9.log
rename "C:\WHS Backup Logs\External-Shares-7.log"  External-Shares-8.log
rename "C:\WHS Backup Logs\External-Shares-6.log"  External-Shares-7.log
rename "C:\WHS Backup Logs\External-Shares-5.log"  External-Shares-6.log
rename "C:\WHS Backup Logs\External-Shares-4.log"  External-Shares-5.log
rename "C:\WHS Backup Logs\External-Shares-3.log"  External-Shares-4.log
rename "C:\WHS Backup Logs\External-Shares-2.log"  External-Shares-3.log
rename "C:\WHS Backup Logs\External-Shares-1.log"  External-Shares-2.log
rename "C:\WHS Backup Logs\External-Shares-0.log"  External-Shares-1.log
rename "C:\WHS Backup Logs\External-Shares.log"    External-Shares-0.log

REM Shared Folders backed up and logs saved



This is the original code:
Code:
REM Copy WHS Backup Database and Shared Folders to an unmanaged disk drive
REM This script assumes the target is Disk E: -- modify script in two places if that is to be changed
REM The logs are placed in the C:\root directory. That can also be changed (22 places)
REM A Critical Error will be present on the server when the Backup Database is being backed up as the Backup Service will be shut down.
REM       - This may be safely ignored as it will be restarted at the completion of the database backup


REM Backup the WHS BACKUP DATABASE

REM Stop the Backup Service
NET STOP PDL
NET STOP WHSBackup

REM Copy the "backup" folder in Mirror Mode, Restartable, All attributes, No progress, 4 retries, 5 second wait between retries, Log to file in c:\temp, Console output
Robocopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} E:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /MIR /Z /COPYALL /NP /R:4 /W:5 /LOG:"C:\External-Backups.log" /TEE

REM Start the Backup Service
NET START WHSBackup
NET START PDL 

REM Rename the log files to save last NINE
erase  C:\External-Backups-1.log
rename C:\External-Backups-2.log  External-Backups-1.log
rename C:\External-Backups-3.log  External-Backups-2.log
rename C:\External-Backups-4.log  External-Backups-3.log
rename C:\External-Backups-5.log  External-Backups-4.log
rename C:\External-Backups-6.log  External-Backups-5.log
rename C:\External-Backups-7.log  External-Backups-6.log
rename C:\External-Backups-8.log  External-Backups-7.log
rename C:\External-Backups-9.log  External-Backups-8.log
rename C:\External-Backups.log    External-Backups-9.log

REM Backup Database backed up, backup service has been restarted, and logs saved


REM Backup of the SHARES folder

REM Copy the shares folder in Mirror Mode, Restartable, All attributes, No progress, 4 retries, 5 second wait between retries, Log to file in c:\temp, Console output
Robocopy D:\shares E:\Shares /MIR /Z /COPYALL /NP /R:4 /W:5 /LOG:"C:\External-Shares.log" /TEE

REM Rename the log files to save last NINE
erase  C:\External-Shares-1.log
rename C:\External-Shares-2.log  External-Shares-1.log
rename C:\External-Shares-3.log  External-Shares-2.log
rename C:\External-Shares-4.log  External-Shares-3.log
rename C:\External-Shares-5.log  External-Shares-4.log
rename C:\External-Shares-6.log  External-Shares-5.log
rename C:\External-Shares-7.log  External-Shares-6.log
rename C:\External-Shares-8.log  External-Shares-7.log
rename C:\External-Shares-9.log  External-Shares-8.log
rename C:\External-Shares.log    External-Shares-9.log

REM Shared Folders backed up and logs saved


Future Plans

In the near future I will post an expanded version of this script that uses TrueCrypt to encrypt the target drive and add additional protection for the off-site storage of the backup. When I do that I will edit this post and add a link to that thread here.

If there are any changes to the script above I will update this post to make the changes.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
The following user would like to thank JohnBick for this post
mwtea

Attention Guest: Remove this ad by Registering with the MediaSmartServer.net Forums. It's Free!
PostPosted: Sat Jan 24, 2009 1:33 pm 
Offline
Max Contributor
Max Contributor

Joined: Fri Apr 11, 2008 7:14 am
Posts: 1540
Thanks: 3
Thanked: 117 times in 100 posts
John,

Is there any way you can turn this process into an add-in for the MSS?


Top
 Profile  
Thanks  
PostPosted: Sat Jan 24, 2009 2:34 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
CAN it be done? Probably... But remember, once you have this working there is no interaction required.

Can I do it? No. It has been far too many years since I did any coding and I really have no interest in updating those skills! (If anyone is interested in taking it that additional step let's talk. All that is really needed is to install a file in C:\ and schedule a task. But dfor the encrypted version there WILL be additional steps.)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Mon Mar 02, 2009 8:41 pm 
Offline
.5TB storage
.5TB storage

Joined: Wed Feb 18, 2009 8:23 pm
Posts: 11
Thanks: 0
Thanked: 0 time in 0 post
Thank you for posting this your solution.

I don't understand something about the way Robocopy operates. On my server, the WHS Backup Database files in D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} are all dated today, from the early hours when PC backups ran. I imagine that many of these files are the same from one day to the next, and WHS just updated their times rather than copied them from the PC's at every backup.

Does Robocopy recognize that an existing destination file is identical to a source file even if the two files have different dates and times? If so, does it only update the timestamp on the destination file rather than copy the source to the destination?

If Robocopy does that, it is even more clever than its documentation indicates.

Nancy


Top
 Profile  
Thanks  
PostPosted: Fri Mar 06, 2009 4:19 am 
Offline
Top Contributor
Top Contributor
User avatar

Joined: Tue Apr 01, 2008 5:33 pm
Posts: 902
Location: Florida
Thanks: 53
Thanked: 35 times in 35 posts
John,
I had somehow missed this thread and was re-directed to it from a thread you are I are both watching. Thank you for this solution. I am now watching your Automated Server Backups thread via Subscribe Topic.

I like the automated process. I already have WHS BDBB installed and working but it is not automated as your script is. I will probalby just continue with BDBB for at least a while because of inertia. But I am glad I have found this solution and I will, in the future, definitely recommend it to someone, along with Alex's BDBD Add-In to someone who may be needing to look at ways to backup the WHS Backup Database. They can choose thier perferred method then. It definitely has some advantages over BDBB. One disadvantage (which may not actually be a disadvantage at all because it just dosen't occur once you set it up) is the dependency on a drive letter NOT changing. I guess if it does you just fix that like any other error you encounter by changing the script or fixing the reason the drive letter assignment might have changed.

I have heard of RoboCopy and never investigated it. I have been using TeraCopy interactively and like many of its features (including my "perhaps perceived"[haven't done any speed testing] notion that it moves files to and from the server faster that the the Windows OS does, but I have not looked to see if it has a command line capability like RoboCopy does and, of course, that would be necessary if you are to use it in a command file.

One final comment. I guess you are watching this thread, so I wondered why you had not replied to Nancy's question in post 4?

Thanks for a great idea and documentation of same. I like command files and this is one that meets a big need since PP1 promised but did not deliver on the WHS Backup file option. Thank you for going to the trouble of building the scrpt and for educating me on its use with this thread.

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)


Top
 Profile  
Thanks  
PostPosted: Mon Mar 09, 2009 11:41 am 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
NANCY: The backup "Data.4096.xx.dat" files act a bit differently than typically expected. As client files are changed data is added to the backup database and new files are created. Then, generally early Sunday morning, the cleanup of the monthly-weekly-daily backups occurs and the entire backup database is reorganized (my wording) in a manner that requires each of these files to be re-written. After that the dates will all be the same (with the Sunday morning timestamp). New daily backups will cause the LAST (highest numbered) of these files to be rewritten until it is "full", after which a new one will be created. This continues until the next cleanup on the next Sunday morning. At the moment I have 99 of these backup files. All of them get changed each Sunday morning. By the next Sunday I may have 100 or 101 of them which are again consolidated to 99 and the process continues. (When I started watching these the total was 95; it gradually increases along with the increased use of client storage.)

So yes, one of these files will change daily (along with many of the smaller files in that folder), an occasional new file will be added daily and one\ce a week they will all change. RoboCopy correctly copies only the new and changed files, a small number and quick process during the week and a very long process (for me) on Sunday. Monday through Saturday the entire process (backups + shares) takes about 1/2 hour for me. On Sunday it takes 10-12 hours.

I do not know id RoboCopy sees differences in the files or relies on the timestamps, but in this application it does not matter. (Some programs that synchronize files actually do recognize files as identical even if the timestamps are different. I have not tested RoboCopy to see how it handles that case.)

My apologies for the delay in responding. I missed your post while trying to catch up after travelling for a few weeks and finally saw it this morning.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Mon Mar 09, 2009 11:48 am 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
George: I THOUGHT I was watching the thread but got a little behind...! (Just answered Nancy's post!)

I have experimented with RoboCopy, SyncToy and some others. I did not use SyncToy as it creates a file in the source and target directories and I did not want to do that for this application. Feel free to experiment!

As you indicated, the script can be edited to adjust the drive letter. Can split the script up to put the shares on one drive and the backups on another. Can even split up the shares if necessary!

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Mon Mar 09, 2009 8:10 pm 
Offline
.5TB storage
.5TB storage

Joined: Wed Feb 18, 2009 8:23 pm
Posts: 11
Thanks: 0
Thanked: 0 time in 0 post
John,
Thank you for the wonderful reply!
Nancy


Top
 Profile  
Thanks  
PostPosted: Tue Mar 17, 2009 12:57 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
On another board I was asked about the pros and cons of the WHS (PP1) backups as compared to the method outlined here. This was my response....

The WHS server backup is manual, in a backup database and for Shared folders only, but it does keep a history of changes over time. The result requires a WHS system to access the backups. (The WHS BDBB Add-In can separately provide a backup of the client database.)

The approach outlined here can be automated, is in NTFS format, can include both the Shared Folders and the client backup database, and the Shared Folders can be read by any PC as they are on a normally formatted disk. But there is no "history" of changes to the Shared Folders, only an updated copy of what was in them at the last time the routine was executed. (It uses a synchronization tool, RoboCopy, to duplicate the contents of the WHS server.) To have different points in time one needs to rotate copies of the disk. Note that the backup database is copied intact and will contain the time-history of the clients, but this will require a WHS system to extract the data after it is restored to a WHS server.)

It really comes down to whether you want to have a history for the Shared Folders or the reliability of automation. At this level of a backup I value automation over history, liking to have the server backed up each morning. When I want to take a copy off-site I just remove the drive and replace it with another. The content of my Shared Folders is fairly stable (as it likely is for the vast majority of WHS users) and history is not particularly important. (It is even less important if "working copies" are retained on one of the clients, in which case the versions over time are available in the client backups.) And there is, of course, a history based on the frequency of the exchange of backup disks.

But, other than the cost of a hard drive, there is nothing preventing you from doing both.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Wed Mar 25, 2009 10:50 am 
Offline
Newbie
Newbie

Joined: Wed Oct 08, 2008 4:34 pm
Posts: 4
Thanks: 0
Thanked: 0 time in 0 post
I would like to create an offsite "buddy backup" system and wondering if this is a possible scenario:

∙ Two WHS servers in two locations.
∙ An initial backup is made to a local disk, as described in this thread for both servers.
∙ Each disk is swapped out and hand-carried to the other offsite server.
∙ Subsequent incremental backups are made across the Internet.

Seems that would get around the initial bandwidth demands by using sneakernet for the initial backup.
Anyone see a simple way to do so in an automated way?


Top
 Profile  
Thanks  
PostPosted: Wed Mar 25, 2009 1:10 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
jpinsl wrote:
I would like to create an offsite "buddy backup" system ...

YES, this is very possible. (NO, I have not done it though.)

HOW, you ask? The key is to set up a VPN between the two systems so that the two local networks appear as one. There is discussion of a "freebie" VPN in this local thread: http://www.mediasmarthome.com/forum/thread/11714/WHS-streaming-from-and-XBOX-360-setup/ . (Another link is inside that thread.)

Mount the disk on the other server and share it locally. It should then be available via the VPN.

A few things to to be aware of:

    1. The copy of the client backup database will essentially totally refresh every Sunday morning as the older daily/weekly/monthly copies are removed and the database is reorganized. These deletions will cause (almost) the entire database to be rewritten. Such is NOT the case with the Shared Folders and this should be fine for them. And it may not matter for the backups if your backup database is small -- but most aren't.
    2. The VPN is (probably) not two-way, it is most likely unidirectional. Your end would be the client, initiating the session and having access to his system, not the reverse. To do it both ways you would need to set it up both ways on both ends.
    3. You will have complete access to his LAN and everything he has shared on the LAN. Big security risk for him to let you have that access.
    4. For this version (unencrypted) he would have full access to everything on your backup copy. As stated before, I expect to post instructions for an encrypted version (soon!).
I have not tried this over a VPN but I have tried it to a shared drive on another computer.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
PostPosted: Sat Apr 11, 2009 10:39 pm 
Offline
MVP/Moderator
MVP/Moderator
User avatar

Joined: Sat Dec 15, 2007 8:21 am
Posts: 4725
Location: Dutchess County, NY
Thanks: 77
Thanked: 273 times in 268 posts
At the suggestion of a user (Searay33/George) I have modified the procedure to use XCOPY instead of RoboCopy:

Code:
REM Copy WHS Backup Database and Shared Folders to an unmanaged disk drive
REM This script assumes the target is Disk E: -- modify script in two places if that is to be changed
REM The logs are placed in the C:\root directory. That can also be changed (22 places)
REM A Critical Error will be present on the server when the Backup Database is being backed up as the Backup Service will be shut down.
REM       - This may be safely ignored aas it will be restarted at the completion of the database backup


REM Backup the WHS BACKUP DATABASE

REM Stop the Backup Service
NET STOP PDL
NET STOP WHSBackup

REM Copy the "backup" folder

XCopy D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} E:\Backups\{00008086-058D-4C89-AB57-A7F909A47AB4} /D /E /V /C /I /G /H /R /K /X /Y >C:\Temp\External-Backups.log

REM Start the Backup Service
NET START WHSBackup
NET START PDL 

REM Rename the log files to save last NINE

erase  C:\Temp\External-Backups-1.log
rename C:\Temp\External-Backups-2.log  External-Backups-1.log
rename C:\Temp\External-Backups-3.log  External-Backups-2.log
rename C:\Temp\External-Backups-4.log  External-Backups-3.log
rename C:\Temp\External-Backups-5.log  External-Backups-4.log
rename C:\Temp\External-Backups-6.log  External-Backups-5.log
rename C:\Temp\External-Backups-7.log  External-Backups-6.log
rename C:\Temp\External-Backups-8.log  External-Backups-7.log
rename C:\Temp\External-Backups-9.log  External-Backups-8.log
rename C:\Temp\External-Backups.log    External-Backups-9.log

REM Backup Database backed up, backup service has been restarted, and logs saved


REM Windows Home Server backup of the SHARES folder

REM Copy the “shares” folder

XCopy D:\shares E:\Shares /D /E /V /C /I /G /H /R /K /X /Y >C:\temp\External-Shares.log

REM Rename the log files to save last NINE

erase  C:\Temp\External-Shares-1.log
rename C:\Temp\External-Shares-2.log  External-Shares-1.log
rename C:\Temp\External-Shares-3.log  External-Shares-2.log
rename C:\Temp\External-Shares-4.log  External-Shares-3.log
rename C:\Temp\External-Shares-5.log  External-Shares-4.log
rename C:\Temp\External-Shares-6.log  External-Shares-5.log
rename C:\Temp\External-Shares-7.log  External-Shares-6.log
rename C:\Temp\External-Shares-8.log  External-Shares-7.log
rename C:\Temp\External-Shares-9.log  External-Shares-8.log
rename C:\Temp\External-Shares.log    External-Shares-9.log

REM Shared Folders backed up and logs saved


Advantages:

1. This is MUCH faster, taking about 30% of the time (EDITED to fix typo)
2. There is no need to obtain and install RoboCopy

Disadvantages:

1. RoboCopy is more rigorous in terms of being able to retry operations if a file is unavailable, etc.
2. RoboCopy produces a richer report format
3. This depends on time-stamps to determine if a file is newer

Certainly there are trade-offs between these alternative procedures. Were they to take approximately the same amount of time to execute I would have no hesitation to stay with RoboCopy exclusively. But time is a factor, and users with more data were worried that the procedure may, in time, begin to run into the next backup window on Sundays. I have begun to use the XCOPY version for my personal backups due to the speed of execution.

Discussion and feedback is encouraged....

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch


Top
 Profile  
Thanks  
The following user would like to thank JohnBick for this post
SeaRay33
PostPosted: Thu Apr 23, 2009 9:06 pm 
Offline
Newbie
Newbie

Joined: Thu Apr 23, 2009 9:03 pm
Posts: 3
Thanks: 1
Thanked: 0 time in 0 post
cavediver wrote:
Is there any way you can turn this process into an add-in for the MSS?

This Add-In does a semi-automated backup. It would be nice to see to these 2 projects merged with the best of both (add-in plus automated scheduling). http://www.mediasmartserver.net/2009/04 ... -released/


Top
 Profile  
Thanks  
PostPosted: Fri Apr 24, 2009 8:36 am 
Offline
Founder
Founder
User avatar

Joined: Tue Jul 31, 2007 11:54 am
Posts: 10735
Location: Fort Collins, Colorado
Thanks: 630
Thanked: 931 times in 885 posts
John and I have discussed this, I understand the interest in scheduling BDBB backups but there are a few issues that need to be taken into account. I may add this one day but don't have plans at this time. The reality is that John's script largely does most of what BDBB does, it just takes a little more technical aptitude.

_________________
Alex Kuretz
Remote Notification: Monitoring and alerting for your Home Server via email, Twitter, text message, RSS, push notifications, and on your Mac desktop
Have you tried our Add-Ins?


Top
 Profile  
Thanks  
PostPosted: Sun May 03, 2009 11:24 pm 
Offline
Top Contributor
Top Contributor
User avatar

Joined: Tue Apr 01, 2008 5:33 pm
Posts: 902
Location: Florida
Thanks: 53
Thanked: 35 times in 35 posts
JohnBick wrote:
Discussion and feedback is encouraged....

Hello John,
I’m going to be doing some testing with your Automated Batch file. However, I have some specific needs that are not met by your current solution. Even though my needs may be specific to my share folders I want to discuss them here because I think my Permanent Archive Server Backup needs may not actually be that unusual when compared to others. Let’s see if I’m right. If I am, some modifications to your batch file for folks with needs similar to mine might be in order. I am not suggesting that you do the modifications. I have some specific suggestions (with proposed answers) to get the discussion going and if we are lucky it may produce some potentially useful updates to you existing solution. I hope you (and any others who care to) comment will so that I might be guided to the best possible modification to your “already pretty darn good” solution.

Here’s my problem and proposed starting point solution.

I have some relatively static folders with small files that lend themselves well to your existing solution. But I also have some very active folders with lots of 2 to 4 gigabyte files that contain Recorded TV (RTV). I do a lot of "time shifting" for TV series and movies and sometimes it may be months before I get to some the "time shifted" RTV. Some of those RTV folders contain files I want to permanently retain and this works well with your current solution. For instance, a PBS series on The American Relovution might be archived on the server but many RTV files are not important enought to permanently retain.

I think the solution for these "mixed needs folders of very large files" is to use the /Exclude option (using you XCOPY batch file) to exclude the folders where I don’t need permanent copies of all the files then use Sync Toy with the full synchronization option on those folders. That way when I delete files I don’t want to keep after watching they are automatically deleted from the Sync Toy backup folder instead of going into permanent archive.

I think I have the proper syntax in mind for the modifications but I am open to other suggestions and would certainly appreciate your help in verifying my logic here.

If I just modify your XCOPY parameters to include a portion of the string in the folders I want to exclude using the syntax
/EXCLUDE: folder1[+folder2][+folder3]... where folder1,etc. specifies a list of folders containing strings. When any of the strings match any part of the absolute path of the folder to be copied, that folder will be excluded from copying. For example, specifying a string like \obj\ or .obj will exclude all files underneath the directory obj or all files with the .obj extension respectively. Credit for this paragraph goes to this link.

Then those Share Folders will not be backed up by your batch file. Is that your understanding? A seperate SyncToy operation (using Windows Task Manager) would be scheduled daily for the excluded folders.

For instance given the Share Folders below:

Attachment:
File comment: SeaRay's Current set of Share Folders
ScreenHunter_07 May. 03 21.30.gif
ScreenHunter_07 May. 03 21.30.gif [ 57.96 KiB | Viewed 41067 times ]


To use and example, I could exclude two of my Share Folders, eBooks and Recorded TV, by including the parameter /Exclude: /eBooks+/Rec in your existing command line parameters.

Then set Task Manager (Edit: Per JohnBick's comment below, yes I did mean to say Task Scheduler here, not Task Manager) to call a SyncToy sync task for each folder excluded with a target set on a specific external "drive letter" hard drive.

An added advantage to your currert solution or the proposed solution as well is that once backups are in production and tested, you could turn Duplication OFF for most or maybe even all folders thereby substantially extending the space for your Data Pool files "inside the server" since "outside" server drive letter drives provide duplication. However, some would argue (cavediver, for one, I expect) that you would be substantially burdened if a MSS or tower drive died. They would argue it would take a lot of time to determine which files were lost from the failed drive and to retrive same from he drive letter HDDs. Well that is a discussion for anothe thread. :lol:

One final though. It occurs to me that this /Exclude command line parameter could also be useful in testing where you might like to perform a "time reduced" test of the bacth file by excluding all Share Folders except your smallest size folder for the test.

All comments from anyone are welcomed.

UPDATE: As I was checking this post before release, it just occurred to me that I don't even need to run Sync Toy on the /Exclude"ed" folders. Turning Duplication On for those folders should suffice. If I am really hyper about losing the Excluded folders I could take the extra step to Sync Toy those folders to a "Synced" archive to protect against catastrophic failure. But If I am willing to take some risk with the excluded folders by using Duplication On that would save a good bit of daily "production" time. If I feel “safe enough” with that solution is my logic right here?

After having made this statement, now I am waffling on that decision. I guess it does add the extra security I was seeking for permanent backup by doing the Sync Toy and if the server crashes and needs a Factory reset or if catastrophic events occur I am covered. I think it is getting too late. Its well after midnight here and I’m going to bed. :D

_________________
It is the mark of an educated mind to be able to entertain a thought without accepting it.
Aristotle - Greek critic, philosopher, physicist, & zoologist (384 BC - 322 BC)


Last edited by SeaRay33 on Mon May 04, 2009 8:24 am, edited 1 time in total.

Top
 Profile  
Thanks  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 307 posts ]  Go to page 1, 2, 3, 4, 5 ... 21  Next

All times are UTC - 7 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 13 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group