I have no idea on the client database backups. My Windows machines are all VMs running on Macs (different configurations of Server 2003 and 2008 mainly). I periodically zip archive the VMs and copy the files to the server. The server seems unhappy if it doesn't have a Windows client to backup, so I throw it a bone now and then and let it back up a Win7 VM. I have no idea what the client backup add-in creates in the way of files. That would be something you need to explore.

What are you contemplating in terms of non-shared folders?



Richcopy is threaded, so you can copy more than one file at once. 5-6 seems to be about the optimum. There is no richcopy add-in for WHS. I haven't looked for one for robocopy - does one exist? FWIW, the WHS Console is one overrated piece of software.

I don't understand what you hope to accomplish with your music folder.

_________________
Synology DS411: 2 * WD20EARX and 2 * WD20EARS
DIY Small Business Server 2011
MP 8-core 2.26GHz, 2012 MBA13 i7/8GB/256GB, AppleTVs, iPhone

well i just have a bunch of private folders that arent shared folders. can i set those to get backed up as well? or does richcopy just backup the shared folders?

EDIT: ok jsut installed richcopy

how long will my first back up be of my 160 gig music folder? and will it just copy over as a regular folder?

Richcopy (or robocopy - from the page I linked to I gather that richcopy is simply a refinement of robocopy with an integral GUI and threaded architecture) is a file copy program. It doesn't care whether the files or folders are shared or not.

(see my edit in my prior post)

_________________
Synology DS411: 2 * WD20EARX and 2 * WD20EARS
DIY Small Business Server 2011
MP 8-core 2.26GHz, 2012 MBA13 i7/8GB/256GB, AppleTVs, iPhone

yea, a WHS add in for robocopy does exist but i agree with you, its kind of pointless. why not RDP and just use it that way.

i basically want my music folder on my destination drive (my unmanaged drive) to be an EXACT, mirrored copy of the music folder in my shared drive.

so if i delete a song, when i backup using richcopy, i want to make sure that song is also removed on the destination drive. i believe this is the PURGE option in richcopy?

i also wish to make sure that if i noticed a typo in a song name (via the mp3's ID tag) and if i edit it, when i back up using richcopy, that it will reflect that change on the destination drive.

and also only copy over new or changed files and not overwrite identical files. im reading thru the help file as we speak but any extra guidance would be greatly appreciated

i believe most of the options mentioned above are in the COPY IF... section.

also, do you check PRE PROCESS under the MODE section? read the help file on that...not exactly sure if checking it would help or slow down the process

EDIT: NEVERMIND...apparently you cant create directories on the root drive with RichCopy

ok im doing a test run and i copied my LED ZEPPELIN folder in MUSIC into the shared MAC folder (i dont have a mac so i figured id use it as a test folder)

anyway, for the source i select the whole MAC folder and for the destination, i just choose the root E:\

however, when i do the copy, i get an ACCESS IS DENIED error in rich copy but then it goes on and continues to copy the LED ZEPPELIN folder onto E:\

so rather than getting E:\MAC\LED ZEPPELIN i just have E:\LED ZEPPELIN

i cant copy any of the folders actually, music, videos, etc. whats up with this?

im logged in as administrator and all my permissions seem to be set up correctly

any ideas?

ok so im running my test and when i delete a file or folder from the source disk, it doesnt remove that folder from the destination disk when i run richcopy

i have selected PURGE in options which, reading the help section, seems to do exactly what i addressed above. but for some reason, the deleted folder is still there and not being deleted

however, added files get copied over. just not files deleted being removed

EDIT: further testing....FILES that i delete from the source actually DO get reflected in the destination however, whole FOLDERS that i delete do not. whats up with that?

EDIT 2: just noticed, the folder's contents actually DID get deleted however the actual folder did not

so to clarify, i delete HOUSE OF THE HOLY on the source drive (which is the folder HOUSES OF THE HOLY and within...all the songs of the album) and then backup using Richcopy. the folder HOUSES OF THE HOLY still remains however all the files (songs) within that folder get deleted. but the actual folder HOUSES OF THE HOLY still remains. is this a glitch? it recognizes the files are not there and deletes them but doesnt delete the actual folder

EDIT 3: so it seems to do everything (sans the edit 2 part above) that i need however, the moment i back up another folder and then go back to the original folder, it seems to forget everything and re-copy all the contents over again. when nothing has changed on the source. why doesn't it remember or realize nothing has changed? in other words, if i backup my LED ZEPPELIN folder and change, add, delete files and each time i hit back up, it does everything accordingly and remembers and applies everything. however, if i switch source folders and back up THE BEATLES and finish that process then switch BACK to the LED ZEPPELIN folder and hit backup, it re-copies every file and folder over even tho nothing has changed. its like it completely forgot it had just backed it up previously

can you NOT change source folders like...ever? then it will forget everything it had done?

You are correct, folder contents are deleted while folder structures are not. I tested this behavior with a GUI-driven version of robocopy and it deleted the folders along with the content. Richcopy definitely has a defect in this regard and robocopy is a safer choice. I will likely use robocopy going forward.

My folder structures are fairly stable, particularly those in my photo library (date-based: {year}/{month}/{day}) and media collection (iTunes-controlled). New nodes are added to these structures while existing nodes are generally not deleted. However, the tool needs to produce a completely faithful mirror of the source, and richcopy doesn't appear to do that.

Back to my original point, I don't plan to automate my backup process. I am using removable drives for my backups and these drives live in a fire resistant lockbox. I need to manually insert and swap drives to complete a backup of my server. The need to test for the correct drive being mounted and deal with user interaction and retries if the correct drive is not mounted adds complexity to the automation logic. The cost of that complexity exceeds the benefits of the automation for my situation.

Edit: I am only using robocopy/richcopy to create copies of the server's drives. I do not use it to copy folders on my client PCs as those are not Windows machines. I use an OS X backup utility for that alongside Time Machine. I do see permission errors backing up to the root of a share when using that utility, so the permission error you saw is not caused by richcopy (or at least it's not unique to it). The permission error is not of sufficient severity to abort the copy, so I just ignore it.

_________________
Synology DS411: 2 * WD20EARX and 2 * WD20EARS
DIY Small Business Server 2011
MP 8-core 2.26GHz, 2012 MBA13 i7/8GB/256GB, AppleTVs, iPhone

so wait, what about that issue regarding changing source folders and then going back and it not remembering anything of the previous backup?

am i never suppose to change the source folder ever?

should i choose D:\ in the source and just never ever change it?

What problem are you hoping to solve with robocopy/richcopy?

_________________
Synology DS411: 2 * WD20EARX and 2 * WD20EARS
DIY Small Business Server 2011
MP 8-core 2.26GHz, 2012 MBA13 i7/8GB/256GB, AppleTVs, iPhone

YES, if set up as I have in the script.


Generally, YES. It is important to note that backups done this way are NOT "archives" in the manner of the Client Backup Database. There is NO history. It makes an exact copy of what exists, no more and, generally, no less.

Why do I say "generally"? Because it depends on the parameters passed to the executable command, be it XCOPY, SYNCTOY, ROBOCOPY, RICHCOPY, SECOND COPY, or something else.

For this scheduled function a command-line executable is used. XCOPY is the fastest but is not so good at recovery if there is a failure. (But, for locally-attached drives, this is not much of an issue.) It also is only a COPY program, not a SYNCHRONIZATION tool. It does NOT erase files that were copied to the target earlier; it leaves them in place. ROBOCOPY and SYNCTOY can be configured to remove a file on the target if it no longer exists on the source, but it takes considerably longer to run.

I use both. I schedule a rutine with XCOPY Tuesday through Sunday mornings and I schedule one with ROBOCOPY on Monday mornings. The target accumulates a few extra files during the week and cleans them up once a week. (I do NOT run that on Sunday as the changes to the backup database are quite large that day and the run time will, for me, go into the following evening. I prefer to not load down the MSS for that long!


YES, it is most definitely reliable and consistent.


Not sure I remember what all the issues were but I know a number of folks use this regularly (as do most of my clients) and I have had no complaints of failures.

One caveat not addressed before though: If this is running and you log on to the console, DO NOT LOG OFF until the job is complete. That will cause anything running there to end, the backup will not complete, and if the client backup database is being backed up the backup service will not get restarted. (Also not mentioned elsewhere is that the SAME thing happens to the BDBB Add-In.)


The way the sample is written it will copy ALL the Shared Folders PLUS all the client backup database. You can easily drop it down a level for the Shared Folders and copy only those which you wish to send to that target. You could, for example, back up everything EXCEPT your VIDEOS folder -- or even send that folder to a different target. And you could even take this to lower levels if you wish. I have one client that uses a version that sends her business data folders across a VPN to s shared disk in another state every Saturday morning.

And it CAN back up non-shared folders, including folders on the C-Drive if so desired. If the WHS can see it, it can be copied.


You do not have to do anything IN RoboCopy but you DO need to edit the script to fit your environment and schedule it for execution.


It should stay there and be updated with the updates, ass-u-me-ing that your manual copy uses the sane destination (target) schema as you use for the script!


All the copies are to regular NTFS files. The disk can be placed in another computer and the Shared Folders can be accessed directly. And this is another ADVANTAGE of this approach for the backups, unlike the built-in WHS backup which is not readable externally -- but is more efficient in storage. (The backup database, while being a collection of NTFS files, is not directly readable in another computer -- it needs a WHS system to decipher the database.)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

Realistically it is probably a combination of BOTH. Once this is set up and running the only action required it to occasionally swap a drive and move it off-site. If it is very old, or if it is a new drive, then the first backup takes a long time. But it's automated and you do not have to do anything! And if it should, somehow, get interrupted? It just re-does everything the next time it is scheduled. If, worst case, it is interrrupted and leaves the Backup Service stopped? You may miss a nightly backup but the re-running of the routine the following morning will start it again and all will be well again the following morning. (You will have missed one backup cycle for your clients. If you are paying attention you would notice the Console icon changed to red AND WAS STAYING THAT WAY longer than the normal backup would take. You could go in and restart the backup service if necessary.)

Routines of a set-and-forget nature do not generate a lot of discussion.

And that it the HUGE advantage of this over many of the discussions recently posted (since the quoted reference herein) in this thread. It is AUTOMATIC and self-recovering.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

The key is in the "remembering". If you can ALWAYS remember, great. You do NOT need this you can use the built-in functionality of WHS to back up your Shared Folders and the BDBB Add-In to backup the client backup database.

Or you can make even more work for yourself and copy and paste. Or set up RICHCOPY. Or use SECOND COPY. Or a number of other tools.

But you gotta REMEMBER!!!! And most of us don't. And that's almost always when the disaster strikes!

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

That is, of course, your call any your value trade-off. But you have to REMEMBER to do it. And the work is only in the initial setup. I have not changed my setup in about a year now; when you consider that I have FAR, FAR less time invested that any manual process! (Heck, I'm not even awake when this runs most of the time!)


I have been using TrueCrypt for this for a year now. Works like a champ. (See http://www.mediasmartserver.net/forums/viewtopic.php?f=3&t=2934 for some discussions.) I keep trying to find the time to document the setup steps in more detail (screen shots, etc. but....

But the basic command structures are described in that thread.

With the target disk encrypted I don't even need a safe-deposit box. I keep one backup in the car and another at a neighbor's house when I am on vacation. (I still go to the bank, just not as often as I used to!)


XCOPY is, as you say, not as reliable as, say, ROBOCOPY. But if it fails one day it will fix the target the next day. (And, by the way, the odds of a a failure on a local disk are VERY remote. I keep my logs (see the script) for 30 days and do look through them for errors. Only time I found any was when I interrupted the process for some reason -- usually testing the impact of an interrupt!)

But I use ROBOCOPY once a week anyway -- and generally rotate my disks (after checking the log) that day.

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

By the way, the advantage to the method described in the beginning of this thread is that the encryption is easily added to the script and shown in the related thread mentioned a couple posts above. Using any of the directly scheduled tools invites the problem of enabling the encryption, running the backup and then closing the encrypted volume.

Wanting to have the result encrypted was what drve me to this approach in the first place.

(What I REALLY want to do is encrypt the actual MSS/WHS "pool" drives. THAT I have been unable to accomplish so far!)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch

First, guys, this is a thread to discuss AUTOMATED BACKUPS, not for a discussion of RICHCOPY vs ROBOCOPY vs ....

ANY synchronization program that works in a command-line (script) environment is fine. The results aill be the same if (1) the tool works and (2) it is set up properly. And, in terms of getting data TO the target any good copy routine works equallu well: COPY, XCOPY, and even a few others that are more error-tolerant. (These jut end up with MORE files on the target as they do not delete. And THAT MAY BE WHAT IS DESIRED!)

The GUI, direct-scheduling and Add-In possibilities and setups may differ for these, but the end RESULT should be the same.

(By the way, I am not aware of any Add-In for Robocopy. If there is such a thing please post a link. The closest I have seen to this is a request on 12 Jan 2010 for such an Add-In!)

_________________
....JBick

EX475, 2 GB, LE-1640
PC1: Vista-->W7 Ultimate/32, (D-Drive RAID-5 Array)
PC2: Lenovo Laptop, Win XP Home SP3
2xLinksys WRT54G v1.1 and 2xNetGear GS105 Gbit switch