Unruh <unruh-spam@physics.ubc.ca> wrote:
> All rsync these days have ssh as the default, not rsh.

Sadly that's not the case with rsync for Solaris as provided by
sunfreeware.com. I've had to modify a number of my cross-platform scripts
to allow for that.

Chris

Unruh wrote:
> Joe Beanfish <joe@nospam.duh> writes:
>
>> Unruh wrote:
>>> Mtek <mtek@mtekusa.com> writes:
>>>
>>>> On Jul 23, 6:54=A0am, Frank Elsner <Frank.Els...@spamfence.net> wrote:
>>>>> Unruh wrote:
>>>>>> Mtek <m...@mtekusa.com> writes:
>>>>>>> Hi,
>>>>>>> I have 2 servers. =A0One is a web/mail server and the other is a backup
>>>>>>> server. =A0Currently, to back up the server I have a script that runs
>>>>>>> which basically tar's the entire apache tree along with dumps of the
>>>>>>> MySQL databases and use SCP to send them to the backup server.
>>>>>>> Later, a script runs on the backup server which untar's the files and
>>>>>>> imports the databases.
>>>>>>> My question is, is this a good backup method? =A0I see many people usin=
>>>> g
>>>>>> OK but cumbersome.
>>>>>>> RSYNC. =A0Would this be a better solution? =A0If yes, how would I do it=
>>>> ??
>>>>>> Yes, it would be. rsync also checks the data for correctness ( hashes t=
>>>> he
>>>>>> same) while transfering it.
>>>>>> It also transfers only what has changed, not everything every time.
>>>>> What about database file with holes, postgres comes to my mind.
>>>>> Is it save to rsync them?
>>>>>
>>>>> --Frank- Hide quoted text -
>>>>>
>>>>> - Show quoted text -
>>>> Well, are you also saying that I do not need to export and re-import
>>>> the database tables? That it will copy the databse files and mySQL
>>>> will work fine on the backup server?
>>> I do not use database tables, so do not know what problem it is that
>>> you are describing. HOwever what rsync does is to make an exact, bit for
>>> bit copy of the files, including all of the times associated with the files.
>>> It then uses a checksum (md4) to make sure that the
>>> two files are identical. I assume that identical files are treated by mySQL
>>> in an identical manner. If mySQL somehow mushes together the CPU serial
>>> number, the hard disk size, the network MAC address in determining how it
>>> treats a file, then all bets are off. But I can see no reason why it
>>> should.
>
>> The problem is that the database may change while the rsync is in
>> progress such that the copy is not only old, but corrupt. To ensure
>> a good copy the database must be inactive or locked for the duration
>> of the copy. The dump takes care of that. rsync doesn't.
>
> But rsync will tell you that it could not do the transfer because the file
> had changed. As I said, rsync does a md4 hash of both and makes sure they
> are the same after the transfer. If the file has changed they will not be
> the same.

Telling is great. But in the end the necessary result of a backup did
not occur.

> Yes, you should switch OFF any file changing while the transfer is going
> on. That is true for ALL backup options. Ie, while backing up do notallow
> the database to be updated. rsync at least lets you know that has happened.
> (Note that there must be enough space on the target to hold an extra copy
> of the largest file to be transfered as rsync makes copies to preserve the
> old file until the new one has been checked to make sure it is valid).
>
> Note that you can run rsync say 3 times in a row to catch any files which
> were active on the first run. This will not cost anything as the good files
> will not get transfered again, just the changed ones.

Multiple tries could be a good strategy for low to moderately active
databases. But those in continuous flux it'll never get it right unless
there's a tool to allow locking of the entire database for the duration
of the rsync process.

Also note that rsyncing nothing isn't free. There's still the time and
disk bandwidth used to find and checksum the desired files. We've found
that on filesystems with many many files the time it takes for rsync
to even start copying anything is very substantial.

Joe Beanfish <joe@nospam.duh> writes:

>Unruh wrote:
>> Joe Beanfish <joe@nospam.duh> writes:
>>
>>> Unruh wrote:
>>>> Mtek <mtek@mtekusa.com> writes:
>>>>
>>>>> On Jul 23, 6:54=A0am, Frank Elsner <Frank.Els...@spamfence.net> wrote:
>>>>>> Unruh wrote:
>>>>>>> Mtek <m...@mtekusa.com> writes:
>>>>>>>> Hi,
>>>>>>>> I have 2 servers. =A0One is a web/mail server and the other is a backup
>>>>>>>> server. =A0Currently, to back up the server I have a script that runs
>>>>>>>> which basically tar's the entire apache tree along with dumps of the
>>>>>>>> MySQL databases and use SCP to send them to the backup server.
>>>>>>>> Later, a script runs on the backup server which untar's the files and
>>>>>>>> imports the databases.
>>>>>>>> My question is, is this a good backup method? =A0I see many people usin=
>>>>> g
>>>>>>> OK but cumbersome.
>>>>>>>> RSYNC. =A0Would this be a better solution? =A0If yes, how would I do it=
>>>>> ??
>>>>>>> Yes, it would be. rsync also checks the data for correctness ( hashes t=
>>>>> he
>>>>>>> same) while transfering it.
>>>>>>> It also transfers only what has changed, not everything every time.
>>>>>> What about database file with holes, postgres comes to my mind.
>>>>>> Is it save to rsync them?
>>>>>>
>>>>>> --Frank- Hide quoted text -
>>>>>>
>>>>>> - Show quoted text -
>>>>> Well, are you also saying that I do not need to export and re-import
>>>>> the database tables? That it will copy the databse files and mySQL
>>>>> will work fine on the backup server?
>>>> I do not use database tables, so do not know what problem it is that
>>>> you are describing. HOwever what rsync does is to make an exact, bit for
>>>> bit copy of the files, including all of the times associated with the files.
>>>> It then uses a checksum (md4) to make sure that the
>>>> two files are identical. I assume that identical files are treated by mySQL
>>>> in an identical manner. If mySQL somehow mushes together the CPU serial
>>>> number, the hard disk size, the network MAC address in determining how it
>>>> treats a file, then all bets are off. But I can see no reason why it
>>>> should.
>>
>>> The problem is that the database may change while the rsync is in
>>> progress such that the copy is not only old, but corrupt. To ensure
>>> a good copy the database must be inactive or locked for the duration
>>> of the copy. The dump takes care of that. rsync doesn't.
>>
>> But rsync will tell you that it could not do the transfer because the file
>> had changed. As I said, rsync does a md4 hash of both and makes sure they
>> are the same after the transfer. If the file has changed they will not be
>> the same.

>Telling is great. But in the end the necessary result of a backup did
>not occur.

>> Yes, you should switch OFF any file changing while the transfer is going
>> on. That is true for ALL backup options. Ie, while backing up do notallow
>> the database to be updated. rsync at least lets you know that has happened.
>> (Note that there must be enough space on the target to hold an extra copy
>> of the largest file to be transfered as rsync makes copies to preserve the
>> old file until the new one has been checked to make sure it is valid).
>>
>> Note that you can run rsync say 3 times in a row to catch any files which
>> were active on the first run. This will not cost anything as the good files
>> will not get transfered again, just the changed ones.

>Multiple tries could be a good strategy for low to moderately active
>databases. But those in continuous flux it'll never get it right unless
>there's a tool to allow locking of the entire database for the duration
>of the rsync process.

>Also note that rsyncing nothing isn't free. There's still the time and
>disk bandwidth used to find and checksum the desired files. We've found
>that on filesystems with many many files the time it takes for rsync
>to even start copying anything is very substantial.

The default is to use the time and date and size of the file to determine
if it is worth checking. Ie, it assumes that if the file has the same time
and date at both ends it is OK. This may be a bad assumption. Then you want
to use the -c flag which checks each files hash to see if it should be
transfered. Much more expensive in computer time.

Yes, I agree that you should lock the database if you have one that is in
constant use. But that need not be part of rsync ( and is not). So run th
elock program, and then run rsync.