`rsync` freezes and hangs on large files

rsync: connection unexpectedly closed (... bytes received so far) [sender]

I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.

What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.

On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, solution is to add below settings to your /etc/ssh/sshd_config.

# This will make the server send the clients a “null packet” every 120 seconds and not disconnect them until the client 
# have been inactive for 720 intervals (120 seconds * 720 = 86400 seconds = 24 hours).
ClientAliveInterval 120
ClientAliveCountMax 720

Both issues are likely a non issues if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers. If this has helped you please comment or follow me on twitter @danielsokolow

Comments

Duplicate value found: duplicates value on record with id: <unknown>.

System.DmlException: Insert failed. First exception on row 0; first error: DUPLICATE_VALUE, duplicate value found: <unknown> duplicates value on record with id: <unknown>. The above error is triggered in the database layer and caused by a trigger or workflow outside of your main code of block that is bubbling this exception. This is rather difficult to track down especially if you are unfamiliar with the code, I am sharing my procedure in the hopes this saves you time - if you find this helpful drop me a line or follow me on twitter @danielsokolows . This error is caused by unique field constraint on the object, so the first step is to examine the object and locate the API names of all unique fieds. You can do this through SF direclty 'Setup < Customize &lt <object being inserted> &lt Fields' or by downloading the `src/objects` metadata information and searching for <unique> ; I preffer the latter and actually download ALL matadata i...

Softeher 'Error occurred. (Error code: 2)' sollution

Protocol error occurred. Error was returned from the destination server. The Softether server by default to run on port 443 , if you server also hosts normal https then 443 is already taken and so Softether can't bind to it. When you run `vpncmd` it attempts to connect, find an active port, but of course fails with 'Protocol error occurred. Error was returned from the destination server.' because it's not actually connecting to the vpn server. By default Softether also listens on 992 , 1194 , and 5555 so the sollution is to modify specify `localhost:5555` when executing the `vpncmnd`. If this has helped you feel free to comment or follow me on twitter @danielsokolows .

GeoDjango + Spatialite (SQLite3) Setup Issues

Because there are fresh ones every time I start a new project! As of May 03 2013 I have opted to install virtualenv --system-site-packages and running aptitude-install python-pysqlite2 spatialite-bin gdal-bin ; the site are not as self contained any more but sure beats fighting with the issues. All my Django projects are self contained (as much as possible) with Virtualenv and use Spatialite database (SQLite3 +geo spatial stuff) installed so to enable GeoDjango functionality. This post is a list of issues I encounter related to that setup and how I resolved them; as I seem to run into a new one on every project start. And as always if you found this useful do +1, or follow me on twitter @danielsokolows /usr/include/spatialite.h:61: note: previous declaration of ‘spatialite_init’ was here Say what? --- this one threw me for a three hour fun time. It happened because I executed 'pip install pyspatialite' will install the 3.X vesion which is is broken and craps out with: ...

Daniel Sokolowski's Blog

Search This Blog