`rsync` freezes and hangs on large files

rsync: connection unexpectedly closed (... bytes received so far) [sender]

I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.

What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.

On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, solution is to add below settings to your /etc/ssh/sshd_config.

# This will make the server send the clients a “null packet” every 120 seconds and not disconnect them until the client 
# have been inactive for 720 intervals (120 seconds * 720 = 86400 seconds = 24 hours).
ClientAliveInterval 120
ClientAliveCountMax 720

Both issues are likely a non issues if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers. If this has helped you please comment or follow me on twitter @danielsokolow

Comments

Storing passwords in PuTTY

How to save SSH username/password for auto login in PuTTy The answer is you can't do it...at least in plain PuTTy. However there is an awesome fork with that let's you store the username and password and other additional features called KiTTy . So grab yourself a copy and +1 this if you do, thanks.

Softeher 'Error occurred. (Error code: 2)' sollution

Protocol error occurred. Error was returned from the destination server. The Softether server by default to run on port 443 , if you server also hosts normal https then 443 is already taken and so Softether can't bind to it. When you run `vpncmd` it attempts to connect, find an active port, but of course fails with 'Protocol error occurred. Error was returned from the destination server.' because it's not actually connecting to the vpn server. By default Softether also listens on 992 , 1194 , and 5555 so the sollution is to modify specify `localhost:5555` when executing the `vpncmnd`. If this has helped you feel free to comment or follow me on twitter @danielsokolows .

LightScribe Template Labeler for Windows 10

LightScribe Template Labeler installation "Operating System Inadequate" Issue Unfortunately 'LightScribe Template Labeler' installation will not work on Windows 8 and Windows 10 machines as the install simply will error out with 'Operating System Inadequate'. Trying to install using compatibility mode did not help. Installing LightScribe Template Labeler Workaround However if you simply copy the installed folder from an older machine like Windows 7 into your `C:\\Program Files (x86)\LightScribe Template Labeler`, create a shortcut, then the software will still happily work, see: http://answers.microsoft.com/en-us/windows/forum/windows8_1-performance/lightscribe-templatesimple-labeler-the-system-is/c2ebee28-bfb5-4aef-beb3-a43329561da8?db=5&auth=1 So you can install Windows 7 machine in a virtual machine, install 'LightScribe Template Labeler', and then copy the resulting folder to your Windows 8 or Windows 10 machine. For my own convenien...

Daniel Sokolowski's Blog

Search This Blog