`rsync` freezes and hangs on large files

rsync: connection unexpectedly closed (... bytes received so far) [sender]

I have regular server backups which transfers large files over 500GB+ and have --append-verify or --checkusm over ssh parameters specified.

What I have found upon analysis is that once the client side completes it's file checks then the server side checks start. Which means while the server is doing it's checks the client side will appear hanged and frozen - run htop on the server to rsync working away.

On related note, this very LONG wait would trigger SSH timeout and a rsync: connection unexpectedly closed (254 bytes received so far) [sender] error message, solution is to add below settings to your /etc/ssh/sshd_config.

# This will make the server send the clients a “null packet” every 120 seconds and not disconnect them until the client 
# have been inactive for 720 intervals (120 seconds * 720 = 86400 seconds = 24 hours).
ClientAliveInterval 120
ClientAliveCountMax 720

Both issues are likely a non issues if rsync is run in deamon mode on the server and using the rsync protocol instead of ssh for transfers. If this has helped you please comment or follow me on twitter @danielsokolow

Comments

Duplicate value found: duplicates value on record with id: <unknown>.

System.DmlException: Insert failed. First exception on row 0; first error: DUPLICATE_VALUE, duplicate value found: <unknown> duplicates value on record with id: <unknown>. The above error is triggered in the database layer and caused by a trigger or workflow outside of your main code of block that is bubbling this exception. This is rather difficult to track down especially if you are unfamiliar with the code, I am sharing my procedure in the hopes this saves you time - if you find this helpful drop me a line or follow me on twitter @danielsokolows . This error is caused by unique field constraint on the object, so the first step is to examine the object and locate the API names of all unique fieds. You can do this through SF direclty 'Setup < Customize &lt <object being inserted> &lt Fields' or by downloading the `src/objects` metadata information and searching for <unique> ; I preffer the latter and actually download ALL matadata i...

Softeher 'Error occurred. (Error code: 2)' sollution

Protocol error occurred. Error was returned from the destination server. The Softether server by default to run on port 443 , if you server also hosts normal https then 443 is already taken and so Softether can't bind to it. When you run `vpncmd` it attempts to connect, find an active port, but of course fails with 'Protocol error occurred. Error was returned from the destination server.' because it's not actually connecting to the vpn server. By default Softether also listens on 992 , 1194 , and 5555 so the sollution is to modify specify `localhost:5555` when executing the `vpncmnd`. If this has helped you feel free to comment or follow me on twitter @danielsokolows .

How to child proof a fireplace

DIY - Do it yourself fireplace child guard Our wonderful 8.5 month old Sofia has become a crawling race car with an untamed thirst for exploration. And so with the cold nights approaching we needed to child proof the fireplace. This however proved to be more difficult than would reasonably expect, I've checked the local Toys "R" Us, Walmart, and even a Canadian Tire with no success for a ready to use product. Internet search was more fruitful and returned a few online stores one could order from, however in all honestly they didn't look too sturdy to me. So I build my own relatively quickly and inexpensively. Materials needed is a privacy plastic lattice - the smallest hole pattern - a few screws and anchors; tools needed are a drill, and a handsaw if you don't have the lattice cut at the store - that’s it. The construction consits of screwing the lattice into the wall and the final product is easiest explained through following pictures. ...

Daniel Sokolowski's Blog

Search This Blog