Much faster borg backup over SSH via ProxyCommand's use of nc

[ttsiodras]

This is not a bug - it's more like a gentle suggestion :-)

I am using borgbackup to do a weekly backup of my Arch Linux laptop contents to an Orange PI SBC running Armbian. I am doing this over borgbackup, and recently realized that the SSH speeds used are far inferior to what my wifi link can deliver. You can read my debugging story here or watch a video about it here - the executive summary is that I got almost an order of magnitude faster transfers via use of netcat in SSH's ProxyCommand:

$ cat .ssh/config
Host orangepi
    Hostname 192.168.1.150
    User root
    Port 22
    ProxyCommand nc %h %p

....which can also be done in the command line - as my debugging story shows via ssh -o "ProxyCommand nc %h %p".

Perhaps you can have a look and see if this improves speed in your tests as well (it definitely did on my own) - and if so, perhaps consider it for use as a default.

[dragetd]

For reference: The Orange Pi Zero has a quadcore Cortex-A7 which is moderately quick. I used to use a Raspberry Pi 1 which struggled badly with SSH and Borg to get over 600kb/s.

Since Benchmarks show that the Orage Pi Zero is not too far off from the Pine64, which I am currently using as a backup target, I would assume it would handle 2-5mb/S which I am currently pushing to mine.

In your Stack Exchange post you assume it might be the driver detecting SSH traffic and by going through NC it is somehow packaged differently… ? This is a rather scary and strange theory. SSH-traffic is encrypted in software and there is not much to be done with it on the network layer… but that being said, the most crazy stupid theory would be the Chinese actually messing with SSH in their binary blob kernel. Are there sources for the same kernel available? If we'd manage to compile it from source and verify the same effect, that we could investigate the source.

Adding nc ProxyCommand to my setup did not change much at all, tho I am limited by my upload capacity.

I doubt that this is a borg-specific effect.

[ttsiodras]

Hi, @dragetd - indeed, I actually debugged this further.

Look at my post again - I just updated the answer, fully tracking this down. It was SSH in the end - it was disabling Nagle's algorithm... The final form of my answer includes a very convincing proof - a few lines of C source code that I used (via LD_PRELOAD) to "suppress" the syscall done by SSH to disable Nagle (i.e. TCP_NODELAY). The result is that I instantly get back to the (much higher) iperf3-reported speeds.

Have a look at the final form of my answer - I think you will see that Borgbackup, rsync, and everybody else that uses SSH for transport would benefit from this (at least in slow links).

I am unsure how to do it "formally", though... I wish SSH had a command line option to do this properly - (i..e. --dont-disable-nagle). But in its absence, I think the closest "hack" is my use of nc under ProxyCommand.

[klausenbusk]

I am unsure how to do it "formally", though... I wish SSH had a command line option to do this properly - (i..e. --dont-disable-nagle). But in its absence, I think the closest "hack" is my use of nc under ProxyCommand.

At least some version of SSH contain a NoDelay option (https://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch07_04.htm). I think you should open a feature request upstream: https://bugzilla.mindrot.org/

[ttsiodras]

@klausenbusk Done:

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

[ThomasWaldmann]

This sounds interesting. Guess we need some more data for this from other users - post your link types and the speed you get with borgbackup using the 2 different ways to setup the ssh connection.

If this can be reproduced and the difference is significant, I guess we at least want to add this to the docs.

BTW, I'ld appreciate if the tests are done with fast machines on both sides, so we are really measuring the network related stuff and not the cpu / bus system (so no raspis & other sbc please).

[ttsiodras]

Makes sense, Thomas - perhaps it merits a mention in the docs even now.

But in relation to your comment about SBCs, I believe I've proven conclusively in my debugging that use of an SBC at the remote end had nothing to do with the phenomenon. Look at the output of dstat - during the much more stressful test from Machine 2, the SBC reported less than 25% CPU utilization of only one of its 4 cores...the processing power available was more than enough to handle things.

Put simply: SSH is currently always using setsockopt(...., TCP_NODELAY), no matter what the link is. In slow WiFis, this means that the link is immediately transmitting whatever packet data are sent to the socket - and depending on the quality of the link, that can end up wasting a significant part of the available bandwidth (think of wifi packets that are mostly headers).

I'll just leave this asciicast here as the best possible proof I can provide.

[stevesbrain]

@ThomasWaldmann Sorry, are you after tests with + without disabled Nagle, or with + without the netcat ProxyCommand?

[ThomasWaldmann]

@stevesbrain as far as i understood, using netcat was the only way to enable it. if it could be done within ssh, that would be of course the more direct way.

[ttsiodras]

@stevesbrain You can indeed use the LD_PRELOAD "hack" that I described in my post - but I doubt we'd want this monstrosity to become a part of borgbackup :-)

So indeed, Thomas is right - try using the ProxyCommand. If the WiFi link you are using is fast enough, you may not see much of a difference - but if it's a lousy one, you should. The lousier the WiFi link, the more Nagle should help with bandwidth...

[ThomasWaldmann]

The docs which is read by most people is the one for the latest release, so we have some time left until next release.

[stevesbrain]

I'll see if I can get some useful stats :)

[jdchristensen]

In my tests, over wifi, I get the same speed with ordinary ssh and with the nc proxy.

I'm skeptical that this will affect many people. If you are feeding data to ssh fast enough, then it shouldn't matter whether you use Nagle or not. And ssh has been so extensively used that if this was a common problem, it would be well known. I think there must be something fishy going on in @ttsiodras 's setup, maybe with the driver for that particular wifi chip. The fact that packet counts weren't very different with and without Nagle suggest that that might be a red herring, and it might instead be a bug that's triggered only when Nagle is off.