sometimes slow backup of unchanged files on NFS

[ThomasWaldmann]

Note: debugging this on a specific system right now, keeping some notes here.

Is this a BUG / ISSUE report or a QUESTION?

Unclear yet.

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.17

Operating system (distribution) and version.

Redhat or rh-derivate.

Hardware / network configuration, and filesystems used.

6(?) cores, 32GB RAM

How much data is handled by borg?

TBD

Full borg commandline that lead to the problem (leave away excludes and passwords)

Roughly this:

borg create --compression lz4 --list --files-cache=ctime,size ... /mnt/localhdd::repo /mnt/remotenfs/

Describe the problem you're observing.

Sometimes backup slows down from the usual 100..1000 unchanged files per second (status U) to only a few per second.

Usually when run as a nightly cronjob. When trying to reproduce interactively, it is hard to reproduce.

Analysis / Suspicions

When processing lots of unchanged files, borg is doing:

• a stat call on the src file to determine ctime,size for change detection
• if unchanged (U status), it will not need to read the file contents as they did not change and are already in the repo. thus there is not a lot of data volume read from the source neither there is a lot of data written to the repo.
• but it will still need to read metadata like bsdflags, xattrs, acls (because we do not have them in files cache, but need them for the current backup) - this is the default behaviour.
• if one is not interested in archiving these, one can omit them using options --nobsdflags --noacls --noxattrs.
• id-to-name lookups to map uid / gid to username / groupname (usually fast when local, but can be slow if delegated remotely).
• if one is not interested in archiving username / groupname (but only numeric uid and gid), one can omit them using --numeric-owner (borg 1.1) or --numeric-ids (borg > 1.1.x).

Other stuff coming to mind as difference between cron use and interactive use:

• environment variables?
• sh vs. bash? any quoting / expansion differences? do the borg excludes work correctly?

Notes

Added now --numeric-owner --noatime --noacls --noxattrs --nobsdflags to try if this resolves the issue.

Also considering monitoring with atop and doing a timestamped strace when it's running slowly.

[FabioPedretti]

For the record I experienced similar slowdowns on NFS, with many previous releases which got some fixes in the later 1.1 releases, and again with borg 1.2, this was discussed at #4498. See also #3039. Thanks for looking at it!