create: add an --add-parents option?

[Woi]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question, maybe a bug

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.15

Operating system (distribution) and version.

Fedora 32

Hardware / network configuration, and filesystems used.

No network, local filesystem only: btrfs, fuse, tmpfs

How much data is handled by borg?

~1GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

Let's check ownership and file permissions of home directories:

ls -l /home/
drwx------. 62 user1 user1  4096  9. Feb 21:15 user1
drwx------. 16 user2 user2  4096  8. Feb 22:19 user2

As root:
Create a repository and backup /home/user2 and /home/user1/subdir

borg init --encryption=repokey-blake2 --storage-quota=10G --make-parent-dirs /tmp/borgtest/{hostname}
borg create --progress --stats  /tmp/borgtest/{hostname}::{now} /home/user1/subdir /home/user2

Mount the backup to allow users to restore their own files:

mkdir /tmp/borgtest/mount
borg mount -o allow_other /tmp/borgtest/{hostname} /tmp/borgtest/mount
chmod 755 /tmp/borgtest

As user1:

cd /tmp/borgtest/mount/2021-02-09T20:46:05/home
ls -l
drwxr-xr-x. 1 root  root  0  9. Feb 20:47 user1
drwx------. 1 user2 user2 0  8. Feb 22:19 user2

Homedirectory of user1 is owned by root and permissions are set to 755!?

As root again:
Let's check what borg extract does, to see if this behaviour is fuse specific:

borg umount /tmp/borgtest/mount
mkdir /tmp/borgtest/restore
cd /tmp/borgtest/restore/
borg extract --progress --strip-components=1 /tmp/borgtest/{hostname}::2021-02-09T20:46:05
ls -l
drwx------.  3 root  root   60  9. Feb 20:51 user1
drwx------. 16 user2 user2 600  8. Feb 22:19 user2

Ownership is still root/root, but permissions looks okay. So this differs from source and borg mount.

Describe the problem you're observing.

Directory ownership differs from source if only a subdirectory is backed up.
Directory ownership and permissions differs from source if only a subdirectory is backed up and repository is mounted with fuse.

Questions is: Did I made a mistake? Is this behaviour intended and I missed something in the documentation? Or did I found a bug?

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, I can reproduce this as described. The above is a real world example, only user and directory names have been replaced for privacy .
I double checked with other homedirs and subdirectories and still had the same behaviour when mounting with fuse.

Include any warning/errors/backtraces from the system logs

None on stdout, dmesg or journalctl

[ThomasWaldmann]

You did not backup the directory object user1, so how should borg give you the right user/group/mode?

[Woi]

Thanks for the quick replay. I used the last days to think about it and do some more testing.
It could argue that this is a technical answer, while from a users point of view it's desirable to backup full paths with permissions, ownerships and {a,c,m}times.
But since tar behaves similar, if run as root, I like to make this a question:
How can I preserve ownerships, permissions and {a,c,m}times for all directories in my backup?
I tested quite a while with --patterns as explained in borg help patterns but failed for even this simple example.

[ThomasWaldmann]

you can start at the root dir and exclude all what you don't want.

[Woi]

I could archive the desired behaviour with
borg create --pattern=+/home/user1/subdir --pattern=-/home/user1/* --pattern=-/home/user3 --pattern=-/home/lost+found /tmp/borgtest/{hostname}::{now}{now:%z} /home
For my use case it's sufficient and more importantly I understood the logic behind it. Thanks ThomasWaldmann.

However, It seems like there are more users who didn't expect this behaviour, so maybe it's worth to have an option like --parentdirectrorieswithmetata or --addownershipforfullpath. This would clarify the default behaviour and make borg create a lot easier for cases with more subdirectories or lists of single directories. What do you think?

[ThomasWaldmann]

borg create --add-parents ... root1 root2 ... could add directory items for all parent directories of all roots given.

For borg extract, things would be still a bit complicated, because when borg encounters an item for extraction and notices that some directories in the parent hierarchy are missing in the filesystem, it already has passed by the respective parent item (which come before the childs in the sequential archive).

[Woi]

--add-parents would be a great name for a very nice feature :)
Even without support in borg extract it's useful:

• it would be possible to fix permissions manually, right now
• metadata can be backed up now and restored with a later version of borg extract
• it fixes issues with access rights of borg mount

I guess it should be marked as "experimental" till it's supported in borg extract

Speaking of borg extract: do we need a --add-parents or a --restore-parents for it? Changing the default of borg extract could behave undesired, if used with an old backup or if backup was created without --add-parents.

[Woi]

I thought a bit about implementing --add-parents for borg extract:

Since borg create --pattern=+/home/user1/subdir --pattern=-/home/user1/* --pattern=+/home/user1 --pattern=-/home/* /home is already working fine in borg extract and borg create --add-parents /home/user1/subdir should be equivalent to it, why do we need --add-parents support in borg extract at all? Isn't all the information just stored in the archive?

[ThomasWaldmann]

@Woi using that (rather complex) command, yes, all needed fs items would be in the archive. But still, if you just extract a child item, the matcher would not match (and extract) the parent dirs.

[cyfdecyf]

I noticed this behavior too. I would also like to see option like --add-parents that can make the behavior more natural for users.

My use case is to backup only source files. Using --patterns-from option, I can backup only files with specific extensions.

The pattern file looks like (note this also backs up directory named as foo.c/, but not big problem for me):

P sh
R /home
# Ignore hidden directories.
- **/.*
+ **/*.c
- *

After borg extract:

• for files that match the patterns (and thus backed up and restored), user, group & permission are right
• for directories that match the pattern, it's user, group & permission are also correct
• for all parent directories of the matched path, they all have same same user, group and 0755 unix permission

It took me some time to figure out the behavior is reasonable because parent directories is excluded thus there's no information backed up by borg. Guess it's better to mention this in document. I'll create a PR for the doc change.