Add FUSE_DEV_IOC_CLONE support for multi-threaded reading #421

ejc3 · 2025-12-19T05:12:22Z

Summary

Add support for multi-threaded FUSE request processing using FUSE_DEV_IOC_CLONE ioctl.

This enables multiple threads to read FUSE requests in parallel from cloned file descriptors, improving throughput for high-concurrency workloads.

API

// Primary session handles INIT and runs in main thread
let mut session = Session::new(fs, mountpoint, options)?;

// Clone fd for additional reader threads
let cloned_fd = session.clone_fd()?;
let reader_session = Session::from_fd_initialized(fs_clone, cloned_fd, acl);
std::thread::spawn(move || reader_session.run());

session.run()?;

Changes

Add Session::clone_fd() - clones the FUSE fd using FUSE_DEV_IOC_CLONE ioctl (Linux only)
Add Session::from_fd_initialized() - creates a session from a cloned fd that skips INIT handshake
Handle ENOENT gracefully when sending replies during unmount (matches libfuse behavior)
Handle EBUSY during unmount with lazy unmount fallback
Keep channel module private per review feedback

Tests

Added two integration tests:

clone_fd_multi_reader - verifies clone_fd() creates a valid fd
from_fd_initialized_works - tests concurrent readers processing requests

Platform Support

clone_fd() is only available on Linux (#[cfg(target_os = "linux")]).

- Add clone_fd() method to Channel that clones the /dev/fuse fd using ioctl - Add from_fd() constructor to create Channel from an OwnedFd - Add channel() accessor to Session to get reference to the underlying Channel - Export channel module publicly for external use This enables multi-threaded FUSE request processing by allowing multiple threads to read from cloned fds in parallel. Each cloned fd shares the same FUSE connection but can independently read and process requests.

The ioctl request type is i32 on musl but c_ulong on glibc. Use conditional compilation to handle both cases.

Adds Session::from_fd_initialized() for use with FUSE_DEV_IOC_CLONE multi-reader setups. When using clone_fd() to create additional reader threads, those sessions need to be marked as initialized since the INIT handshake only happens on the primary session. Without this, cloned sessions return EIO for all requests because the FUSE protocol requires initialization before processing requests.

When using FUSE_DEV_IOC_CLONE to clone /dev/fuse file descriptors for multi-reader setups, replies must be written to the ORIGINAL fd, not the cloned fd. Previously, from_fd_initialized would use the cloned fd's ChannelSender for replies, causing EIO errors at high concurrency. Changes: - Add reply_sender: Option<ChannelSender> field to Session - Update from_fd_initialized() to require a ChannelSender parameter - Update run() to use reply_sender when available for Request creation - from_fd() sets reply_sender: None (not needed for single-fd usage)

- Add #[cfg(target_os = "linux")] guard to clone_fd() method - Add target_os = "linux" to ioctl constant cfg attributes - Tighten unsafe blocks with SAFETY comments - Improve documentation with Platform Support and Errors sections - Add intra-doc link from from_fd() to clone_fd()

- Use standard rustdoc sections (# Arguments, # Important) - Add intra-doc links to Channel::clone_fd() and ChannelSender - Document all parameters explicitly - Clarify consequence of using with uninitialized fd

Remove the reply_sender parameter from from_fd_initialized() since each cloned FUSE fd handles its own request/response pairs - the FUSE kernel requires that the fd which reads a request is the same fd that sends the response. This change: - Removes the reply_sender field from Session struct - Simplifies from_fd_initialized() signature - Updates documentation to clarify cloned fd behavior

During unmount, the kernel aborts all pending FUSE requests. When our reply arrives, the kernel returns ENOENT because the request was already cleaned up. This is expected behavior, not an error. libfuse explicitly handles this: https://github.com/libfuse/libfuse/blob/master/lib/fuse_lowlevel.c "ENOENT means the operation was interrupted" Before this change, clean unmounts would log spurious errors like: ERROR reply{unique=X}: fuser::reply: Failed to send FUSE reply: ... Now these are silently ignored, matching libfuse behavior.

When umount() returns EBUSY during Mount::drop(), fall through to fuse_unmount_pure() which uses MNT_DETACH for lazy unmount. This prevents spurious ERROR logs during parallel test execution when the kernel still has transient references to the FUSE filesystem. EBUSY can occur transiently because: - The FUSE protocol is still completing cleanup after FUSE_DESTROY - Kernel inode/dentry caches have brief reference counts - Async I/O machinery is still completing Using MNT_DETACH is safe here because FUSE_DESTROY has already been sent, so no new operations can start.

Resolve conflict in src/lib.rs by keeping both: - pub mod channel (our multi-reader support) - pub mod experimental (upstream async API) Also add doc comment for ChannelSender.

cberner · 2025-12-19T16:34:29Z

Please add a test for this functionality. Also, I don't really like the idea of making channel public. I think we should find a way to hide this functionality behind mount() or something like that

Replace manual nul-terminated byte string with Rust 1.77+ C string literal syntax. This fixes clippy::manual_c_str_literals warning.

Per review feedback, keep the channel module as an internal implementation detail and expose clone_fd() directly on Session. Changes: - Make channel module private (remove pub from mod channel) - Add Session::clone_fd() that delegates to Channel::clone_fd() - Remove Channel::from_fd() (no longer needed externally) - Remove Session::channel() (no longer needed) Users now call session.clone_fd() instead of session.channel().clone_fd().

Add integration tests for the clone_fd and from_fd_initialized APIs: - clone_fd_multi_reader: Verifies clone_fd() creates a valid fd that can be used for multi-reader setups - from_fd_initialized_works: Tests that multiple reader threads can process FUSE requests concurrently using cloned fds

Sort std::sync imports alphabetically (Arc before atomic).

cberner · 2025-12-20T21:25:20Z

tests/integration_tests.rs

+    thread::sleep(Duration::from_millis(100));
+
+    // Access the mountpoint - this triggers FUSE requests
+    let _ = std::fs::metadata(tmpdir.path());


I'd like to see a few more things covered in this test:

the test should verify that both the filesystems in session and reader_session receive requests to test that the multithreading support is working as expected

it should check that the result of metadata() is the expected value

Thank you for the review and sorry for the delay. I have it incorporated into a "fuse-pipe" crate as part of this project, so it's well tested given all the CI it does:

https://github.com/ejc3/fcvm

https://github.com/ejc3/fcvm/tree/main/fuse-pipe

I will add the additional testing.

Addressed the review feedback:

Both filesystems receive requests: Added artificial 50ms delay in getattr handler so while one reader is processing, the kernel dispatches concurrent requests to the other reader. Separate counters verify distribution (test shows primary=10, reader=10).

Verify metadata() returns expected values: Added assertions for is_dir() and permissions().mode() & 0o777 == 0o755.

Test output:
Request distribution: primary=10, reader=10, total=20
test from_fd_initialized_works ... ok

The channel module was already made private in an earlier commit (c7e3f69).

Adds userspace support for the new FUSE_REMAP_FILE_RANGE opcode (53) which enables reflink operations (FICLONE/FICLONERANGE ioctls) through FUSE filesystems. Changes: - Add FUSE_REMAP_FILE_RANGE opcode (53) to fuse_opcode enum - Add fuse_remap_file_range_in wire struct matching kernel ABI - Add RemapFileRange request parsing in ll/request.rs - Add remap_file_range() method to Filesystem trait with default ENOSYS - Wire up request dispatch in request.rs Wire format (fuse_remap_file_range_in): - fh_in: u64, off_in: i64, nodeid_out: u64, fh_out: u64 - off_out: i64, len: u64, remap_flags: u32, padding: u32 Requires kernel patch (FUSE_REMAP_FILE_RANGE not yet upstream). Tested: cargo build --features abi-7-28 cargo test --features abi-7-28 E2E: ejc3/fcvm#21 - cp --reflink=always through FUSE → btrfs - filefrag confirms shared extents

Opcode 53 is already used by FUSE_COPY_FILE_RANGE_64 in kernel 6.12+ (ABI 7.45). Use opcode 54 for remap_file_range to avoid conflict. Discovered when cross-referencing with libfuse's fuse_kernel.h which has all opcodes through 7.45.

- Use artificial 50ms delay in handler to ensure kernel dispatches requests to both readers (not just the first one blocked in read()) - Add separate counters per reader instance to verify distribution - Assert both readers process requests (verifies multi-threading works) - Verify metadata() returns expected values (is_dir, permissions 0o755) - Generate concurrent requests from 4 threads × 5 iterations Tested: cargo test --release --test integration_tests Result: primary=10, reader=10, total=20

ejc3 and others added 11 commits November 26, 2025 19:33

Fix ioctl type for musl target

f2dba49

The ioctl request type is i32 on musl but c_ulong on glibc. Use conditional compilation to handle both cases.

Improve from_fd_initialized documentation

a2c23f1

- Use standard rustdoc sections (# Arguments, # Important) - Add intra-doc links to Channel::clone_fd() and ChannelSender - Document all parameters explicitly - Clarify consequence of using with uninitialized fd

Run cargo fmt

7773884

Merge upstream/master: add experimental async API

cbd727a

Resolve conflict in src/lib.rs by keeping both: - pub mod channel (our multi-reader support) - pub mod experimental (upstream async API) Also add doc comment for ChannelSender.

ejc3 added 3 commits December 20, 2025 09:05

Use C string literal for /dev/fuse path

a752281

Replace manual nul-terminated byte string with Rust 1.77+ C string literal syntax. This fixes clippy::manual_c_str_literals warning.

ejc3 marked this pull request as ready for review December 20, 2025 09:20

ejc3 force-pushed the master branch from c9188e4 to 0ed8a17 Compare December 20, 2025 09:25

Fix import ordering in integration tests

008dc1d

Sort std::sync imports alphabetically (Arc before atomic).

cberner requested changes Dec 20, 2025

View reviewed changes

ejc3 and others added 4 commits December 26, 2025 07:04

Fix FUSE_REMAP_FILE_RANGE opcode: 53 -> 54

0365c6a

Opcode 53 is already used by FUSE_COPY_FILE_RANGE_64 in kernel 6.12+ (ABI 7.45). Use opcode 54 for remap_file_range to avoid conflict. Discovered when cross-referencing with libfuse's fuse_kernel.h which has all opcodes through 7.45.

Merge branch 'cberner:master' into master

58fd54a

ejc3 requested a review from cberner January 5, 2026 20:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add FUSE_DEV_IOC_CLONE support for multi-threaded reading #421

Add FUSE_DEV_IOC_CLONE support for multi-threaded reading #421

Uh oh!

ejc3 commented Dec 19, 2025 •

edited

Loading

Uh oh!

cberner commented Dec 19, 2025

Uh oh!

cberner Dec 20, 2025

Uh oh!

ejc3 Jan 5, 2026

Uh oh!

ejc3 Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add FUSE_DEV_IOC_CLONE support for multi-threaded reading #421

Are you sure you want to change the base?

Add FUSE_DEV_IOC_CLONE support for multi-threaded reading #421

Uh oh!

Conversation

ejc3 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

API

Changes

Tests

Platform Support

Uh oh!

cberner commented Dec 19, 2025

Uh oh!

cberner Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

ejc3 Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

ejc3 Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ejc3 commented Dec 19, 2025 •

edited

Loading