Skip to content

[improvement](spill) Monitor and log inode usage during spill gc - #67498

Draft
mrhhsg wants to merge 1 commit into
apache:masterfrom
mrhhsg:spill-gc-inode-monitor
Draft

[improvement](spill) Monitor and log inode usage during spill gc#67498
mrhhsg wants to merge 1 commit into
apache:masterfrom
mrhhsg:spill-gc-inode-monitor

Conversation

@mrhhsg

@mrhhsg mrhhsg commented Sep 3, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Problem Summary:

Spill creates one directory per query and per operator plus one file per
part, so under high concurrency a spill disk can run out of inodes long
before it runs out of bytes. The spill capacity checks and the periodic
spill gc time log only report bytes, so an inode leak or a growing
spill_gc backlog cannot be diagnosed from the BE log or metrics.

This PR adds inode monitoring to the spill file manager:

  • LocalFileSystem::get_inode_info() returns the total and available
    inode count of the file system holding a path via statvfs.
  • SpillDataDir::update_capacity() refreshes the inode statistics and
    exposes them as spill_disk_inode_total and
    spill_disk_inode_available metrics. A throttled warning is logged
    when the inode usage reaches storage_flood_stage_usage_percent.
  • The spill gc summary log now prints, per spill store, the inode usage,
    the number of query directories still pending under spill_gc, the
    number of directories/files deleted in this round and the number of
    failed deletions. Deletion failures are no longer silently discarded
    but logged with a throttled warning.

Release note

None

Check List (For Author)

  • Test:
    • Unit Test: LocalFileSystemTest.GetInodeInfo,
      SpillFileTest.UpdateCapacityTracksInodeUsage,
      SpillFileTest.GCCleansUpGcRootBacklog
    • Regression test: No
  • Behavior changed: No
  • Does this need documentation: No

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

https://claude.ai/code/session_0149x6DMH4C9RUmHLjNKDFxS

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Spill creates one directory per query and per operator plus one file per
part, so under high concurrency a spill disk can run out of inodes long
before it runs out of bytes. The spill capacity checks and the periodic
`spill gc time` log only report bytes, so an inode leak or a growing
`spill_gc` backlog cannot be diagnosed from the BE log or metrics.

This PR adds inode monitoring to the spill file manager:

- `LocalFileSystem::get_inode_info()` returns the total and available
  inode count of the file system holding a path via `statvfs`.
- `SpillDataDir::update_capacity()` refreshes the inode statistics and
  exposes them as `spill_disk_inode_total` and
  `spill_disk_inode_available` metrics. A throttled warning is logged
  when the inode usage reaches `storage_flood_stage_usage_percent`.
- The spill gc summary log now prints, per spill store, the inode usage,
  the number of query directories still pending under `spill_gc`, the
  number of directories/files deleted in this round and the number of
  failed deletions. Deletion failures are no longer silently discarded
  but logged with a throttled warning.

### Release note

None

### Check List (For Author)

- Test:
    - Unit Test: `LocalFileSystemTest.GetInodeInfo`,
      `SpillFileTest.UpdateCapacityTracksInodeUsage`,
      `SpillFileTest.GCCleansUpGcRootBacklog`
    - Regression test: No
- Behavior changed: No
- Does this need documentation: No

### Check List (For Reviewer who merge this PR)

- Confirm the release note
- Confirm test cases
- Confirm document
- Add branch pick label

Claude-Session: https://claude.ai/code/session_0149x6DMH4C9RUmHLjNKDFxS
@mrhhsg
mrhhsg force-pushed the spill-gc-inode-monitor branch from f37fd33 to 990d180 Compare September 3, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants