Skip to content

[Subtask]: Modify DefaultTableService to be compatible with master-slave mode.#3927

Merged
xxubai merged 16 commits intoapache:masterfrom
wardlican:amoro#3923
Mar 25, 2026
Merged

[Subtask]: Modify DefaultTableService to be compatible with master-slave mode.#3927
xxubai merged 16 commits intoapache:masterfrom
wardlican:amoro#3923

Conversation

@wardlican
Copy link
Copy Markdown
Contributor

Why are the changes needed?

In DefaultTable Service, it is compatible with master-slave mode. In master-slave mode, it is necessary to load tables from catalog, determine the addition and deletion of tables, complete the allocation of tables to bucketId, and the logic of ams nodes loading the tables they are responsible for based on bucketId.

Close #3923 .

Brief change log

  • Add a bucketTableSyncScheduler to DefaultTableService for table loading logic in master-slave mode.
  • The history of adding upgrades and switching to master-slave mode is that the allocation table is initialized with SQL for bucket allocation.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions
Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Dec 15, 2025
@wardlican
Copy link
Copy Markdown
Contributor Author

Please perform CR on these implementations.

@github-actions github-actions Bot removed the stale label Dec 16, 2025
@github-actions
Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jan 15, 2026
@github-actions
Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Jan 22, 2026
@czy006 czy006 reopened this Feb 26, 2026
@github-actions github-actions Bot removed the stale label Feb 27, 2026
@github-actions github-actions Bot added the type:docs Improvements or additions to documentation label Mar 19, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 19, 2026

Codecov Report

❌ Patch coverage is 9.60699% with 207 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.65%. Comparing base (cc03949) to head (20f0632).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...apache/amoro/server/table/DefaultTableService.java 5.58% 179 Missing and 7 partials ⚠️
...apache/amoro/server/persistence/BucketIdCount.java 0.00% 7 Missing ⚠️
...org/apache/amoro/server/AmoroServiceContainer.java 25.00% 5 Missing and 1 partial ⚠️
...java/org/apache/amoro/server/AmsAssignService.java 57.14% 2 Missing and 1 partial ⚠️
...ats/iceberg/maintainer/IcebergTableMaintainer.java 0.00% 3 Missing ⚠️
...o/server/ha/DataBaseHighAvailabilityContainer.java 0.00% 1 Missing ⚠️
...amoro/server/ha/NoopHighAvailabilityContainer.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3927      +/-   ##
============================================
- Coverage     29.90%   29.65%   -0.26%     
+ Complexity     4160     4155       -5     
============================================
  Files           670      667       -3     
  Lines         53533    53598      +65     
  Branches       6797     6797              
============================================
- Hits          16008    15893     -115     
- Misses        36343    36533     +190     
+ Partials       1182     1172      -10     
Flag Coverage Δ
core 29.65% <9.60%> (-0.26%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@czy006 czy006 requested review from czy006 and xxubai March 19, 2026 13:50
Comment thread amoro-common/src/main/java/org/apache/amoro/ServerTableIdentifier.java Outdated
Comment thread amoro-ams/src/main/resources/mysql/upgrade.sql
Comment thread amoro-ams/src/main/java/org/apache/amoro/server/AmoroManagementConf.java Outdated
Comment thread amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableService.java Outdated
Comment thread amoro-ams/src/main/java/org/apache/amoro/server/AmoroManagementConf.java Outdated
Comment thread amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableService.java Outdated
Comment thread amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableService.java Outdated
// Create and start AmsAssignService for bucket assignment
// The factory will handle different HA types (ZK, database, etc.)
amsAssignService = new AmsAssignService(haContainer, serviceConfig);
bucketAssignStore = BucketAssignStoreFactory.create(haContainer, serviceConfig);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, when BucketAssignStore initialization fails, the service does not fully fall back to non-master-slave behavior. initialize() falls back to loading all table runtimes because bucketAssignStore is null, but other paths still keep master-slave semantics, for example only the leader runs exploreTableRuntimes() and triggerTableAdded() may still return early without creating local runtimes. This can leave the in-memory table runtime state inconsistent with the expected runtime state derived from the database.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is indeed a risk associated with this bug; I will fix it.

Copy link
Copy Markdown
Contributor

@xxubai xxubai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xxubai xxubai merged commit bf222b7 into apache:master Mar 25, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module type:docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask]: Modify DefaultTableService to be compatible with master-slave mode

4 participants