[FLUSS-2262][lake] Improved Stability for Iceberg Log Table Compaction Test #2265
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luoyuxia
reviewed
Dec 29, 2025
| checkFileStatusInIcebergTable(t1, 3, false); | ||
|
|
||
| // Ensure tiering job has fully processed the previous writes | ||
| assertReplicaStatus(t1Bucket, i); |
Contributor
There was a problem hiding this comment.
I'm curious about it. If the files count already is 3 in iceberg, shouldn't it also mean that the tiering has already tiered all records since we only write 3 records.
Contributor
Author
There was a problem hiding this comment.
I agree that it’s a bit odd. The additional replica assertion seems to alleviate any race conditions within this specific test case that we were seeing arise within the failing CI instances (at least based on my local testing to reproduce the issue).
It may simply be a testing artifact as opposed to a legitimate issue, but if it doesn’t seem like a fix, we can explore a few other avenues. It seemed like one of those common race conditions, so leveraging an existing function to help seemed like a decent approach. Happy to explore some additional avenues though, if we feel that we need a bit more exhaustive checks.
Contributor
There was a problem hiding this comment.
@rionmonster I tried to reporduce it in my local. But fails. I think it'll be better to explore the root cause for I'm afraid of it is caused by another critical issue.
Contributor
Author
There was a problem hiding this comment.
Good to know! It seemed to be pretty consistent regarding pass/fail on my end, but clearly there’s something else at play.
I’ll do some more exploration and see what I find. Thanks for the feedback!
Contributor
Author
There was a problem hiding this comment.
I've been digging into this a bit further and it seems there's some disparity between the actual files being written and retrieving the latest offsets for those (specifically after any types of asynchronous operations, such as compaction, etc.). I think we may need some mechanism to improve the consistency, at least within the bounds of the tests.
I wrote a little monitor to run through several iterations of the tests to see what the state of various bits looked like during each iteration. Here's a sample of those:
Successful Test (Successfully Performs Compaction as Expected)
[MONITOR] log_table_33 - TableBucket{tableId=32, bucket=0}
| IcebergFiles | IcebergSnapshotId | LakeSnapshotId | ReplicaLakeSnapId | ReplicaLakeLogEndOff | Timestamp |
|---|---|---|---|---|---|
| 0 | -1 | -1 | -1 | -1 | 1767113560800 |
| 0 | -1 | -1 | -1 | -1 | 1767113561046 |
| 0 | -1 | -1 | -1 | -1 | 1767113561294 |
| 0 | -1 | -1 | -1 | -1 | 1767113561547 |
| 0 | -1 | -1 | -1 | -1 | 1767113561795 |
| 0 | -1 | -1 | -1 | -1 | 1767113562045 |
| 0 | -1 | -1 | -1 | -1 | 1767113562299 |
| 0 | -1 | -1 | -1 | -1 | 1767113562547 |
| 0 | -1 | -1 | -1 | -1 | 1767113562797 |
| 0 | -1 | -1 | -1 | -1 | 1767113563054 |
| 0 | -1 | -1 | -1 | -1 | 1767113563298 |
| 0 | -1 | -1 | -1 | -1 | 1767113563546 |
| 0 | -1 | -1 | -1 | -1 | 1767113563796 |
| 0 | -1 | -1 | -1 | -1 | 1767113564048 |
| 0 | -1 | -1 | -1 | -1 | 1767113564295 |
| 0 | -1 | -1 | -1 | -1 | 1767113564544 |
| 0 | -1 | -1 | -1 | -1 | 1767113564796 |
| 0 | -1 | -1 | -1 | -1 | 1767113565047 |
| 0 | -1 | -1 | -1 | -1 | 1767113565295 |
| 0 | -1 | -1 | -1 | -1 | 1767113565545 |
| 0 | -1 | -1 | -1 | -1 | 1767113565797 |
| 0 | -1 | -1 | -1 | -1 | 1767113566046 |
| 0 | -1 | -1 | -1 | -1 | 1767113566294 |
| 0 | -1 | -1 | -1 | -1 | 1767113566545 |
| 0 | -1 | -1 | -1 | -1 | 1767113566795 |
| 0 | -1 | -1 | -1 | -1 | 1767113567047 |
| 0 | -1 | -1 | -1 | -1 | 1767113567296 |
| 0 | -1 | -1 | -1 | -1 | 1767113567547 |
| 0 | -1 | -1 | -1 | -1 | 1767113567793 |
| 0 | -1 | -1 | -1 | -1 | 1767113568048 |
| 0 | -1 | -1 | -1 | -1 | 1767113568296 |
| 0 | -1 | -1 | -1 | -1 | 1767113568546 |
| 0 | -1 | -1 | -1 | -1 | 1767113568797 |
| 0 | -1 | -1 | -1 | -1 | 1767113569048 |
| 0 | -1 | -1 | -1 | -1 | 1767113569297 |
| 0 | -1 | -1 | -1 | -1 | 1767113569547 |
| 0 | -1 | -1 | -1 | -1 | 1767113569796 |
| 0 | -1 | -1 | -1 | -1 | 1767113570045 |
| 0 | -1 | -1 | -1 | -1 | 1767113570298 |
| 0 | -1 | -1 | -1 | -1 | 1767113570547 |
| 0 | -1 | -1 | -1 | -1 | 1767113570799 |
| 0 | -1 | -1 | -1 | -1 | 1767113571045 |
| 0 | -1 | -1 | -1 | -1 | 1767113571294 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113571547 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113571799 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113572047 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113572301 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113572549 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113572800 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113573044 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113573301 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113573549 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113573798 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113574048 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113574298 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113574551 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113574801 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113575047 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113575300 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113575549 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113575796 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113576051 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113576300 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113576547 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113576800 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113577048 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113577304 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113577551 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113577800 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113578046 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113578300 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113578550 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113578800 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113579050 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113579301 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113579548 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113579799 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113580048 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113580298 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113580550 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113580798 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113581056 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113581297 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113581550 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113581801 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113582049 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113582301 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113582548 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113582798 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113583046 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113583298 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113583548 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113583800 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113584054 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113584303 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113584550 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113584804 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113585051 |
| 1 | 5443797100773076340 | 5443797100773076340 | 5443797100773076340 | 1 | 1767113585294 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113585558 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113585810 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113586062 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113586310 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113586559 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113586805 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113587059 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113587307 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113587559 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113587810 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113588056 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113588310 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113588558 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113588809 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113589059 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113589310 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113589558 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113589813 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113590057 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113590309 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113590557 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113590807 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113591058 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113591311 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113591557 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113591811 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113592058 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113592310 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113592560 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113592809 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113593058 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113593310 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113593559 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113593810 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113594057 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113594312 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113594560 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113594808 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113595063 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113595309 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113595559 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113595810 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113596060 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113596316 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113596558 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113596808 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113597060 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113597312 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113597557 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113597809 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113598057 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113598307 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113598558 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113598817 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113599062 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113599311 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113599562 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113599809 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113600058 |
| 2 | 5171148310110237921 | 5171148310110237921 | 5443797100773076340 | 2 | 1767113600306 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113600560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113600811 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113601060 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113601309 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113601560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113601807 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113602059 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113602307 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113602560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113602809 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113603057 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113603307 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113603561 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113603806 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113604057 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113604307 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113604553 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113604812 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113605056 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113605311 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113605559 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113605809 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113606058 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113606308 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113606559 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113606811 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113607060 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113607314 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113607557 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113607808 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113608059 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113608310 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113608561 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113608811 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113609056 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113609309 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113609560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113609810 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113610060 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113610312 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113610560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113610811 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113611061 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113611308 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113611554 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113611808 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113612060 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113612311 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113612561 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113612811 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113613062 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113613311 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113613561 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113613808 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113614061 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113614310 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113614560 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113614809 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113615059 |
| 3 | 5182733673261799288 | 5182733673261799288 | 5443797100773076340 | 3 | 1767113615307 |
| 2 | 2575057976625237982 | 2575057976625237982 | 5443797100773076340 | 4 | 1767113615557 |
| 2 | 2575057976625237982 | 2575057976625237982 | 5443797100773076340 | 4 | 1767113615808 |
Failing Test (File appeared to never be properly written before expected offset)
[MONITOR] log_table_34 - TableBucket{tableId=33, bucket=0}
| IcebergFiles | IcebergSnapshotId | LakeSnapshotId | ReplicaLakeSnapId | ReplicaLakeLogEndOff | Timestamp |
|---|---|---|---|---|---|
| 0 | -1 | -1 | -1 | -1 | 1767113616327 |
| 0 | -1 | -1 | -1 | -1 | 1767113616573 |
| 0 | -1 | -1 | -1 | -1 | 1767113616823 |
| 0 | -1 | -1 | -1 | -1 | 1767113617072 |
| 0 | -1 | -1 | -1 | -1 | 1767113617326 |
| 0 | -1 | -1 | -1 | -1 | 1767113617569 |
| 0 | -1 | -1 | -1 | -1 | 1767113617826 |
| 0 | -1 | -1 | -1 | -1 | 1767113618068 |
| 0 | -1 | -1 | -1 | -1 | 1767113618326 |
| 0 | -1 | -1 | -1 | -1 | 1767113618574 |
| 0 | -1 | -1 | -1 | -1 | 1767113618823 |
| 0 | -1 | -1 | -1 | -1 | 1767113619071 |
| 0 | -1 | -1 | -1 | -1 | 1767113619321 |
| 0 | -1 | -1 | -1 | -1 | 1767113619572 |
| 0 | -1 | -1 | -1 | -1 | 1767113619825 |
| 0 | -1 | -1 | -1 | -1 | 1767113620069 |
| 0 | -1 | -1 | -1 | -1 | 1767113620322 |
| 0 | -1 | -1 | -1 | -1 | 1767113620576 |
| 0 | -1 | -1 | -1 | -1 | 1767113620822 |
| 0 | -1 | -1 | -1 | -1 | 1767113621069 |
| 0 | -1 | -1 | -1 | -1 | 1767113621324 |
| 0 | -1 | -1 | -1 | -1 | 1767113621572 |
| 0 | -1 | -1 | -1 | -1 | 1767113621826 |
| 0 | -1 | -1 | -1 | -1 | 1767113622072 |
| 0 | -1 | -1 | -1 | -1 | 1767113622321 |
| 0 | -1 | -1 | -1 | -1 | 1767113622574 |
| 0 | -1 | -1 | -1 | -1 | 1767113622823 |
| 0 | -1 | -1 | -1 | -1 | 1767113623069 |
| 0 | -1 | -1 | -1 | -1 | 1767113623324 |
| 0 | -1 | -1 | -1 | -1 | 1767113623573 |
| 0 | -1 | -1 | -1 | -1 | 1767113623821 |
| 0 | -1 | -1 | -1 | -1 | 1767113624070 |
| 0 | -1 | -1 | -1 | -1 | 1767113624320 |
| 0 | -1 | -1 | -1 | -1 | 1767113624569 |
| 0 | -1 | -1 | -1 | -1 | 1767113624821 |
| 0 | -1 | -1 | -1 | -1 | 1767113625073 |
| 0 | -1 | -1 | -1 | -1 | 1767113625324 |
| 0 | -1 | -1 | -1 | -1 | 1767113625573 |
| 0 | -1 | -1 | -1 | -1 | 1767113625824 |
| 0 | -1 | -1 | -1 | -1 | 1767113626071 |
| 0 | -1 | -1 | -1 | -1 | 1767113626319 |
| 0 | -1 | -1 | -1 | -1 | 1767113626573 |
| 0 | -1 | -1 | -1 | -1 | 1767113626824 |
| 0 | -1 | -1 | -1 | -1 | 1767113627070 |
| 0 | -1 | -1 | -1 | -1 | 1767113627324 |
| 0 | -1 | -1 | -1 | -1 | 1767113627574 |
| 0 | -1 | -1 | -1 | -1 | 1767113627818 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113628077 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113628325 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113628580 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113628828 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113629073 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113629330 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113629577 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113629829 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113630076 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113630324 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113630578 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113630828 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113631075 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113631326 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113631580 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113631827 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113632076 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113632326 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113632577 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113632826 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113633074 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113633325 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113633578 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113633829 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113634077 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113634327 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113634573 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113634826 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113635072 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113635326 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113635582 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113635828 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113636072 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113636327 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113636576 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113636826 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113637076 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113637327 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113637576 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113637829 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113638076 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113638326 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113638580 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113638827 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113639077 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113639327 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113639575 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113639829 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113640073 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113640326 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113640575 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113640827 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113641071 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113641323 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113641576 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113641834 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113642073 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113642330 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113642574 |
| 1 | 6752171114485570420 | 6752171114485570420 | 6752171114485570420 | 1 | 1767113642820 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113643084 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113643343 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113643586 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113643835 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113644086 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113644333 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113644589 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113644833 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113645090 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113645339 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113645588 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113645838 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113646087 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113646340 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113646591 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113646842 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113647089 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113647340 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113647593 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113647847 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113648089 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113648337 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113648593 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113648839 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113649086 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113649339 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113649593 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113649843 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113650094 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113650345 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113650597 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113650842 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113651089 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113651339 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113859238 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113859394 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113859633 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113859872 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113860134 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113860382 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113860628 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113860879 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113861129 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113861384 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113861627 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113861882 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113862135 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113862381 |
| 2 | 7273969972093574431 | 7273969972093574431 | 7273969972093574431 | 2 | 1767113862633 |
[ASSERTION FAILURE] Expected offset 3 but got 2 for bucket TableBucket{tableId=33, bucket=0}
Replica Lake Snapshot ID: 7273969972093574431
Current State:
Iceberg Files: 2
Iceberg Snapshot ID: 7273969972093574431
Lake Snapshot ID (from admin): 7273969972093574431
Replica Lake Snapshot ID: 7273969972093574431
Replica Lake Log End Offset: 2
I'm not sure if this is more of an artifact of the tests themselves or a legitimate issue. Any thoughts? Happy to continue digging. It feels like a race-condition due to the inconsistency, either on writing the files or reading stale offsets from the data lake directly. I'm sure in a real-world environment, this might be tolerable (as I suspect it's just a minor latency spike which would eventually resolve), but in the confines of a test, it's flaky.
luoyuxia
reviewed
Dec 29, 2025
Contributor
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@rionmonster Thanks for the pr. Only one question
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #2262
Per Issue #2262, this pull request addresses a race condition that could sometimes occur and result in the
IcebergRewriteITCase.testLogTableCompactiontest case failing (particularly during CI builds).Brief change log
This change updates the
testLogTableCompactiontest with an additional call to the existingassertReplicaStatus()function to ensure that the associated tiering job had fully processed the previous rewrites to avoid the race condition prior to the subsequent assertions.Tests
The
IcebergRewriteITCase.testLogTableCompactionwas initially updated to use an iterative approach (e.g., repeat 50 times) as mentioned in the original issue to reproduce the issue. After this was repeatably reproducible, the proposed fix was introduced to verify the test would repeatedly pass through all of the iterations successfully and repeatably.API and Format
N/A
Documentation
N/A
Reviewer(s) Requested
@swuferhong (as original reporter), @beryllw (as original author)