[SYSTEMDS-3946] Merge PR Enable sending of large (>2GiB) FederatedRequests - #2591
[SYSTEMDS-3946] Merge PR Enable sending of large (>2GiB) FederatedRequests#2591ywcb00 wants to merge 14 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2591 +/- ##
============================================
- Coverage 71.39% 71.39% -0.01%
- Complexity 50539 50586 +47
============================================
Files 1632 1637 +5
Lines 196237 196452 +215
Branches 38201 38224 +23
============================================
+ Hits 140113 140261 +148
- Misses 45099 45162 +63
- Partials 11025 11029 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
aa8c801 to
6437c2a
Compare
ywcb00
changed the title
|
The python federated CI test is failing due to the code changes of this PR. However, I cannot explain why the current changes cause the test to fail. I ran the test on two different machines w/ three different setups but was unable to reproduce the error. |
… Responses Federated transfers previously failed for payloads above 2GiB because the single Netty frame size is bounded by a 32-bit length field, capping any request or response at Integer.MAX_VALUE bytes. This patch adds a streaming chunked codec that splits a large payload into bounded frames on the sender and reassembles them on the receiver, so the on-wire size is no longer limited by a single frame. A format detector and format encoder select the chunked path only when the payload exceeds the frame limit, leaving the existing small-message path unchanged to avoid added overhead for the common case. Adds FederatedMaxPayloadTest to exercise the boundary around the former 2GiB cap.
Document the codec on methods and fail fast on an unknown frame type in the chunk decoder.
Both negative tests go through a helper, since writeInbound rethrows if the deserializer thread wins the race.
….java): change class and members to visibility public refactor(test/component/federated/**): use the constants from the federated chunk protocol general refactoring refactor(test/functions/federated/network/FederatedMaxPayloadTest.java): general refactoring
…ove test to io folder instead of network folder and delete the empty network folder
…e federated request encoder again fix(main/runtime/controlprogram/federated/FederatedFormatEncoder.java): throw an io exception when the message is not serializable instead of writing it to the channel
…dify the predicate if it is reusable by lineage to always be true when the lineage object is not null chore(test/component/federated/FederatedChunkCodecTest.java): change polling for exceptions etc. to try fewer times and wait longer
apply formatting
…: fix problem with failing python test in cases with low computing power (e.g., only two cores; set by 'taskset 0,1 ./runTest)
|
Thanks to @Biranavan-Parameswaran, I was able to reproduce the error. taskset -c 0,1 ./tests/federated/runFedTest.sh@Biranavan-Parameswaran also directly provided a patch for preventing this error (applied in commit c17574a). Thank you very much. :) |
2 participants
Intermediate PR for merging #2496