Support For Large File Upload by PujaDeshmukh17 · Pull Request #277 · cap-js/sdm

PujaDeshmukh17 · 2026-06-17T07:47:41Z

Describe your changes

This PR adds support for uploading files larger than 400 MB to SAP Document Management (SDM) without causing out-of-memory errors.

What changed:

lib/handler/index.js — createAttachment now routes files >400 MB through a new chunked upload path: creates an empty placeholder document in SDM first, then streams the file in 20 MB chunks via appendContentStream. Files ≤400 MB continue through the existing single-POST path unchanged.
lib/ReadAheadStream.js (new) — A bounded read-ahead buffer that pre-loads up to 4 chunks (80 MB max) while the previous chunk is being uploaded, improving throughput. Handles both Buffer and Readable stream inputs, with exponential backoff retry for transient read errors and clean handling of client disconnects.
lib/sdm.js + lib/persistence/index.js — An orphan queue tracks the SDM objectId of any in-progress chunked upload. If the upload fails mid-way, the incomplete document is deleted with retry backoff; if cleanup also fails, the orphan queue entry survives so a startup reconciliation job can clean it up later.
lib/util/index.js — Added getContentLength helper to determine file size when contentLength is not set by the caller.

Any documentation

Test was performed on file sizes 500MB, 1GB and 2GB

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist before requesting a review

I have tested the functionality on my cloud environment.
I have provided sufficient automated/ unit tests for the code.
I have increased or maintained the test coverage.
I have ran integration tests on my cloud environment.
I have validated blackduck portal for any vulnerability after my commit.

Upload Screenshots/lists of the scenarios tested

I have Uploaded Screenshots or added lists of the scenarios tested in description

- Add ReadAheadStream: parallel read-ahead queue (4×20MB = 80MB max), backpressure, exponential-backoff retry, client-disconnect detection - handler/index.js: route uploads >400MB to chunked CMIS appendContent path; single-chunk path unchanged for files <=400MB; inline cleanup of incomplete documents retries up to 3 times (2s/4s/8s backoff) - util/index.js: add getContentLength() to detect content size from Buffer, Readable stream, or size-bearing objects - index.cds: add sap.sdm.OrphanCleanupQueue entity to persist objectIds of documents that could not be cleaned up inline - persistence/index.js: enqueueOrphan / dequeueOrphan / getAllOrphans - sdm.js: read Content-Length header and pass contentLength + orphan queue callbacks into attachment data; reconciliation job runs on server startup (cds.on served) to delete any persisted orphans

Root cause (from CF logs, exit 137 OOM on 1GB file): - uploadLargeFileInChunks called streamToBuffer() which collected the entire 1GB content into a single Buffer before ReadAheadStream could process it, negating the chunked upload design entirely. Fixes: - Remove streamToBuffer() call; pass req.data.content directly to ReadAheadStream — the Buffer is already in memory from CDS body parsing, no second copy needed. - ReadAheadStream._preloadChunks: add zero-copy Buffer fast path using buf.slice() references instead of allocUnsafe+copy for each chunk. Queue holds ≤4 slice refs (no extra memory) rather than 4×20MB copies. - ReadAheadStream.startReading: poll for first chunk explicitly so _loadNextChunk is only called once the queue has data (avoids race). - package.json: raise CDS body_parser limit to 2gb so the request body is not rejected before reaching the upload handler. Note: the CF app manifest memory quota must also be raised to ≥2.5GB to accommodate the 1GB body buffer + Node.js overhead + 80MB queue.

…ap-js/sdm into SDMEXT-largeFileUpload-feature

rashmiangadi05 · 2026-06-22T06:36:06Z

 const config = {
  verbose: true,
  testTimeout: 100000,
+  forceExit: true,


why do v need this

rashmiangadi05 · 2026-06-22T06:36:31Z

+ * chunked upload did not complete successfully and could not be deleted inline.
+ * A reconciliation job on server startup retries deletion of each entry.
+ */
+entity sap.sdm.OrphanCleanupQueue {


why do v need this entity?

rashmiangadi05 · 2026-06-22T06:36:53Z

    "lint": "npx eslint --fix . --no-cache"
  },
  "cds": {
+    "server": {


does this limit upload to 2gb

rashmiangadi05 · 2026-06-22T06:37:57Z

+
+    console.log(`[orphan-queue] Reconciling ${orphans.length} orphaned SDM document(s)...`);
+
+    for (const orphan of orphans) {


can we document this somewhere in wiki how does this work.

PujaDeshmukh17 added 4 commits June 11, 2026 16:21

Merge branch 'develop' into SDMEXT-largeFileUpload-feature

8785d10

Unit tests for large file upload

5e42736

PujaDeshmukh17 had a problem deploying to pr-analysis June 17, 2026 07:47 — with GitHub Actions Failure

PujaDeshmukh17 temporarily deployed to pr-analysis June 17, 2026 07:47 — with GitHub Actions Inactive

github-code-quality Bot found potential problems Jun 17, 2026

View reviewed changes

Comment thread lib/ReadAheadStream.js Fixed

Comment thread lib/handler/index.js Fixed

Comment thread lib/handler/index.js Fixed

Comment thread test/lib/handler/index.test.js Fixed

Comment thread test/lib/handler/index.test.js Fixed

Removing unused variables

f9551a9

PujaDeshmukh17 temporarily deployed to pr-analysis June 17, 2026 07:54 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 17, 2026 07:54 — with GitHub Actions Failure

Increased test coverage

f722d55

PujaDeshmukh17 temporarily deployed to pr-analysis June 17, 2026 09:02 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 17, 2026 09:02 — with GitHub Actions Failure

UT fix

79f3a87

PujaDeshmukh17 temporarily deployed to pr-analysis June 17, 2026 17:32 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 17, 2026 17:32 — with GitHub Actions Failure

Increasing test coverage

82f613c

PujaDeshmukh17 temporarily deployed to pr-analysis June 18, 2026 04:39 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 18, 2026 04:39 — with GitHub Actions Failure

github-code-quality Bot found potential problems Jun 18, 2026

View reviewed changes

Comment thread test/lib/handler/index.test.js Fixed

Comment thread test/lib/util/index.test.js Fixed

Removing unused variables

f430427

PujaDeshmukh17 temporarily deployed to pr-analysis June 18, 2026 04:44 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 18, 2026 04:44 — with GitHub Actions Failure

Merge branch 'develop' into SDMEXT-largeFileUpload-feature

98c1d2f

PujaDeshmukh17 temporarily deployed to pr-analysis June 18, 2026 04:48 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 18, 2026 04:48 — with GitHub Actions Failure

PujaDeshmukh17 added 2 commits June 18, 2026 10:25

Removing unused variable

621229c

Merge branch 'SDMEXT-largeFileUpload-feature' of https://github.com/c…

7a4bdcc

…ap-js/sdm into SDMEXT-largeFileUpload-feature

PujaDeshmukh17 temporarily deployed to pr-analysis June 18, 2026 04:57 — with GitHub Actions Inactive

PujaDeshmukh17 had a problem deploying to pr-analysis June 18, 2026 04:57 — with GitHub Actions Failure

PujaDeshmukh17 had a problem deploying to pr-analysis June 18, 2026 05:28 — with GitHub Actions Failure

PujaDeshmukh17 requested a review from rashmiangadi05 June 18, 2026 05:29

Sonar fix

42d494e

PujaDeshmukh17 temporarily deployed to pr-analysis June 18, 2026 06:07 — with GitHub Actions Inactive

rashmiangadi05 reviewed Jun 22, 2026

View reviewed changes

PujaDeshmukh17 added 2 commits June 22, 2026 14:04

InsufficientDataException fix

ba5a5a0

Fix for chunk size

e06dd02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support For Large File Upload#277

Support For Large File Upload#277
PujaDeshmukh17 wants to merge 15 commits into
developfrom
SDMEXT-largeFileUpload-feature

PujaDeshmukh17 commented Jun 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rashmiangadi05 Jun 22, 2026

Uh oh!

rashmiangadi05 Jun 22, 2026

Uh oh!

rashmiangadi05 Jun 22, 2026

Uh oh!

rashmiangadi05 Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		console.log(`[orphan-queue] Reconciling ${orphans.length} orphaned SDM document(s)...`);

		for (const orphan of orphans) {

Conversation

PujaDeshmukh17 commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes

Any documentation

Type of change

Checklist before requesting a review

Upload Screenshots/lists of the scenarios tested

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rashmiangadi05 Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

rashmiangadi05 Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

rashmiangadi05 Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

rashmiangadi05 Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PujaDeshmukh17 commented Jun 17, 2026 •

edited

Loading