Skip to content

Conversation

@hugomrdias
Copy link
Member

  • remove pdp in sdk
  • take File as upload input
  • reorganize sp/pdp actions in core
  • add sp response validation

@hugomrdias hugomrdias requested a review from rvagg as a code owner February 6, 2026 19:36
@github-project-automation github-project-automation bot moved this to 📌 Triage in FOC Feb 6, 2026
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Feb 6, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
synapse-dev ae45545 Commit Preview URL

Branch Preview URL
Feb 10 2026, 05:14 PM

@socket-security
Copy link

socket-security bot commented Feb 6, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedp-defer@​4.0.11001007680100
Addediso-web@​2.1.0941007693100
Addedplaywright-test@​14.1.128710010083100
Addeddnum@​2.17.08610010085100
Addedchai@​6.2.29910010088100

View full report

@hugomrdias hugomrdias self-assigned this Feb 6, 2026
@hugomrdias hugomrdias moved this from 📌 Triage to 🔎 Awaiting review in FOC Feb 6, 2026
@rvagg
Copy link
Collaborator

rvagg commented Feb 8, 2026

It would have been good if you had have separated these out, removal of pdp is fine, but this needs discussion:

const file = new File([new Uint8Array([1, 2, 3, 4, 5])], "foo.txt");
const result = await synapse.storage.upload(file);

I'd like to understand why we need to require a File. The main DX problem with this pattern is that it implies sematic meaning of "foo.txt". Why would we require you wrap your bytes in a File if we're just going to extract your bytes from the File and throw away the metadata?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why PdpDataSet / get-pdp-data-set? are you worried about the confusion with pdp/get-data-set? cause I'm not sure putting "pdp" in front really helps the case. maybe you have to accept "extended" or "enhanced" here .. cause that's what it is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, sp/get-data-set will go away when we clean up the pieceStatus

theres:

  • warmstorage.getDataSet which is the barebones contract call
  • warmstorage.getPdpDateSet that adds pdp verifier data and PDP sp to the object

): Promise<getPdpDataSets.OutputType> {
const data = await getClientDataSets(client, options)

const promises = data.map(async (dataSet) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to hurt for users like filecoin-pin demo, I wouldn't be surprised if it breaks for that case

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why, too many calls ?
we were already doing all theses calls before

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the sdk does not use this action

}),
}
})
.filter((piece) => !removals.includes(piece.id)),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hah, are you sure you want to do this? this feels like opt-in behaviour to me, the pieces are still there, they're only scheduled for removal and won't be removed from on-chain or on the SP until the next proving period

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that why theres the getActivePieces action with no dedupe same output


export type CreateDataSetOptions = {
/** Whether the data set should use CDN. */
cdn: boolean
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing convention here, not a firm objection but I think you should stick to the existing convention of withCDN unless you're planning on changing all of the instances and causing even more breakage

@rvagg
Copy link
Collaborator

rvagg commented Feb 9, 2026

k, I've done a review, my biggest objection is requiring File here and removing the other optionality. Some generic streaming option that doesn't require hoops is my preference, ReadableStream<Uint8Array> is fine.

In #593 I have Uint8Array | ReadableStream<Uint8Array>, can we just stick File in that list to? And is it really that big a deal to allow users to pass in Uint8Array at the SDK level and we fix it up for them? We don't even need .from() cause it's a simple case. In fact, I believe that taking data: Uint8Array | File | ReadableStream<Uint8Array> and normalising to ReadableStream<Uint8Array> internally is this easy:

const stream =
  data instanceof ReadableStream ? data :
  data instanceof File ? data.stream() :
  new ReadableStream({ start(c) { c.enqueue(data); c.close(); } });

You could push that all the way down to uploadPieceStreaming but then those new options.data.size need to be optional, IMO they definitely should be. I know that a lot of places we will have .size, but there are going to be places where it's too awkward to have one. The streaming CAR is one of those places - it's really expensive to pre-generate a CAR, "don't duplicate my data" has been a fundamental complaint of our IPFS stack and we can avoid it easily if we don't have to have the bytes ready before we start.

Other items as mentioned:

  • PdP included in DataSet feels a bit weird, is it necessary?
  • getPieces filtering out scheduled deletions might seem right, but it's not protocol-correct, I think it should be opt-in behaviour and documenting it is a good chance for protocol communication.
  • cdn and withCDN mix / change feels unnecessary and potentially footgunny
  • The nonce creation timing for createDataSet and createDataSetAndAddPieces

Zod is good, endpoint -> serviceURL is good, normalistaion of some of the synapse-core patterns like OutputType is good. Removing PDPServer class from SDK is reasonable.

PR could have been titled better! Probably best if it's not squash merged with that change. OTY

@BigLep BigLep moved this from 🔎 Awaiting review to ⌨️ In Progress in FOC Feb 9, 2026
@BigLep BigLep mentioned this pull request Feb 10, 2026
11 tasks
@hugomrdias hugomrdias merged commit 67a17ee into master Feb 10, 2026
11 checks passed
@github-project-automation github-project-automation bot moved this from ⌨️ In Progress to 🎉 Done in FOC Feb 10, 2026
@hugomrdias hugomrdias deleted the hugomrdias/pdp-server-out branch February 10, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🎉 Done

Development

Successfully merging this pull request may close these issues.

2 participants