feat: add 53-bit unique ID generator & database-generated sharded sequences by levkk · Pull Request #858 · pgdogdev/pgdog

levkk · 2026-03-30T18:06:36Z

Smaller unique ID

This feature is still experimental and subject to change.

Add support for 53-bit unique ID. This makes them smaller and "javascript-safe" in case identifiers are exposed as integers to the app API.

This is configurable thourough pgdog.toml:

[general]
unique_id_function = "compact" # or "standard", which is default

Trade-offs: it only allows 64 pgdog nodes and 64,000 IDs generated per second per node. This is considerably lower than the standard 64-bit unique ID. It should be used for omnisharded tables only with a low write frequency.

Database-generated sharded sequences

Add the ability to generate cross-shard unique IDs using sequences. This produces smaller starting integers, but can only work with direct-to-shard INSERTs. For omnisharded tables, the unique ID function should continue to be used.

This is configurable through pgdog.toml:

[rewrite]
primary_key = "rewrite_omni" # Will only inject Unique ID to omnisharded table inserts, leaving others to be
                                                # generated by the database instead.

The sharded sequence is installed automatically on all tables with a BIGINT primary key when running pgdog setup or SETUP SCHEMA via the admin DB.

Identity columns

GENERATED ... WITH IDENTITY is now removed entirely during schema sync. This constraint blocks our implementation of replication and doesn't work with sharded databases, incl. with our unique ID generator because it prevents the column value from being inserted by the query.

Smaller features

Add RESET PREPARED admin command which evicts all unused prepared statements from the global cache
log any notice/warning received from Postgres when executing queries via execute_checked

Bug fixes

unique_id_min setting was being ignored by the unique ID generator. This didn't affect any deployments because unique IDs start very large already.
Admin command SETUP SCHEMA was applied to all database/user pairs; it's now applied to the schema_owner only

codecov · 2026-03-30T18:20:58Z

Codecov Report

❌ Patch coverage is 82.32446% with 73 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog/src/unique_id.rs	86.61%	17 Missing ⚠️
pgdog/src/admin/reset_prepared.rs	0.00%	13 Missing ⚠️
pgdog-config/src/sharding.rs	0.00%	11 Missing ⚠️
pgdog/src/backend/schema/mod.rs	92.24%	10 Missing ⚠️
pgdog/src/backend/databases.rs	0.00%	8 Missing ⚠️
...og/src/backend/replication/logical/orchestrator.rs	0.00%	3 Missing ⚠️
pgdog-config/src/rewrite.rs	0.00%	2 Missing ⚠️
pgdog-stats/src/schema.rs	0.00%	2 Missing ⚠️
pgdog/src/admin/parser.rs	33.33%	2 Missing ⚠️
pgdog/src/admin/setup_schema.rs	0.00%	2 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

meskill · 2026-03-31T13:45:25Z

pgdog/src/unique_id.rs

+        // Compact (JS-safe) IDs only have 6 node bits.
+        if node_id > COMPACT_MAX_NODE_ID {
+            return Err(Error::CompactNodeIdTooLarge(node_id));
+        }
+


that's probably should be checked only if the function is compact. For standard it could cause unexpected error

Good catch!

meskill · 2026-03-31T14:12:23Z

pgdog/src/unique_id.rs

        if min_id > MAX_OFFSET {
            return Err(Error::OffsetTooLarge(min_id));
        }


check is done only for MAX_OFFSET which is based on the standard generator, but it really depends on the actual layout.

might worth to move the checks into the id_type or state itself to avoid adding more conditions here and be ready for other extensions

meskill · 2026-03-31T14:15:33Z

pgdog/src/unique_id.rs

@@ -31,6 +32,14 @@ const TIMESTAMP_SHIFT: u8 = (SEQUENCE_BITS + NODE_BITS) as u8; // 22
 const MAX_OFFSET: u64 = i64::MAX as u64


btw, MAX_OFFSET if effectively 0. That makes config.general.unique_id_function unusable except for default 0 value. It seems unit tests are bypasses this since the offset tests use next_id directly that has no validation for MAX_OFFSET

I'm not sure what you mean. Could you elaborate? I think we test the min ID in unit tests by passing in as an argument.

here

if we do the math the value is 0. And the tests are not catching this because the tests for id_offset are only calling next_id function that has no validation.

Nice catch. Let me see what we can do here...

meskill · 2026-04-01T21:48:32Z

pgdog/src/unique_id.rs

        if now >= target_ms {
            return now;
        }
        thread::sleep(Duration::from_millis(1));


a sidenote:
I wonder if we really need the sleep here to correct the clock drift. It seems like it's used to make sure the call to SystemTime::now() fits our expectation for monotonically increasing timestamp. But it's not strictly necessary for this calculation since it could be emulated by now_ms().max(self.last_timestamp_ms). Yes, it won't be not exactly the system time, but we fulfill the requirements.
Just because the sleep is not reliable and scheduler could skew the time in this case

I think we're just trying to guarantee that the next call to now() returns the subsequent second or greater. This is used when the internal sequence is exhausted and we have to wait until next clock tick to guarantee that the sequence is monotonically increasing.

This reverts commit 8e9f761.

meskill reviewed Mar 31, 2026

View reviewed changes

levkk force-pushed the levkk-configurable-unique-id branch from 7256483 to 6a91de6 Compare April 1, 2026 19:04

meskill reviewed Apr 1, 2026

View reviewed changes

levkk force-pushed the levkk-configurable-unique-id branch 3 times, most recently from 9614cbe to 1778c85 Compare April 7, 2026 21:54

levkk changed the title ~~feat: 53-bit unique id~~ feat: add 53-bit unique ID generator Apr 7, 2026

levkk changed the title ~~feat: add 53-bit unique ID generator~~ feat: add 53-bit unique ID generator & database-generated sharded sequences Apr 7, 2026

levkk added 17 commits April 8, 2026 07:56

feat: 53-bit unique id

bd48013

json schema

68d6e9a

wip: passthrough id generation

f3805c8

ok this works

69720b8

ok

38c1483

fix checks

24625c0

schema

c493319

woops

b9e3e40

rewrite default before early return

cbbe7d6

handle with identity columns

72d4d65

feat: handle partitioned identity columns

57ef3fd

handle partitioned tables correctly

f9c3833

apply sharded sequence to all primary bigint primary keys

9f536de

fix rebase

f1425c7

remove dead code

3c329a9

fix identity

18123b7

cleanup

3108b04

levkk force-pushed the levkk-configurable-unique-id branch from 702f74a to 3108b04 Compare April 8, 2026 14:56

levkk added 2 commits April 8, 2026 08:26

gate identity columns to pg17+

8e9f761

Revert "gate identity columns to pg17+"

22fb0f6

This reverts commit 8e9f761.

levkk merged commit 38eccf3 into main Apr 8, 2026
10 of 11 checks passed

levkk deleted the levkk-configurable-unique-id branch April 8, 2026 15:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add 53-bit unique ID generator & database-generated sharded sequences#858

feat: add 53-bit unique ID generator & database-generated sharded sequences#858
levkk merged 19 commits intomainfrom
levkk-configurable-unique-id

levkk commented Mar 30, 2026 •

edited

Loading

Uh oh!

codecov bot commented Mar 30, 2026 •

edited

Loading

Uh oh!

meskill Mar 31, 2026

Uh oh!

levkk Apr 1, 2026

Uh oh!

meskill Mar 31, 2026 •

edited

Loading

Uh oh!

meskill Mar 31, 2026

Uh oh!

meskill Mar 31, 2026

Uh oh!

levkk Apr 1, 2026

Uh oh!

meskill Apr 1, 2026

Uh oh!

levkk Apr 2, 2026

Uh oh!

meskill Apr 1, 2026

Uh oh!

levkk Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -31,6 +32,14 @@ const TIMESTAMP_SHIFT: u8 = (SEQUENCE_BITS + NODE_BITS) as u8; // 22
		const MAX_OFFSET: u64 = i64::MAX as u64

Conversation

levkk commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Smaller unique ID

Database-generated sharded sequences

Identity columns

Smaller features

Bug fixes

Uh oh!

codecov bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

meskill Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

levkk commented Mar 30, 2026 •

edited

Loading

codecov bot commented Mar 30, 2026 •

edited

Loading

meskill Mar 31, 2026 •

edited

Loading