Content addressable gems POC

[jenshenny]

Summary

Adds support for content-addressable ("skinny") precompiled binaries to
RubyGems and Bundler, encoded entirely in the v1 compact index — no /v2/
endpoint required.

A "skinny" binary is a platformed gem pinned to a single Ruby ABI (e.g.
required_ruby_version = "~> 3.3.0", or rake-compiler's >= 3.3, < 3.4.dev).
Today every per-Ruby build of the same name–version–platform collides on
one identity, so a registry can host only one. This PR makes such gems built,
named, installed, and resolved by content address — name-version-<sha>,
where <sha> = sha256(.gem)[0, 10] — so multiple per-Ruby builds of the same
version+platform can coexist, and the correct one is selected at resolve time
based on the running Ruby.

Old clients keep working: skinny rows in the index carry a rubygems:>=
requirement, so a client older than the cutoff transparently ignores them and
installs the ordinary fat binary instead.

Why v1 (and not a v2 endpoint)

This is the alternative to a separate /v2/ compact-index namespace. Instead of
hiding content-addressable entries behind a new endpoint, we keep everything in
the existing v1 index and hide skinny rows from old clients with a rubygems:>=
gate. The per-gem info line carries the content address in the version token's
platform slot and the real platform in a platform: requirement token:

1.0.0                  |checksum:…,ruby:>= 3.1,rubygems:>= 3.3.22
1.0.0-x86_64-linux     |checksum:…,ruby:>= 3.1
1.0.0-9f3c1a2b…        |checksum:…,ruby:~> 3.3.0,rubygems:>= 4.1.0.dev,platform:= x86_64-linux

• The <sha> slot becomes the gem's full_name (and download path
  /gems/name-version-<sha>.gem).
• The platform: token carries the real platform, used for compatibility
  matching.
• rubygems:>= 4.1.0.dev gates the row: clients older than the version that
  ships this support don't satisfy it and drop the row during resolution. (The
  exact cutoff is a placeholder to be set to the release version; consuming
  support should ship before any registry starts publishing skinny gems.)

Design

• Identity: version_suffix = sha256(.gem)[0, 10], computed from bytes, so
  the builder, the installer, and a registry all derive the same value without
  coordination.
• "Skinny" detection: a gem is content-addressable when it is platformed and
  pinned to a single Ruby minor. Fat binaries (spanning many ABIs) and pure-Ruby
  gems are untouched.
• Build: Gem::Package.build builds a skinny gem in memory, computes the
  content address from the bytes, and writes name-version-<sha>.gem in a single
  pass — no write-then-read-then-rename. (A streaming digest can't be used here
  because TarWriter seeks backward to backfill headers, so the build IO must be
  seekable.)
• Install: installs under gems/name-version-<sha>/, with the 10-char sha
  recorded in the stub line so full_name reconstructs the content-addressed
  name offline (no .gem bytes required).
• Lockfile: stays portable. The lock records name (version-platform);
  Bundler bridges that back to the on-disk name-version-<sha> at install time.
• Resolution preference: when a skinny binary compatible with the running
  Ruby exists, Bundler prefers it exclusively; otherwise it falls back to the
  fat/ruby variant so a usable binary is never stranded. Per-minor ~> ranges
  are disjoint, so at most one skinny qualifies (no skinny-vs-skinny tie-break).

Changes

RubyGems core

• lib/rubygems/platform.rb — preserve a 10-hex-char version_suffix token
  verbatim instead of normalizing it to the bogus platform "unknown";
  round-trips through to_a/to_s, and ==/hash/=== treat it as an
  exact-match token.
• lib/rubygems/specification.rb — content_addressable? /
  content_addressable_ruby_abi (skinny detection, both ~> X.Y.Z and
  rake-compiler's >= X.Y, < X.(Y+1).dev); to_ruby writes the version suffix
  as an optional 5th stub-line field.
• lib/rubygems/package.rb — Gem::Package.build builds skinny gems in
  memory and writes name-version-<sha>.gem once; the build success message
  tolerates a nil path (so IOSource in-memory builds don't crash).
• lib/rubygems/package_task.rb — moves the (renamed) built file to the
  package dir.
• lib/rubygems/basic_specification.rb — version_suffix accessor;
  full_name returns name-version-<sha> when present.
• lib/rubygems/stub_specification.rb — reads the optional 5th stub-line
  field; full_name reconstructs the content-addressed name; to_spec carries
  the suffix onto the loaded full spec.
• lib/rubygems/installer.rb — assign_version_suffix derives the sha from
  the gem's bytes before any path is computed.

Bundler

• endpoint_specification.rb — parses the compact-index platform: token
  into platform_requirement; content_addressable?; installable_on_platform?
  uses the real platform (since @platform holds the sha).
• match_platform.rb — usable_skinny? and the "prefer a Ruby-compatible
  skinny exclusively, else fall back to fat" selection.
• lazy_specification.rb — carries platform_requirement/version_suffix;
  content-addressed full_name; on a --local exact-match miss, retries by
  name+version so the portable lockfile entry resolves to the on-disk
  name-version-<sha> gem.
• stub_specification.rb — delegates full_name/version_suffix to the
  underlying RubyGems stub.

Tests / tooling

• dev/content-addressable-v1-demo/ — self-contained, dependency-free,
  native end-to-end tests (no containers, no rubygems.org): a tiny native demo
  gem, a fake v1 compact-index server, and scripts documented in a README.

Testing

Verified natively on macOS (arm64-darwin); each script ends with ALL GOOD:

• test_resolution.sh — using the real parser + EndpointSpecification +
  MatchPlatform, a new client picks the skinny variant over source+fat, and an
  old client drops it via the rubygems:>= gate.
• test_local.sh — rake native gem builds the content-addressed gem;
  gem install lands it under gems/name-version-<sha>/ with the sha in the
  stub; bundle install --local resolves it (lockfile round-trip), require +
  bundle list work, and re-install is idempotent.
• test_remote_v1.sh — against a fake v1 server, Bundler fetches
  /versions + /info/<gem> (no /v2/), selects the skinny variant, downloads
  /gems/name-version-<sha>.gem, installs, and require works.
• test_remote_v1_oldclient.sh — a stock RubyGems/Bundler older than the
  gate installs the fat binary and never requests the skinny gem, while the
  patched client picks the skinny one, against the same index.

cd dev/content-addressable-v1-demo
./test_resolution.sh
./test_local.sh
PORT=8920 ./test_remote_v1.sh
PORT=8920 ./test_remote_v1_oldclient.sh

Relationship to #168

This is the v1-index alternative to #168 (which negotiates a /v2/ compact-index
namespace). The build/install/resolution machinery is shared in spirit; the key
differences are no v2 endpoint (old clients are handled by the rubygems:>=
gate instead) and the naming (version_suffix / platform_requirement here vs
content_address / real_platform in #168).

Known nuance

On the remote path the gem currently installs into
gems/name-version-<real-platform>/, whereas the --local path installs
into gems/name-version-<sha>/. Both are internally consistent and work for a
single active Ruby, but they disagree on the on-disk directory name.