Add corpus-based verification for the metric report

[ebihara99999]

Summary

Add corpus-based verification so measurement values and scope discovery stay correct as the metric set grows. The work splits into a deterministic CI layer and an out-of-band harness.

Background

The unit specs already verify that CodeKeeper calls the RuboCop calculators faithfully on minimal fixtures. Three things are not covered:

• Equivalence with the output of RuboCop's actual cop pipeline, not just the calculator utilities.
• Scope discovery on real-world code: CodeKeeper must find the same def/defs scopes that RuboCop reports offenses for.
• Behavior at scale: encodings, DSL-heavy files, very large classes.

Real-world Ruby code provides the input distribution, and RuboCop acts as a value oracle for RuboCop-backed metrics, so expected values do not need to be written by hand.

Design

Two layers:

1. CI layer: a corpus vendored into the repository, exercised by specs. Deterministic and offline.
2. Out-of-band harness (not CI): full-repository sweeps for crash and encoding hunting, golden regeneration, and timing. Network-dependent and slow by nature.

Corpus

gitlab-org/gitlab at revision 85ed8239cfc6fe3cf5d5bde5975fe2f6d687ff6d, CE files only.

License: in the GitLab repository, ee/ and jh/ are covered by proprietary licenses and doc/ by CC BY-SA 4.0, so files from those directories must not be vendored. Files outside them are MIT (Expat); every selected file falls in that scope. The vendored corpus directory must carry the GitLab MIT license notice and a README recording the source repository, revision, and file list.

Planned location: spec/fixtures/corpus/gitlab/, which keeps the files out of the packaged gem and out of RuboCop's scan (spec/fixtures/** is the excluded path). Goldens and the accepted-differences baseline are data files under spec/fixtures/. Regeneration helpers live under spec/ as ordinary RuboCop-checked code; only the packaged gem exclusion applies to them.

Breadth files (13):

• app/models/project.rb (4147-line class)
• app/models/merge_request.rb
• app/models/concerns/has_repository.rb
• app/services/merge_requests/create_service.rb
• app/services/ci/create_pipeline_service.rb
• app/graphql/types/project_type.rb (GraphQL DSL)
• app/policies/project_policy.rb (policy DSL)
• lib/gitlab/ci/config/entry/job.rb
• lib/gitlab/ci/config/entry/rules.rb
• lib/gitlab/ci/pipeline/chain/validate/abilities.rb
• lib/gitlab/background_migration/backfill_award_emoji_sharding_key.rb
• lib/gitlab/database/migrations/runner_backoff/communicator.rb
• config/routes.rb (route-drawing DSL: one method definition and no class-like scopes)

Construct-targeted files (7):

File	Construct exercised
app/models/ability.rb	class << self inside a class (skipped by class_length)
app/models/concerns/issuable.rb	class << self inside a module (measured as singleton_class) and included do DSL
app/models/release_highlight.rb	QueryResult = Struct.new(...) do (constant assignment inside a class)
app/models/concerns/ignorable_columns.rb	ColumnIgnore = Struct.new(...) do (constant assignment inside a module)
app/models/concerns/from_set_operator.rb	define_method with a dynamic name (skipped) and a literal name (measured) in one file
app/models/concerns/ci/partitionable.rb	define_method(:sym) parenthesized literal form
spec/models/ability_spec.rb	RSpec block DSL

All 20 files run through code_keeper at this revision without crashing: cyclomatic_complexity count 1013, class_length count 44, abc_metric count 1013.

Checks

1. Invariants (metric-agnostic, CI):
   • Output conforms to a committed report schema (JSON Schema or an equivalent shape assertion in specs), so "valid report" is a checkable statement.
   • summary (count, max, top_hotspots) equals a recomputation from measurements.
   • abc_metric and cyclomatic_complexity have the same measurement count, because both enumerate the same method scopes.
   • Line ranges fall inside the file.
   • Output is identical for number_of_threads 1 and N.
   • Decide and encode whether metrics with no measurements appear in summary with zero counts: today they are absent entirely (for example class_length on config/routes.rb).
2. RuboCop differential (RuboCop-backed metrics, CI):
   • Run the RuboCop Metrics cops through the cop pipeline with a forced in-memory configuration:
     • Metrics/AbcSize: Max: 0, CountRepeatedAttributes: true, AllowedMethods: [], AllowedPatterns: []
     • Metrics/CyclomaticComplexity: Max: 0, AllowedMethods: [], AllowedPatterns: []
     • Metrics/ClassLength: Max: 0, CountComments: false, CountAsOne: []
   • Run with disable comments ignored and without loading this repository's .rubocop.yml.
   • Pin TargetRubyVersion explicitly: without it RuboCop falls back to the oldest supported parser and silently reports zero metric offenses for files using newer syntax, which shows up as unexplained CodeKeeper-only measurements instead of an error.
   • Capture raw metric values through a test-only cop wrapper: RuboCop::Cop::Offense does not expose the computed value and offense messages contain rounded values, so message parsing is not sufficient for exact comparison.
   • Match offenses and measurements bidirectionally and uniquely by path and start line; values must be equal, and unmatched entries on either side fail the check unless baselined.
   • Compare every RuboCop-reported scope that CodeKeeper supports, including literal define_method and constant-assignment class definitions.
   • Record any accepted difference in a baseline with metric, path, line, and reason; the check fails on new or stale baseline entries.
   • Known accepted difference to seed the baseline: singleton class scopes. RuboCop Metrics/ClassLength skips class << self inside class bodies and measures it elsewhere, including dynamic-self positions such as included do blocks. CodeKeeper measures singleton classes only where self statically is the enclosing constant. Compare singleton class scopes through baseline rules rather than strict matching.
3. Golden snapshots:
   • One pretty-printed JSON report per corpus file, committed to the repository.
   • The runner invokes CodeKeeper with repository-relative paths so snapshots do not depend on the execution environment.
   • Value drift appears as a reviewable git diff when regenerating.
   • Regeneration is a documented out-of-band task, not CI.

Ordering note: capture goldens only after the 1.0 scope naming is final (singleton scope names are being revised in #39).

PR plan

1. Corpus vendoring, per-file runner, crash check, invariants (CI).
2. RuboCop differential with an accepted-differences baseline (CI).
3. Golden snapshots and the regeneration task.

Metamorphic and property-based checks are deferred until the first metric without a RuboCop counterpart (see #40).

Non-goals

• Running full-repository sweeps in CI.
• Vendoring ee/ files.
• Property-based generation of Ruby code for the current metrics.

Acceptance Criteria

• CI verifies schema conformance, the invariants, and the RuboCop differential over the vendored corpus without network access.
• Accepted differences are recorded in a baseline that fails on new or stale entries.
• The corpus directory records license attribution, source revision, and file list, and contains no files from ee/, jh/, or doc/.
• A golden report exists for every corpus file and the regeneration procedure is documented.

Refs #40.