feat(metrics): add Rust no_std encoder for V3

.github/workflows/test.yml

@@ -52,5 +52,69 @@ jobs:
     - run: inv codegen.all
       env:
         GOPATH: "/home/runner/work/agent-payload/agent-payload/go"
+    - name: Read Rust version from rust-toolchain.toml
+      id: rust-version
+      run: echo "version=$(grep -Po '^channel = "\K[^"]+' rust-toolchain.toml)" >> $GITHUB_OUTPUT
+    - name: Install ${{ steps.rust-version.outputs.version }} toolchain
+      run: |
+        rustup set profile minimal
+        rustup install ${{ steps.rust-version.outputs.version }}
+        rustup default ${{ steps.rust-version.outputs.version }}
+    # `inv codegen.all` (above) already ran `inv codegen.install-protoc`, so `metrics/dd-metrics-v3`'s
+    # build script (which reuses that same pinned protoc, see tasks/codegen.py's `protoc_version`)
+    # finds it without a separate install step.
+    - name: Regenerate metrics/dd-metrics-v3's protobuf test bindings
+      run: cargo build --features generate-protobuf
+      working-directory: "/home/runner/work/agent-payload/agent-payload/go/src/github.com/DataDog/agent-payload/metrics/dd-metrics-v3"
     - name: Check for diffs
       run: git diff --exit-code
+
+  test-rust:
+    name: "cargo test / fmt / clippy (dd-metrics-v3)"
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: metrics/dd-metrics-v3
+    steps:
+    - uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
+    - name: Read Rust version from rust-toolchain.toml
+      id: rust-version
+      run: echo "version=$(grep -Po '^channel = "\K[^"]+' ../../rust-toolchain.toml)" >> $GITHUB_OUTPUT
+    - name: Install ${{ steps.rust-version.outputs.version }} toolchain
+      run: |
+        rustup set profile minimal
+        rustup install ${{ steps.rust-version.outputs.version }}
+        rustup default ${{ steps.rust-version.outputs.version }}
+        rustup component add rustfmt clippy
+    - name: cargo fmt --check
+      run: cargo fmt -- --check
+    - name: cargo test
+      run: cargo test --all-targets
+    - name: cargo clippy
+      run: cargo clippy --all-targets -- -D warnings
+
+  test-rust-no-std-check:
+    # `#![no_std]` alone only stops this crate's own code from referencing `std`; a dependency
+    # that quietly requires std would still link fine on a normal host target. To verify this,
+    # we build for a target with no OS where `std` is not available at all.
+    name: "dd-metrics-v3 builds for a target with no OS and no std"
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: metrics/dd-metrics-v3
+    steps:
+    - uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
+    - name: Read Rust version from rust-toolchain.toml
+      id: rust-version
+      run: echo "version=$(grep -Po '^channel = "\K[^"]+' ../../rust-toolchain.toml)" >> $GITHUB_OUTPUT
+    - name: Install ${{ steps.rust-version.outputs.version }} toolchain
+      run: |
+        rustup set profile minimal
+        rustup install ${{ steps.rust-version.outputs.version }}
+        rustup default ${{ steps.rust-version.outputs.version }}
+        rustup target add x86_64-unknown-none
+        rustup component add clippy
+    - name: Build for no_std target
+      run: cargo build --target x86_64-unknown-none
+    - name: Clippy for no_std target
+      run: cargo clippy --target x86_64-unknown-none -- -D warnings

.gitignore

@@ -19,3 +19,6 @@ toolchains/
 
 # python virtualenv for invoke tasks
 venv/
+
+# rust build artifacts
+target/

README.md

@@ -4,7 +4,7 @@ Payload format description for communication between the Agent and the Datadog b
 
 This repository includes the protocol-buffer IDL used by the agent6 and agent7 to communicate with the Datadog backend.
 Those payloads are only supported by the V2 API endpoints.
-The generated Go, and Java implementations are checked into this repository and can be used directly. Other consumers may copy the `.proto` files into their repository and generate their own bindings.
+The generated Go, Java, and Rust implementations are checked into this repository and can be used directly. Other consumers may copy the `.proto` files into their repository and generate their own bindings.
 
 # Use
 
@@ -25,6 +25,7 @@ You will need
  * Python (3+, CI builds with 3.12)
  * Go (at least the version in `go.mod`)
  * A checkout of this repository within a GOPATH (so, at `$GOPATH/src/github.com/DataDog/agent-payload`)
+ * Rust, only if you're working on [`metrics/dd-metrics-v3/`](./metrics/dd-metrics-v3) — install the toolchain version pinned in `rust-toolchain.toml`.
 
 # Payloads
 
@@ -41,6 +42,14 @@ The metrics payload is defined in [`proto/metrics/agent_payload.proto`](./proto/
 The following implementations are available:
  * Go (gogofast): [github.com/DataDog/agent-payload/gogen](https://pkg.go.dev/github.com/DataDog/agent-payload/gogen)
 
+### Metrics V3
+
+The V3 metrics payload is defined in [`proto/metrics/intake_v3.proto`](./proto/metrics/intake_v3.proto).
+It uses a columnar layout with dictionary-based string deduplication instead of one message per
+time series. The following implementations are available:
+ * Go (protoc-gen-go): [github.com/DataDog/agent-payload/v5/metrics/intake_v3](https://pkg.go.dev/github.com/DataDog/agent-payload/v5/metrics/intake_v3).
+ * Rust: [`dd-metrics-v3`](./metrics/dd-metrics-v3), a hand-rolled `no_std` encoder.
+
 ## Process
 
 The process payload is defined in [`proto/process/agent.proto`](./proto/process/agent.proto).
@@ -66,6 +75,7 @@ After updating the IDL you must:
 
 - Regenerate the code: `inv codegen.all`, invoke will use gimme to run the task command with the current defined go version
 - If you have indentation/newlines changes, run `inv codegen.all` with the same Go version as defined in `go.mod`
+- If you changed `proto/metrics/intake_v3.proto`, also regenerate `metrics/dd-metrics-v3/tests/pb/mod.rs` (`cargo build --features generate-protobuf` from within `metrics/dd-metrics-v3/`) and update the hand-rolled encoder in `metrics/dd-metrics-v3/src/` to match, since it doesn't use a Protocol Buffers library
 - Create a new tag with the updated version of the payload
 
 # Publishing Changes

REVIEWING.md

@@ -8,6 +8,8 @@ This is a _work in progress_ and additions to this list are welcome.
 
 If any of the `.proto` files are changed, then the corresponding Go code should be regenerated in the same PR (`GOPATH=$(go env GOPATH) inv codegen.all`).
 
+If `proto/metrics/intake_v3.proto` changes, also regenerate `metrics/dd-metrics-v3/tests/pb/mod.rs` (`cargo build --features generate-protobuf` from within `metrics/dd-metrics-v3/`) and update `metrics/dd-metrics-v3/src/constants.rs` and the hand-rolled encoder in `metrics/dd-metrics-v3/src/writer.rs` to match — they don't use a Protocol Buffers library, so they won't fail to compile just because the wire format changed underneath them.
+
 Tests should be run locally before making a PR, and should pass in CI before a PR is merged.
 
 ## Implications for Security