feat: update payloads generation for recent mainnet forks

CLAUDE.md

@@ -24,14 +24,18 @@ uv pip install -e .
 ### Running Tests
 
 ```bash
-# Run all tests
-pytest
+# Run all unit tests (integration tests are deselected by default)
+uv run pytest
 
-# Run specific test file
-pytest tests/api/test_something.py
+# Run a specific test file
+uv run pytest tests/payloads/test_execution_requests.py
 
 # Run with verbose output
-pytest -v
+uv run pytest -v
+
+# Run the integration tests that hit live EL/Beacon endpoints
+# (override endpoints with EXPB_TEST_RPC_URL / EXPB_TEST_BEACON_URL)
+uv run pytest -m integration
 ```
 
 ## Core Architecture
@@ -40,7 +44,7 @@ pytest -v
 
 The application uses [Typer](https://typer.tiangolo.com/) for the CLI interface. The main entry point is in [src/expb/\_\_init\_\_.py](src/expb/__init__.py), which aggregates sub-commands from:
 
-- `generate_payloads` - Extracts historical payloads from an Ethereum RPC endpoint
+- `generate_payloads` - Reconstructs the exact `engine_newPayload` / `engine_forkchoiceUpdated` requests for a block range by sourcing them from a Consensus client's Beacon API (execution RPC is used only for block→slot mapping)
 - `execute_scenario` - Runs a single benchmark scenario
 - `execute_scenarios` - Runs multiple benchmark scenarios (optionally in a loop)
 - `compress_payloads` - Compresses multiple smaller payloads into larger blocks
@@ -142,18 +146,24 @@ Per-payload metrics can be enabled with `--per-payload-metrics` flag, generating
 
 ### Generate Payloads
 
-Extract historical Ethereum payloads from an RPC endpoint:
+Reconstruct the Engine API requests for a block range from a Consensus (Beacon) API and an
+execution RPC. Both endpoints must serve the requested range (archive node for older ranges):
 
 ```bash
 expb generate-payloads \
-  --rpc-url https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY \
+  --rpc-url http://localhost:8545 \
+  --beacon-url http://localhost:5052 \
   --start-block 19000000 \
   --end-block 19001000 \
   --output-dir ./payloads \
-  --threads 10 \
-  --workers 30
+  --threads 10
 ```
 
+Payloads are sourced from the beacon block, so the generated `ExecutionPayload`,
+`executionRequests` (EIP-7685), blob versioned hashes and parent beacon block root exactly match
+what the consensus client sends the execution client, across all forks through Osaka. The
+`forkchoiceUpdated` requests set `safeBlockHash` and `finalizedBlockHash` to the parent block hash.
+
 ### Run Single Scenario
 
 Execute a benchmark scenario defined in a config file:

README.md

@@ -19,6 +19,28 @@ uv tool install --from git+https://github.com/NethermindEth/execution-payloads-b
 2. Edit the configuration file.
 3. Execute one or multiple scenarios.
 
+### Generating Payloads
+
+Benchmarks replay the `engine_newPayload` / `engine_forkchoiceUpdated` requests that a consensus
+client sends to an execution client. Generate them for a block range from a Consensus (Beacon) API
+and an execution RPC:
+
+```bash
+expb generate-payloads \
+  --rpc-url http://localhost:8545 \
+  --beacon-url http://localhost:5052 \
+  --network mainnet \
+  --start-block 21000000 \
+  --end-block 21001000 \
+  --output-dir ./payloads
+```
+
+The requests are sourced from the beacon block, so the generated payloads, execution requests
+(EIP-7685), blob versioned hashes and parent beacon block root match exactly what the consensus
+client sends the execution client — across all forks through Osaka. Both endpoints must serve the
+requested range (an archive node is required for older ranges). This produces `payloads.jsonl` and
+`fcus.jsonl`, consumed by scenario execution and by `send-payloads`.
+
 ### Scenarios Execution
 
 #### Single Scenario

docs/USAGE.md

@@ -22,7 +22,16 @@ expb [OPTIONS] COMMAND [ARGS]...
 
 ## `expb generate-payloads`
 
-Generate execution payloads for a given block range.
+Generate `engine_newPayload` / `engine_forkchoiceUpdated` requests for a given block range.
+
+The requests are sourced from the Consensus client's Beacon API so the generated
+`ExecutionPayload`, `executionRequests` (EIP-7685), blob versioned hashes and parent beacon
+block root exactly match what a consensus client sends the execution client — including
+post-Deneb forks (Prague/Electra, Osaka). The execution RPC is used only to map block numbers
+to beacon slots and to resolve the latest block.
+
+Both endpoints must serve the requested block range (an archive node is required for older
+ranges).
 
 **Usage**:
 
@@ -32,17 +41,29 @@ expb generate-payloads [OPTIONS]
 
 **Options**:
 
-* `--rpc-url TEXT`: Ethereum RPC URL  [required]
+* `--rpc-url TEXT`: Ethereum execution RPC URL  [required]
+* `--beacon-url TEXT`: Ethereum consensus (Beacon) API URL  [required]
 * `--network [mainnet]`: Network  [default: mainnet]
 * `--start-block INTEGER`: Start block  [default: 0]
 * `--end-block INTEGER`: End block
 * `--output-dir PATH`: Output directory  [default: payloads]
 * `--join-payloads / --no-join-payloads`: Join payloads and FCUs into a single file (payloads.jsonl and fcus.jsonl)  [default: join-payloads]
 * `--log-level TEXT`: Log level (e.g., DEBUG, INFO, WARNING)  [default: INFO]
 * `--threads INTEGER`: Number of threads for parallel processing  [default: 10]
-* `--workers INTEGER`: Number of workers per thread for parallel processing  [default: 30]
 * `--help`: Show this message and exit.
 
+**Example**:
+
+```bash
+expb generate-payloads \
+  --rpc-url http://localhost:8545 \
+  --beacon-url http://localhost:5052 \
+  --network mainnet \
+  --start-block 21000000 \
+  --end-block 21001000 \
+  --output-dir ./payloads
+```
+
 ## `expb execute-scenario`
 
 Execute payloads for a given execution client using Grafana K6.

example-expb.yaml

@@ -13,7 +13,7 @@ pull_images: true
 
 # Docker images to use (pin to specific versions for reproducible benchmarks)
 images:
-  k6: grafana/k6:1.6.1
+  k6: grafana/k6:2.1.0
   alloy: grafana/alloy:v1.13.2
   payload_server: python:3.12-slim