[Performance] Fail to recurrence e2e metrics on a 8-card H100 testbed for co-location across 4 kinds of workloads

[0xkayne]

Description of Problem

I'm developing and researching on llm serving simulation system and found that Frontier claims it can reach 6.4% e2e latency error under co-location and 2.6% e2e latency error under disaggregation.

So I try to recurrence the performance on a 8-card H100 testbed, and compares vLLM V1 and Frontier online runs. Every pair replays the same EngineCore arrival trace. Values below aggregate 3 repetitions. I put the experiment configuration and results as below.

I find that cannot reach the e2e performance as described in paper and I want to know if this is normal or I should change my parallel configuration and other configurations.

Configuration

Model/device: Qwen3-30B-A3B BF16 on 8 x H100 80GB
Parallel configurations: pp1_tp8_ep8, pp4_tp2_ep2
Runtime: vLLM V1, chunked prefill, eager mode, max 32 sequences
Load: fixed Poisson trace at 70% of each configuration/workload pilot capacity
Workloads: isl1024_osl1024, isl1024_osl256, isl256_osl1024, sharegpt
Primary paper metrics: P95 TTFT, P95 TPOT, throughput, workload E2E makespane

Accuracy By Workload

config	workload	metric	vLLM median	Frontier median	MAPE	signed error
pp1_tp8_ep8	isl1024_osl1024	Decode throughput (token/s)	409.352	454.630	11.37%	+11.37%
pp1_tp8_ep8	isl1024_osl1024	P95 request E2E (ms)	40727.332	24778.103	39.21%	-39.21%
pp1_tp8_ep8	isl1024_osl1024	P95 TPOT (ms)	39.736	24.168	38.80%	-38.80%
pp1_tp8_ep8	isl1024_osl1024	P95 TTFT (ms)	47.009	67.262	61.09%	-1.56%
pp1_tp8_ep8	isl1024_osl1024	Request throughput (req/s)	0.400	0.444	11.37%	+11.37%
pp1_tp8_ep8	isl1024_osl1024	Workload E2E makespan (ms)	160097.061	144152.540	10.20%	-10.20%
pp1_tp8_ep8	isl1024_osl256	Decode throughput (token/s)	439.684	491.022	11.77%	+11.77%
pp1_tp8_ep8	isl1024_osl256	P95 request E2E (ms)	10151.780	6505.950	35.99%	-35.99%
pp1_tp8_ep8	isl1024_osl256	P95 TPOT (ms)	39.604	25.243	36.32%	-36.32%
pp1_tp8_ep8	isl1024_osl256	P95 TTFT (ms)	45.968	67.568	36.71%	+36.71%
pp1_tp8_ep8	isl1024_osl256	Request throughput (req/s)	1.718	1.918	11.77%	+11.77%
pp1_tp8_ep8	isl1024_osl256	Workload E2E makespan (ms)	37263.121	33367.113	10.52%	-10.52%
pp1_tp8_ep8	isl256_osl1024	Decode throughput (token/s)	461.302	518.652	12.86%	+12.86%
pp1_tp8_ep8	isl256_osl1024	P95 request E2E (ms)	40554.501	24489.580	39.69%	-39.69%
pp1_tp8_ep8	isl256_osl1024	P95 TPOT (ms)	39.598	23.895	39.74%	-39.74%
pp1_tp8_ep8	isl256_osl1024	P95 TTFT (ms)	46.230	58.018	22.56%	+22.56%
pp1_tp8_ep8	isl256_osl1024	Request throughput (req/s)	0.450	0.506	12.86%	+12.86%
pp1_tp8_ep8	isl256_osl1024	Workload E2E makespan (ms)	142067.594	126358.296	11.40%	-11.40%
pp1_tp8_ep8	sharegpt	Decode throughput (token/s)	130.804	143.647	9.15%	+9.15%
pp1_tp8_ep8	sharegpt	P95 request E2E (ms)	29856.897	17578.632	40.77%	-40.77%
pp1_tp8_ep8	sharegpt	P95 TPOT (ms)	40.559	23.687	41.97%	-41.97%
pp1_tp8_ep8	sharegpt	P95 TTFT (ms)	52.197	59.268	13.90%	+13.90%
pp1_tp8_ep8	sharegpt	Request throughput (req/s)	0.537	0.590	9.15%	+9.15%
pp1_tp8_ep8	sharegpt	Workload E2E makespan (ms)	119155.348	108501.942	8.38%	-8.38%
pp4_tp2_ep2	isl1024_osl1024	Decode throughput (token/s)	336.120	380.663	14.32%	+14.32%
pp4_tp2_ep2	isl1024_osl1024	P95 request E2E (ms)	50343.553	26359.789	47.81%	-47.81%
pp4_tp2_ep2	isl1024_osl1024	P95 TPOT (ms)	49.158	25.713	47.21%	-47.21%
pp4_tp2_ep2	isl1024_osl1024	P95 TTFT (ms)	68.314	60.837	38.80%	-34.21%
pp4_tp2_ep2	isl1024_osl1024	Request throughput (req/s)	0.328	0.372	14.32%	+14.32%
pp4_tp2_ep2	isl1024_osl1024	Workload E2E makespan (ms)	194978.198	172162.570	12.51%	-12.51%
pp4_tp2_ep2	isl1024_osl256	Decode throughput (token/s)	375.778	429.677	14.41%	+14.41%
pp4_tp2_ep2	isl1024_osl256	P95 request E2E (ms)	12293.382	7169.757	41.57%	-41.57%
pp4_tp2_ep2	isl1024_osl256	P95 TPOT (ms)	48.006	27.896	41.83%	-41.83%
pp4_tp2_ep2	isl1024_osl256	P95 TTFT (ms)	57.199	64.872	21.65%	+21.65%
pp4_tp2_ep2	isl1024_osl256	Request throughput (req/s)	1.468	1.678	14.41%	+14.41%
pp4_tp2_ep2	isl1024_osl256	Workload E2E makespan (ms)	43600.170	38130.951	12.59%	-12.59%
pp4_tp2_ep2	isl256_osl1024	Decode throughput (token/s)	382.988	439.335	14.61%	+14.61%
pp4_tp2_ep2	isl256_osl1024	P95 request E2E (ms)	48607.061	26581.546	45.27%	-45.27%
pp4_tp2_ep2	isl256_osl1024	P95 TPOT (ms)	47.463	25.942	45.30%	-45.30%
pp4_tp2_ep2	isl256_osl1024	P95 TTFT (ms)	55.873	46.107	24.27%	-24.27%
pp4_tp2_ep2	isl256_osl1024	Request throughput (req/s)	0.374	0.429	14.61%	+14.61%
pp4_tp2_ep2	isl256_osl1024	Workload E2E makespan (ms)	171117.456	149171.001	12.75%	-12.75%
pp4_tp2_ep2	sharegpt	Decode throughput (token/s)	118.131	132.395	11.22%	+11.22%
pp4_tp2_ep2	sharegpt	P95 request E2E (ms)	33798.434	17724.543	47.49%	-47.49%
pp4_tp2_ep2	sharegpt	P95 TPOT (ms)	46.449	24.149	47.96%	-47.96%
pp4_tp2_ep2	sharegpt	P95 TTFT (ms)	77.244	49.137	37.57%	-37.57%
pp4_tp2_ep2	sharegpt	Request throughput (req/s)	0.485	0.544	11.22%	+11.22%
pp4_tp2_ep2	sharegpt	Workload E2E makespan (ms)	131938.446	117723.239	10.09%	-10.09%

Overall Error

metric	paired runs	MAPE	median APE	max APE	mean signed error
Decode throughput (token/s)	24	12.46%	12.98%	16.52%	+12.46%
P95 request E2E (ms)	24	42.23%	41.25%	48.25%	-42.23%
P95 TPOT (ms)	24	42.39%	41.95%	48.31%	-42.39%
P95 TTFT (ms)	24	32.07%	25.72%	97.52%	-0.35%
Request throughput (req/s)	24	12.46%	12.98%	16.52%	+12.46%
Workload E2E makespan (ms)	24	11.05%	11.49%	14.18%	-11.05%

[fwyc0573]

Thanks for the question! Accuracy heavily depends on two things:

• Profiling data: Whether you collected enough sample data during profiling (which is used to train the predictors for end-to-end simulation).
• vLLM setup: The vLLM version of your benchmark, and whether you added extra controls on its MoE routing to align with Frontier's MoE load balancing distribution.

To help clear this up, could you share a bit more context? Specifically:

• Your vLLM version, attention backend (e.g., FlashInfer and its version), CUDA, and torch versions.
• Crucially, the exact run commands you used for both Frontier and vLLM.

I don't have access to H100 hardware right now, but I can replay your setup on an H800 and get back to you with more details. By the way, we'll be releasing more accuracy calibration skills in upcoming releases so users can easily calibrate simulation accuracy across different hardware and operator backends themselves. If you're interested, we'd love to have you collaborate on these calibration efforts!

[0xkayne]

Thanks for the detailed guidance! Here’s the context you asked for, and a follow-up question.

Environment & versions

• vLLM version: 0.10.2 (the version used in the paper)
• Attention backend: FlashInfer 0.3.1.post1
• PyTorch: 2.8.0+cu128, CUDA build 12.8
• NCCL: 2.27.3
• CUDA driver: 590.48.1
• GPU: H100

vLLM run commands (executed from the repo root)

unset CUDA_VISIBLE_DEVICES
export VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER
export VLLM_ALL2ALL_BACKEND=naive PYTHONPATH="$PWD" PYTHONNOUSERSITE=1
export TOKENIZERS_PARALLELISM=false TMPDIR=/share/wangziping/tmp
export PATH="$PWD/.venv/bin:$PATH"

PY="$PWD/.venv/bin/python"
RUN=tests/comparison/h100_online_accuracy/run_vllm_online.py
ROOT=/share/outputs/frontier-accuracy/online-paper-matrix-20260722
MODEL=/data_gpu/modelzoo/weights/llm/qwen/qwen3/Qwen3-30B-A3B/safetensor_weights

"$PY" "$RUN" --model-path "$MODEL" --workload-manifest "$ROOT/workload_manifest.json" \
  --output-dir "$ROOT/pp1_tp8_ep8/vllm" --config-name pp1_tp8_ep8 \
  --tensor-parallel-size 8 --pipeline-parallel-size 1 \
  --repetitions 3 --warmup-requests 4 --pilot-requests 32 \
  --num-formal-requests 64 --target-load 0.70 \
  >"$ROOT/logs/vllm_pp1_tp8_ep8.log" 2>&1

"$PY" "$RUN" --model-path "$MODEL" --workload-manifest "$ROOT/workload_manifest.json" \
  --output-dir "$ROOT/pp4_tp2_ep2/vllm" --config-name pp4_tp2_ep2 \
  --tensor-parallel-size 2 --pipeline-parallel-size 4 \
  --repetitions 3 --warmup-requests 4 --pilot-requests 32 \
  --num-formal-requests 64 --target-load 0.70 \
  >"$ROOT/logs/vllm_pp4_tp2_ep2.log" 2>&1

Frontier simulation commands

F=tests/comparison/h100_online_accuracy/run_frontier_online.sh
ROOT=/share/outputs/frontier-accuracy/online-paper-matrix-20260722
P2=/share/wangziping/outputs/frontier-accuracy/gpu0-3-full-matrix-20260722/profiles-corrected

PATH="$PWD/.venv/bin:$PATH" "$F" --base "$ROOT/pp1_tp8_ep8" \
  --trace-dir "$ROOT/pp1_tp8_ep8/vllm/traces_remaining" \
  --profile-root "$ROOT/profiles-tp8/unified" --config-name pp1_tp8_ep8 \
  --tensor-parallel-size 8 --pipeline-parallel-size 1 --log-level error --jobs 8

PATH="$PWD/.venv/bin:$PATH" "$F" --base "$ROOT/pp1_tp8_ep8" \
  --trace-dir "$ROOT/pp1_tp8_ep8/vllm/trace_retry_rep00" \
  --profile-root "$ROOT/profiles-tp8/unified" --config-name pp1_tp8_ep8 \
  --tensor-parallel-size 8 --pipeline-parallel-size 1 --log-level error --jobs 1

PATH="$PWD/.venv/bin:$PATH" "$F" --base "$ROOT/pp4_tp2_ep2" \
  --trace-dir "$ROOT/pp4_tp2_ep2/vllm/trace_warmup" --profile-root "$P2" \
  --config-name pp4_tp2_ep2 --tensor-parallel-size 2 \
  --pipeline-parallel-size 4 --log-level error --jobs 1

PATH="$PWD/.venv/bin:$PATH" "$F" --base "$ROOT/pp4_tp2_ep2" \
  --trace-dir "$ROOT/pp4_tp2_ep2/vllm/traces_remaining" --profile-root "$P2" \
  --config-name pp4_tp2_ep2 --tensor-parallel-size 2 \
  --pipeline-parallel-size 4 --log-level error --jobs 8

Profiling methodology (PP=1, TP=8, EP=8 as an example)
I double-checked: the profiling was not done by a single unified script. Instead, we profiled Linear, Attention, and MoE operators separately. The common characteristics across all profiling runs are:

• Hardware: H100 GPU

• profile_method=cuda_event

• --disable_ray (using multiprocessing)

• Each profiling process only occupies a single GPU

• Outputs are CSV files together with YAML configurations; Frontier reads these CSVs for simulation

• The profiling measures real vLLM/FlashInfer kernel execution time on the hardware
  For the PP=1, TP=8, EP=8 configuration we used in the accuracy comparison, the profiling commands were:

  Linear operator (TP=8)

python -m frontier.profiling.linear_op.main \
    --disable_ray \
    --models qwen3-a3b-30b-moe \
    --device h100 \
    --num_gpus 1 \
    --max_tokens 4096 \
    --num_tensor_parallel_workers 8 \
    --profile_method cuda_event \
    --output_dir .../profiles-tp8/base \
    --yes

Attention operator (TP=8, FlashInfer)

python -m frontier.profiling.attention.main \
    --disable_ray \
    --models qwen3-a3b-30b-moe \
    --device h100 \
    --num_gpus 1 \
    --max_seq_len 2304 \
    --num_tensor_parallel_workers 8 \
    --max_pipeline_parallel_size 1 \
    --attention_backend FLASHINFER \
    --block_size 16 \
    --min_batch_size 1 \
    --max_batch_size 128 \
    --enable_mixed_prefill \
    --enable_true_mixed \
    --enable_chunked_prefill_grid_search \
    --fixed_chunked_prefill_size 512 \
    --profile_method cuda_event \
    --output_dir .../profiles-tp8/base \
    --yes

MoE operator (EP=8)
We collected two contexts separately – prefill_hot and standalone_legacy. Here is the prefill_hot command:

python -m frontier.profiling.moe.main \
    --disable_ray \
    --models qwen3-a3b-30b-moe \
    --device h100 \
    --num_gpus 1 \
    --max_tokens 2048 \
    --num_tensor_parallel_workers 1 \
    --expert_parallel_sizes 8 \
    --profile_method cuda_event \
    --routing_runtime_path standard_fused_topk \
    --gating_runtime_context prefill_hot \
    --disable_load_imbalance \
    --use_vllm_kernel \
    --output_dir .../profiles-tp8/moe-prefill \
    --yes

MoE routing alignment
I did not add any extra routing controls in vLLM to match Frontier’s MoE load balancing distribution. vLLM was set with only enable_expert_parallel=True, and I did not enable enable_eplb, redundant experts, forced uniform routing, or any modification to router logits. It uses Qwen3’s native learned router (128 experts, top-8 per token).
On the Frontier side, I used moe_routing_mode="simulation" with seed=42, generating fixed random expert ratios layer-by-layer, and router_load_balancing_type="None".
How much do you think this routing mismatch could affect accuracy in practice? I’d really appreciate your insight here.

Calibration collaboration
Thanks a lot for the invitation! Unfortunately I need to focus on building our simulation infrastructure and learning more about the underlying algorithms in the near term, so I won’t be able to actively contribute to the calibration effort right now. I’m definitely looking forward to staying in touch and collaborating down the road once things settle on my side.

Please let me know if you need any more details, and thanks again for your help!

[fwyc0573]

Thanks for the detailed reply. Yeah, based on our calibration, aligning MoE routing makes a clear difference to end‑to‑end metric accuracy, since it directly impacts the FFN side—where communication and compute take up a big chunk of the latency. Generally, as long as op sampling (i.e., profiling) is complete enough, most test cases show decent error control; the remaining errors mainly come from vLLM's CPU overhead, as discussed in the paper's discussion section.

Also, quick update: in our recent development, we found some potential bugs in the current profiling module. These could cause the measurement scope to drift from what we set, and the predictor might not extrapolate correctly (see PR #4). That could seriously hurt the in‑flight batch latency prediction, so we're working on fixes. More collaboration and contributions to the repo are always welcome! :)