Skip to content

Add LCS load generator implementation - #1

Merged
Balatripura587 merged 1 commit into
lightspeed-core:mainfrom
Balatripura587:add-load-generator
Aug 25, 2026
Merged

Add LCS load generator implementation#1
Balatripura587 merged 1 commit into
lightspeed-core:mainfrom
Balatripura587:add-load-generator

Conversation

@Balatripura587

@Balatripura587 Balatripura587 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Description

HTTP load generator for LightSpeed Core Service (LCS) using Locust. Runs query (POST /v1/query) and streaming (POST /v1/streaming_query) endpoints sequentially in a single invocation, collecting per-endpoint performance metrics.

Run modes:

  • CLI — run directly from source with ./lcs-load-generator run --host ... --users ... --duration ...
  • Container — all configuration via environment variables, ENTRYPOINT runs the test automatically
  • OpenShift Job — single manifest (config/lcs-load-generator.yaml) with RBAC + Job, deploy with oc apply -f

Metrics collected per endpoint:

  • Latency percentiles (p50, p95, p99), min/max/avg
  • Throughput (req/s), total request count
  • HTTP status code distribution
  • bytesIn/bytesOut
  • TTFT (time to first token) and stream duration percentiles (streaming only)

Prometheus metrics scraping

Uses kube-burner (pinned to v1.10.4) to query Prometheus and index infrastructure metrics. Two metric profiles:

  • metrics-report.yaml — instant aggregates (avg/max CPU, memory, throttling)
  • metrics-timeseries.yaml — time-series data (per-pod CPU/memory over time, LCS application metrics, node network, histograms)

All metric profiles use {{.LCS_NAMESPACE}} template variable — works with any namespace, no hardcoding required.

Note: LCS does not have an operator, so Prometheus monitoring requires manual setup depending on where LCS is deployed.

  • If deployed in an openshift-* namespace, platform Prometheus (prometheus-k8s) scrapes metrics directly
  • If deployed in a user namespace, user-workload monitoring must be enabled and metrics are queried via Thanos Querier (PROMETHEUS_BACKEND=thanos)

Result indexing:

  • When ES_SERVER is configured, results are indexed to Elasticsearch
  • Otherwise, results are written as JSON files under RESULTS_DIR (defaults to /results in the pod, /tmp locally) and printed to stdout (visible via oc logs on OpenShift)

Additional features:

  • Multi-process mode (--processes) at high concurrency
  • Cluster metadata collection (OCP version, platform, worker node count) for jobSummary document
  • -D (debug) and -W (quiet) logging flags for troubleshooting

Testing

  1. Tested in CLI , container and deployed as OpenShift job:
image image

@Balatripura587
Balatripura587 merged commit eb57516 into lightspeed-core:main Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant