test : create large ca-bundle configmap in che installation namespace before running load tests

Signed-off-by: Rohan Kumar <[email protected]>

Author: rohanKanojia

test/load/README.md

@@ -0,0 +1,119 @@
+# Load Testing for DevWorkspace Operator
+
+This directory contains load testing tools for the DevWorkspace Operator using k6. The tests create multiple DevWorkspaces concurrently to measure the operator's performance under load.
+
+## Prerequisites
+
+- `kubectl` (version >= 1.24.0)
+- `curl` (version >= 7.0.0)
+- `k6` (version >= 1.1.0) - Required when using `--mode binary`
+- Access to a Kubernetes cluster with DevWorkspace Operator installed
+- Proper RBAC permissions to create DevWorkspaces, ConfigMaps, Secrets, and Namespaces
+
+## Running Load Tests
+
+The load tests can be run using the `make test_load` target with various arguments. The tests support two modes:
+- **binary mode**: Runs k6 locally (default)
+- **operator mode**: Runs k6 using the k6-operator in the cluster
+
+### Running with Eclipse Che
+
+When running with Eclipse Che, the script automatically provisions additional ConfigMaps for certificates that are required for Che workspaces to function properly.
+
+```bash
+make test_load ARGS=" \
+  --mode binary \
+  --run-with-eclipse-che true \
+  --max-vus ${MAX_VUS} \
+  --create-automount-resources true \
+  --max-devworkspaces ${MAX_DEVWORKSPACES} \
+  --devworkspace-ready-timeout-seconds 3600 \
+  --delete-devworkspace-after-ready false \
+  --separate-namespaces false \
+  --test-duration-minutes 40"
+```
+
+**Note**: When `--run-with-eclipse-che true` is set, the script will:
+- Provision a workspace namespace compatible with Eclipse Che
+- Create additional certificate ConfigMaps required by Che
+
+### Running without Eclipse Che
+
+When running without Eclipse Che, the standard namespace setup is used without additional certificate ConfigMaps.
+
+```bash
+make test_load ARGS=" \
+  --mode binary \
+  --max-vus ${MAX_VUS} \
+  --create-automount-resources true \
+  --max-devworkspaces ${MAX_DEVWORKSPACES} \
+  --devworkspace-ready-timeout-seconds 3600 \
+  --delete-devworkspace-after-ready false \
+  --separate-namespaces false \
+  --test-duration-minutes 40"
+```
+
+## Available Parameters
+
+| Parameter | Description | Default | Example |
+|-----------|-------------|---------|---------|
+| `--mode` | Execution mode: `binary` or `operator` | `binary` | `--mode binary` |
+| `--max-vus` | Maximum number of virtual users (concurrent DevWorkspace creations) | `100` | `--max-vus 50` |
+| `--max-devworkspaces` | Maximum number of DevWorkspaces to create (-1 for unlimited) | `-1` | `--max-devworkspaces 200` |
+| `--separate-namespaces` | Create each DevWorkspace in its own namespace | `false` | `--separate-namespaces true` |
+| `--delete-devworkspace-after-ready` | Delete DevWorkspace once it becomes Ready | `true` | `--delete-devworkspace-after-ready false` |
+| `--devworkspace-ready-timeout-seconds` | Timeout in seconds for workspace to become ready | `1200` | `--devworkspace-ready-timeout-seconds 3600` |
+| `--devworkspace-link` | URL to external DevWorkspace JSON to use instead of default | (empty) | `--devworkspace-link https://...` |
+| `--create-automount-resources` | Create automount ConfigMap and Secret for testing | `false` | `--create-automount-resources true` |
+| `--dwo-namespace` | DevWorkspace Operator namespace | `openshift-operators` | `--dwo-namespace devworkspace-controller` |
+| `--logs-dir` | Directory for DevWorkspace and event logs | `logs` | `--logs-dir /tmp/test-logs` |
+| `--test-duration-minutes` | Duration in minutes for the load test | `25` | `--test-duration-minutes 40` |
+| `--run-with-eclipse-che` | Enable Eclipse Che integration (adds certificate ConfigMaps) | `false` | `--run-with-eclipse-che true` |
+| `--che-cluster-name` | Eclipse Che cluster name (when using Che) | `eclipse-che` | `--che-cluster-name my-che` |
+| `--che-namespace` | Eclipse Che namespace (when using Che) | `eclipse-che` | `--che-namespace my-che-ns` |
+
+## What the Tests Do
+
+1. **Setup**: Creates a test namespace, ServiceAccount, and RBAC resources
+2. **Eclipse Che Setup** (if enabled): Provisions Che-compatible namespace and certificate ConfigMaps
+3. **Load Generation**: Creates DevWorkspaces concurrently based on `--max-devworkspaces`
+4. **Monitoring**: 
+   - Watches DevWorkspace status until Ready
+   - Monitors operator CPU and memory usage
+   - Tracks etcd metrics
+   - Logs events and DevWorkspace state changes
+5. **Cleanup**: Removes all created resources and test namespace
+
+## Test Metrics
+
+The tests track the following metrics:
+- DevWorkspace creation duration
+- DevWorkspace ready duration
+- DevWorkspace deletion duration
+- Operator CPU and memory usage
+- etcd CPU and memory usage
+- Success/failure rates
+
+## Output
+
+- **Logs**: Stored in the `logs/` directory (or custom directory specified by `--logs-dir`)
+  - `{timestamp}_events.log`: Kubernetes events
+  - `{timestamp}_dw_watch.log`: DevWorkspace watch logs
+  - `dw_failure_report.csv`: Failed DevWorkspaces report
+- **HTML Report**: Generated when running in binary mode (outside cluster)
+- **Console Output**: Real-time test progress and summary
+
+## Troubleshooting
+
+- **Permission errors**: Ensure your kubeconfig has sufficient RBAC permissions
+- **Timeout errors**: Increase `--devworkspace-ready-timeout-seconds` for slower clusters
+- **Resource exhaustion**: Reduce `--max-vus` or `--max-devworkspaces` if cluster resources are limited
+- **k6 not found**: Install k6 from https://k6.io/docs/getting-started/installation/
+
+## Additional Notes
+
+- The tests use an opinionated minimal DevWorkspace by default, or you can provide a custom one via `--devworkspace-link`
+- When `--separate-namespaces true` is used, each DevWorkspace gets its own namespace
+- The `--delete-devworkspace-after-ready false` option is useful for testing sustained load scenarios
+- Certificate ConfigMaps are only created when `--run-with-eclipse-che true` is set
+

test/load/che-cert-bundle-utils.sh

@@ -0,0 +1,223 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+log_info()    { echo -e "ℹ️  $*" >&2; }
+log_success() { echo -e "✅ $*" >&2; }
+log_error()   { echo -e "❌ $*" >&2; }
+
+
+run_che_ca_bundle_e2e() {
+  local che_ns="$1"
+  local dw_ns="$2"
+  local dw_name="$3"
+  local cert_count="${4:-500}"
+  local bundle_file="${5:-custom-ca-certificates.pem}"
+
+  check_namespaces "${che_ns}" "${dw_ns}"
+  generate_dummy_certs "${cert_count}" "${bundle_file}"
+  create_che_ca_configmap "${che_ns}" "${bundle_file}"
+  patch_checluster_disable_pki_mount "${che_ns}"
+  restart_che "${che_ns}"
+  create_devworkspace "${dw_ns}" "${dw_name}"
+
+  local pod
+  pod=$(wait_for_workspace_pod "${dw_ns}" "${dw_name}")
+
+  verify_ca_bundle_in_workspace "${pod}" "${dw_ns}" "${cert_count}"
+  cleanup_resources "${dw_ns}" "${dw_name}"
+}
+
+check_namespaces() {
+  local che_ns="$1"
+  local dw_ns="$2"
+
+  log_info "Checking namespaces..."
+  kubectl get ns "${che_ns}" >/dev/null
+  kubectl get ns "${dw_ns}" >/dev/null
+}
+
+generate_dummy_certs() {
+  local cert_count="$1"
+  local bundle_file="$2"
+
+  log_info "Generating ${cert_count} dummy CA certificates..."
+  rm -f "${bundle_file}"
+
+  for i in $(seq 1 "${cert_count}"); do
+    openssl req -x509 -newkey rsa:2048 -nodes -days 1 \
+      -subj "/CN=dummy-ca-${i}" \
+      -keyout "dummy-ca-${i}.key" \
+      -out "dummy-ca-${i}.pem" \
+      >/dev/null 2>&1
+
+    cat "dummy-ca-${i}.pem" >> "${bundle_file}"
+  done
+
+  log_success "Created CA bundle: $(du -h "${bundle_file}" | cut -f1)"
+}
+
+create_che_ca_configmap() {
+  local che_ns="$1"
+  local bundle_file="$2"
+
+  log_info "Creating Che CA bundle ConfigMap..."
+
+  kubectl create configmap custom-ca-certificates \
+    --from-file=custom-ca-certificates.pem="${bundle_file}" \
+    -n "${che_ns}" \
+    --dry-run=client -o yaml \
+  | kubectl apply --server-side -f -
+
+  kubectl label configmap custom-ca-certificates \
+    app.kubernetes.io/component=ca-bundle \
+    app.kubernetes.io/part-of=che.eclipse.org \
+    -n "${che_ns}" \
+    --overwrite
+}
+
+patch_checluster_disable_pki_mount() {
+  local che_ns="$1"
+
+  log_info "Configuring CheCluster..."
+  local checluster
+  checluster=$(kubectl get checluster -n "${che_ns}" -o jsonpath='{.items[0].metadata.name}')
+
+  kubectl patch checluster "${checluster}" \
+    -n "${che_ns}" \
+    --type=merge \
+    -p '{
+      "spec": {
+        "devEnvironments": {
+          "trustedCerts": {
+            "disableWorkspaceCaBundleMount": true
+          }
+        }
+      }
+    }'
+}
+
+restart_che() {
+  local che_ns="$1"
+
+  log_info "Restarting Che..."
+  kubectl rollout status deploy/che -n "${che_ns}" --timeout=5m
+  kubectl wait pod -n "${che_ns}" -l app=che --for=condition=Ready --timeout=5m
+
+  log_success "Che restarted"
+}
+
+create_devworkspace() {
+  local dw_ns="$1"
+  local dw_name="$2"
+
+  log_info "Creating DevWorkspace '${dw_name}'..."
+  cat <<EOF | kubectl apply -n "${dw_ns}" -f -
+kind: DevWorkspace
+apiVersion: workspace.devfile.io/v1alpha2
+metadata:
+  name: ${dw_name}
+  annotations:
+    che.eclipse.org/che-editor: che-incubator/che-code/latest
+    che.eclipse.org/devfile: |
+      schemaVersion: 2.2.0
+      metadata:
+        generateName: ${dw_name}
+    che.eclipse.org/devfile-source: |
+      url:
+        location: https://github.com/che-samples/web-nodejs-sample.git
+      factory:
+        params: che-editor=che-incubator/che-code/latest
+spec:
+  started: true
+  template:
+    projects:
+      - name: web-nodejs-sample
+        git:
+          remotes:
+            origin: "https://github.com/che-samples/web-nodejs-sample.git"
+    components:
+      - name: dev
+        container:
+          image: quay.io/devfile/universal-developer-image:latest
+          memoryLimit: 512Mi
+          memoryRequest: 256Mi
+          cpuRequest: 1000m
+    commands:
+      - id: say-hello
+        exec:
+          component: dev
+          commandLine: echo "Hello from \$(pwd)"
+          workingDir: \${PROJECT_SOURCE}/app
+  contributions:
+    - name: che-code
+      uri: https://eclipse-che.github.io/che-plugin-registry/main/v3/plugins/che-incubator/che-code/latest/devfile.yaml
+      components:
+        - name: che-code-runtime-description
+          container:
+            env:
+              - name: CODE_HOST
+                value: 0.0.0.0
+EOF
+
+
+  kubectl wait devworkspace/"${dw_name}" \
+    -n "${dw_ns}" \
+    --for=condition=Ready \
+    --timeout=5m
+}
+
+wait_for_workspace_pod() {
+  local dw_ns="$1"
+  local dw_name="$2"
+  local pod_name
+
+  log_info "Waiting for workspace pod..."
+
+  kubectl wait pod \
+    -n "${dw_ns}" \
+    -l controller.devfile.io/devworkspace_name="${dw_name}" \
+    --for=condition=Ready \
+    --timeout=5m \
+    >/dev/stderr
+
+  pod_name=$(kubectl get pod \
+    -n "${dw_ns}" \
+    -l controller.devfile.io/devworkspace_name="${dw_name}" \
+    -o jsonpath='{.items[0].metadata.name}')
+
+  echo "${pod_name}"
+}
+
+verify_ca_bundle_in_workspace() {
+  local pod_name="$1"
+  local dw_ns="$2"
+  local expected_count="$3"
+  local cert_path="/public-certs/tls-ca-bundle.pem"
+
+  log_info "Verifying CA bundle in workspace..."
+
+  kubectl exec "${pod_name}" -n "${dw_ns}" -- test -f "${cert_path}"
+
+  local mounted_count
+  mounted_count=$(kubectl exec "${pod_name}" -n "${dw_ns}" -- \
+    sh -c "grep -c 'BEGIN CERTIFICATE' ${cert_path}")
+
+  log_info "Generated certificates : ${expected_count}"
+  log_info "Mounted certificates   : ${mounted_count}"
+
+  if [[ "${mounted_count}" -le "${expected_count}" ]]; then
+    log_error "Mounted certificate count validation failed"
+    return 1
+  fi
+
+  log_success "CA bundle verification passed"
+}
+
+cleanup_resources() {
+  local dw_ns="$1"
+  local dw_name="$2"
+
+  log_info "Cleaning up..."
+  kubectl delete dw "${dw_name}" -n "${dw_ns}" --ignore-not-found
+  rm -f *.pem *.key
+}

test/load/devworkspace_load_test.js

@@ -50,14 +50,6 @@ export const options = {
       iterations: maxDevWorkspaces,
       maxDuration: '3h',
     },
-    final_cleanup: {
-      executor: 'per-vu-iterations',
-      vus: 1,
-      iterations: 1,
-      exec: 'final_cleanup',
-      startTime: '0s',
-      maxDuration: '3h',
-    },
   }, thresholds: {
     'checks': ['rate>0.95'],
     'devworkspace_create_duration': ['p(95)<15000'],
@@ -203,6 +195,11 @@ export function handleSummary(data) {
   return loadTestSummaryReport;
 }
 
+export function teardown(data) {
+  console.log("Running final cleanup after all DevWorkspace creation finished...");
+  final_cleanup();
+}
+
 function createNewDevWorkspace(namespace, vuId, iteration) {
   const baseUrl = `${apiServer}/apis/workspace.devfile.io/v1alpha2/namespaces/${namespace}/devworkspaces`;
 
@@ -259,6 +256,13 @@ function waitUntilDevWorkspaceIsReady(vuId, crName, namespace) {
     attempts++;
   }
 
+  if (!isReady && attempts >= maxAttempts) {
+    console.error(
+        `GET [VU ${vuId}] Timed out waiting for DevWorkspace '${crName}' in namespace '${namespace}' ` +
+        `after ${attempts} attempts (${devWorkspaceReadyTimeout}s). Last known phase: '${lastPhase}'`
+    );
+  }
+
   if (res.status === 200) {
     if (isReady) {
       devworkspaceReady.add(1);