Skip to content

Comments

Kubernetes Job service#5113

Merged
javanlacerda merged 42 commits intomasterfrom
pr/dependencies
Jan 22, 2026
Merged

Kubernetes Job service#5113
javanlacerda merged 42 commits intomasterfrom
pr/dependencies

Conversation

@javanlacerda
Copy link
Collaborator

@javanlacerda javanlacerda commented Jan 8, 2026 •
edited
Loading

This PR introduces full support for scheduling and managing fuzzing tasks on Kubernetes clusters,
specifically targeting GKE. It implements a new KubernetesService to
handle batch job creation, supports Kata Containers for isolation, and includes robust testing
and configuration mechanisms.

Key Features:

  • Kubernetes Service: A new backend for RemoteTaskInterface that schedules tasks as Kubernetes
    Jobs. It supports both standard and Kata Container runtimes, automatic Service Account
    creation with Workload Identity, and intelligent job limiting to prevent cluster overload.
  • Traffic Shifting (RemoteTaskGate): A new gating mechanism (RemoteTaskGate) that intelligently
    routes tasks between the legacy GCP Batch service and the new Kubernetes service based on
    configurable probabilities, allowing for a gradual, controlled migration.
  • Feature Flags: A new dynamic configuration system backed by Datastore to control runtime
    behaviors like job concurrency limits.

Detailed Changes by Module:

  • Kubernetes Integration (src/clusterfuzz/_internal/k8s/):

    • service.py: Implemented KubernetesService for job lifecycle management (creation,
      monitoring, limiting). Includes GKE credential loading, Kata Container spec generation,
      and Service Account provisioning.
    • Tests: Added k8s_service_test.py (unit), k8s_service_limit_test.py (limits), and
      k8s_service_e2e_test.py (integration on Kind).

  • Remote Task Management (src/clusterfuzz/_internal/remote_task/):

    • init.py: Introduced RemoteTaskGate, a smart router that implements
      RemoteTaskInterface. It initializes both GcpBatchService and KubernetesService and
      distributes tasks between them based on probabilities defined in job_frequency.py. This
      enables traffic splitting (e.g., 10% to K8s, 90% to Batch) for safe rollout.
    • job_frequency.py: Added logic to manage task scheduling frequency and split ratios.
    • Refactored core task logic to use the generic RemoteTask and RemoteTaskInterface
      abstractions.

  • Datastore & Configuration (src/clusterfuzz/_internal/datastore/):

    • data_types.py: Added FeatureFlag model to store configuration dynamically.
    • feature_flags.py: Added FeatureFlags enum/helper for type-safe access to flags (e.g.,
      K8S_PENDING_JOBS_LIMITER).

  • Batch & Legacy Refactoring (src/clusterfuzz/_internal/batch/):

    • service.py: Updated to align with the new RemoteTask interface.
    • Removed obsolete gcp.py and google_cloud_utils/batch.py utilities in favor of the new
      structure.

  • Infrastructure & CI:

    • .github/workflows/kubernetes-e2e-tests.yaml: New workflow for running E2E tests on a Kind
      cluster.
    • Pipfile / src/Pipfile: Added kubernetes client and updated Google Cloud dependencies.

  • Bot & Metrics:

    • src/python/bot/startup/run_bot.py: Updates to support K8s-based bot execution via the new
      gate.
    • src/clusterfuzz/_internal/metrics/: Enhanced logging and monitoring for remote tasks.

Evidences:

image Batch and Kata containers fuzzing hours, proving the Remote Gate, the Batch and Kubernetes services are working properly. The Feature Flag is used to set the job_frequency, then it proves the feature flag and its usage is working as well.

@javanlacerda javanlacerda changed the title Pr/dependencies Kubernetes Job service Jan 8, 2026
@javanlacerda javanlacerda force-pushed the pr/dependencies branch 5 times, most recently from 7edfb48 to b271b83 Compare January 10, 2026 22:46
@javanlacerda javanlacerda marked this pull request as ready for review January 10, 2026 22:50
@javanlacerda javanlacerda requested review from ViniciustCosta, decoNR, hunsche and jonathanmetzman and removed request for ViniciustCosta and hunsche January 10, 2026 22:50
jonathanmetzman
jonathanmetzman reviewed Jan 12, 2026
View reviewed changes
Copy link
Collaborator

@jonathanmetzman jonathanmetzman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some very surface level comments.

@javanlacerda javanlacerda force-pushed the pr/dependencies branch 4 times, most recently from d5684e5 to 44737d3 Compare January 15, 2026 00:44
@javanlacerda
feat: Implement Kubernetes job client and service
f02d4a0 
This commit introduces the Kubernetes job client and service, providing a mechanism to schedule tasks on Kubernetes clusters (including GKE and Kind), supporting both standard and Kata Containers.

Key Features & Changes:
- **Kubernetes Service**: Implemented `KubernetesService` in `clusterfuzz._internal.k8s.service` to manage job creation.
- **Kata Support**: Added specialized job creation for Kata Containers (`create_kata_container_job`) with required security context (`privileged`, `capabilities: ALL`), networking (`hostNetwork: True`), and environment variables (`HOST_UID`).
- **Dependency Management**: Added `kubernetes` and necessary Google Cloud dependencies (`google-api-python-client`, `google-cloud-storage`, `google-cloud-ndb`, etc.) to `Pipfile`.
- **E2E Testing**:
    - Created `tests.core.k8s.k8s_service_e2e_test` to verify job lifecycle on a local Kind cluster.
    - Updated `local/tests/kubernetes_e2e_test.bash` to provision the test environment.
    - Updated CI workflow (`.github/workflows/kubernetes-e2e-tests.yaml`) to install JDK 21 (required for Datastore emulator).
    - Tests now verify job "Running" status to avoid timeouts with long-running commands.
    - `KubernetesService` skips default credential loading when `K8S_E2E` is set to utilize the test-provided kubeconfig.
- **Unit Tests**: Added comprehensive unit tests in `tests.core.k8s.k8s_service_test` and `tests.core.kubernetes.kubernetes_test`, including mocking of `load_kube_config` and `_load_gke_credentials` to ensure robust testing without external dependencies.
@javanlacerda
pipenv lock
697cdd7 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
install kubernetes
3310027 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
Update dependencies and fix linting
7f12a07
@javanlacerda
move use_batch
b986275 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
add todo
5b7b4d5 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
Pr/metrics logging (#5115)
ca629cd 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
fixes
234f0ac 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
fix lint
6838442 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
mock gcloud auth default
eec7e6b 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
ViniciustCosta
ViniciustCosta reviewed Jan 19, 2026
View reviewed changes
decoNR
decoNR reviewed Jan 19, 2026
View reviewed changes
@javanlacerda
fix single quotes usage
2114adf 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
refactor
5a92336 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
decoNR
decoNR reviewed Jan 20, 2026
View reviewed changes
@javanlacerda
rename types
5a3a5a2 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
fixes
fcc67fe 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
deal with circular import
1aa9ae5 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
simplify k8s e2e
f4e61b7 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda javanlacerda force-pushed the pr/dependencies branch 2 times, most recently from 30cc0ae to 506f583 Compare January 20, 2026 21:10
@javanlacerda
Copy link
Collaborator Author

javanlacerda commented Jan 21, 2026

Are we using preemptibles btw?

We're not, and will it's not part of the plan using it for the clusters. You can see more details on go/clusterfuzz-to-kubernetes

@javanlacerda
get rid job limiter
43ddec1 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
decoNR
decoNR reviewed Jan 21, 2026
View reviewed changes
jonathanmetzman
jonathanmetzman reviewed Jan 22, 2026
View reviewed changes
src/clusterfuzz/_internal/k8s/service.py Outdated


def _get_config_names(remote_tasks: typing.List[types.RemoteTask]):
""""Gets the name of the configs for each batch_task. Returns a dict
Copy link
Collaborator

@jonathanmetzman jonathanmetzman Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

""""Gets
should be """Gets

src/clusterfuzz/_internal/k8s/service.py Outdated
ca_cert = base64.b64decode(cluster['masterAuth']['clusterCaCertificate'])

# Write CA cert to a temporary file.
fd, ca_cert_path = tempfile.mkstemp()
Copy link
Collaborator

@jonathanmetzman jonathanmetzman Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping

jonathanmetzman
jonathanmetzman reviewed Jan 22, 2026
View reviewed changes
ViniciustCosta
ViniciustCosta approved these changes Jan 22, 2026
View reviewed changes
Copy link
Collaborator

@ViniciustCosta ViniciustCosta left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, lgtm!

Please, fix the final comments and tests before merging :)

@javanlacerda
fix lint
68e950f 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
fixes
04608f1 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
set cert in bot dir
fd8f6b3 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda
get cluster_name from config
b031845 
Signed-off-by: Javan Lacerda <javanlacerda@google.com>
@javanlacerda javanlacerda merged commit c97d8cc into master Jan 22, 2026
11 checks passed
@javanlacerda javanlacerda deleted the pr/dependencies branch January 22, 2026 18:16
vitaliset added a commit that referenced this pull request Jan 22, 2026
@vitaliset
Revert "Kubernetes Job service (#5113)"
2eca076 
This reverts commit c97d8cc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@decoNR decoNR decoNR left review comments

@ViniciustCosta ViniciustCosta ViniciustCosta approved these changes

@jonathanmetzman jonathanmetzman jonathanmetzman approved these changes

@hunsche hunsche Awaiting requested review from hunsche

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@javanlacerda @jonathanmetzman @ViniciustCosta @decoNR