Skip to content

Vuln Processing crashes 4 GB container after upgrading to v4.90.1 #51508

Description

@rfairburn

Fleet versions

  • Discovered: v4.90.1
  • Reproduced: N/A (not yet reproduced on a clean install)

Web browser and operating system: N/A (server-side issue, observed via container crash logs)


💥 Actual behavior

Within a couple of minutes of a vuln processing run starting after upgrading to v4.90.1, the containers were crashing with an out-of-memory error. This was recorded on a number of customer environments as outlined by the attached labels.

Increasing memory to 8 GB appears to have helped for customer-traino, but I don't have retry data for the others yet.

🛠️ Expected behavior

Vuln processing completes without crashing the containers under the standard 4 GB memory limit.

🧑‍💻 Steps to reproduce

These steps:

  • Have been confirmed to consistently lead to reproduction in multiple Fleet instances.
  • Describe the workflow that led to the error, but have not yet been reproduced in multiple Fleet instances.
  1. Upgrade an existing installation to v4.90.1.
  2. Allow a vuln processing run to start.
  3. Observe container OOM crash within a couple of minutes.

🕯️ More info (optional)

  • Affected customer environments are indicated by the attached customer-* labels.
  • Raising the container memory limit to 8 GB appears to mitigate the issue (confirmed so far for one environment; retry data pending for the rest).

Metadata

Metadata

Assignees

Labels

#g-supply-chainSupply Chain product group:loadtestIssue that requires a loadtestP1Critical: Broken workflow (critical bug), potential vuln, new feature for immediate Fleet needbugSomething isn't working as documentedcustomer-gispencustomer-numacustomer-rialtocustomer-schurcustomer-traino~help-p1For oncall tickets, P1~released bugThis bug was found in a stable release.

Type

No type

Projects

Status
✔️Awaiting QA

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions