Skip to content

Comments

Kill Worker If Redis Connection/Job Fetch Fails Repeatedly#13

Merged
tallen42 merged 3 commits intomasterfrom
logging-update-worker
May 3, 2025
Merged

Kill Worker If Redis Connection/Job Fetch Fails Repeatedly#13
tallen42 merged 3 commits intomasterfrom
logging-update-worker

Conversation

@jabbate19
Copy link
Collaborator

Currently, if the redis connection dies, the logs will get flooded with Broken Pipe errors. The only way to fix is to manually reset the pod

This PR:

  • Adds better logging prefixes so you can more easily find the line failing
  • Sleep 3s if the redis connection fails
  • Kill the pod if 3+ Redis Connection Failures

Test Plan:

  • Run updated code worker
  • Kill redis pod
  • Observe failure process in logs
image

@tallen42
Copy link
Contributor

tallen42 commented May 3, 2025

lgtm

@tallen42 tallen42 merged commit 83df93d into master May 3, 2025
2 checks passed
@tallen42 tallen42 deleted the logging-update-worker branch May 3, 2025 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants