Potential Memory leak in a busy production environment

[jgzurano]

When running Robusta chart on a busy production cluster I'm noticing kubewatch container in robusta-forwarder pod shows a constant increase in memory usage over time leading to OOM kills.

Chart version: robusta-0.20.0
Kubewatch image: robustadev/kubewatch:v2.9.0 (same happens when I tested with latest version: kubewatch:v2.11.0)

Kube cluster size is medium, at a busy times could have 250-500 nodes, 6000-10000 pods.

Any ideas how can I remediate ? perhaps from garbage collection side to release memory and avoid hitting pod memory limits.
Any help is appreciated to identify if this is a consequence of watching all kinds of events or a memory management issue on kubewatch code.

Using default config from Chart for watched resources

    resource:                                                                                                                                                                        
      clusterrole: true                                                                                                                                                              
      clusterrolebinding: true                                                                                                                                                       
      configmap: true                                                                                                                                                                
      coreevent: false                                                                                                                                                               
      daemonset: true                                                                                                                                                                
      deployment: true                                                                                                                                                               
      event: true                                                                                                                                                                    
      hpa: true                                                                                                                                                                      
      ingress: true                                                                                                                                                                  
      job: true                                                                                                                                                                      
      namespace: true                                                                                                                                                                
      node: true                                                                                                                                                                     
      persistentvolume: true                                                                                                                                                         
      pod: true                                                                                                                                                                      
      replicaset: true                                                                                                                                                               
      replicationcontroller: false                                                                                                                                                   
      secret: false                                                                                                                                                                  
      serviceaccount: true                                                                                                                                                           
      services: true                                                                                                                                                                 
      statefulset: true

[aantn]

Hi @jgzurano,
Thanks for reporting. I'm looking into this and there might be a bug with leaking http buffers on some of the receivers. Which receivers do you have configured?

[jgzurano]

Hi @aantn
I understand our current sinksConfig include pagerduty_sink and slack_sink.
Please let me know if you need additional details, thank you.

[aantn]

Got it, I fixed a few bugs here (not all related to the receivers) but I'm not entirely sure if it will help in your case. There might still be another bug.

Can you test please? #104
Let me know if you need me to build an image.

[jgzurano]

I run a test with an image built from fix-memory-leaks branch, but I'm getting the same memory usage pattern.

Btw, a few days ago I ran some tests setting pod: false in kubewatch.yaml, that seems to eliminate the memory leak issue.
With that setting disabled the container runs stable.