docs(perf): Considerations for packetparser on high-core-count systems (#1965)

SRodi · web-flow · commit 2935c0de62bb · 2025-12-08T12:28:30.000Z
## Summary Documents community-reported performance considerations when running the `packetparser` plugin on high-core-count systems (32+ cores) under sustained network load. ## Changes - **packetparser.md**: Added performance considerations section with user-reported observations, current implementation details (PERF_EVENT_ARRAY), and mitigation options - **intro.md**: Added known limitations section highlighting performance considerations on large nodes - **architecture.md**: Added note about data transfer mechanisms and their performance implications - **performance.md**: New troubleshooting guide with diagnostic steps and mitigation options - **README.md**: Added performance warning in known limitations section ## Key Points - Acknowledges these are community user reports, not independently verified by maintainers - Accurately reflects current implementation (only supports perf arrays, no ring buffer support yet) - References external analysis ([blog post](https://blog.zmalik.dev/p/who-will-observe-the-observability) and [KubeCon talk](https://www.youtube.com/watch?v=J-Zx64mJzVk)) in packetparser.md only - Provides actionable guidance: use Basic metrics, enable sampling, monitor impact - Notes that the team is evaluating options for future releases ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [x] I have read the [contributing documentation](https://retina.sh/docs/Contributing/overview). - [x] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [x] I have correctly attributed the author(s) of the code. - [x] I have tested the changes locally. - [x] I have followed the project's style guidelines. - [x] I have updated the documentation, if necessary. - [x] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed <img width="2541" height="1280" alt="image" src="https://github.com/user-attachments/assets/e043da80-2be0-4725-bb43-4245e60a7b5f" /> ## Additional Notes Add any additional notes or context about the pull request here. --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project.
diff --git a/README.md b/README.md
@@ -33,6 +33,10 @@ Retina lets you **investigate network issues on-demand** and **continuously moni
 
 See [retina.sh](http://retina.sh) for documentation and examples.
 
+## Known Limitations
+
+⚠️ **Performance on High-Core-Count Systems**: Community users have reported performance considerations when using Advanced metrics (with `packetparser` plugin) on nodes with 32+ CPU cores under high network load. Consider starting with Basic metrics mode on large node types. See [Known Limitations](https://retina.sh/docs/Introduction/intro#known-limitations) for details.
+
 ## Capabilities
 
 Retina has two major features:
diff --git a/docs/01-Introduction/01-intro.md b/docs/01-Introduction/01-intro.md
@@ -94,3 +94,18 @@ Check out our talk from KubeCon 2024 which goes into this topic even further - [
 The following are known system requirements for installing Retina:
 
 - Minimum Linux Kernel Version: v5.4.0
+
+## Known Limitations
+
+### Performance Considerations for High-Core-Count Systems
+
+Community users have reported performance considerations when using **Advanced metrics with the `packetparser` plugin** on nodes with high CPU core counts (32+ cores) under sustained, high-volume network load.
+
+If you plan to deploy Retina in Advanced mode on large node types with network-intensive workloads, consider:
+
+1. **Start with Basic metrics mode** (does not use `packetparser`)
+2. Enable `dataSamplingRate` if you need Advanced metrics
+3. Monitor CPU usage and network throughput after deployment
+4. See [`packetparser` performance considerations](../03-Metrics/plugins/Linux/packetparser.md#performance-considerations) for more information
+
+The Retina team is evaluating options to address these reported concerns in future releases.
diff --git a/docs/01-Introduction/02-architecture.md b/docs/01-Introduction/02-architecture.md
@@ -14,6 +14,8 @@ The plugins have a very specific scope by design, and Retina is designed to be e
 
 The plugins are responsible for installing the eBPF programs into the host kernel during startup. These eBPF programs collect metrics from events in the kernel level, which are then passed to the user space where they are parsed and converted into a `flow` data structure. Depending on the Control Plane being used, the data will either be sent to a Retina Enricher, or written to an external channel which is consumed by a Hubble observer - more on this in the [Control Plane](#control-plane) section below. It is not required for a plugin to use eBPF, it can also use syscalls or other API calls. In either case, the plugins will implement the same [interface](https://github.com/microsoft/retina/blob/main/pkg/plugin/registry/registry.go).
 
+**Data Transfer Mechanisms:** eBPF programs transfer data from kernel to user space using specialized data structures. The `packetparser` plugin currently uses **perf arrays** (BPF_MAP_TYPE_PERF_EVENT_ARRAY), which create per-CPU buffers. Community users have reported performance considerations with this approach on high-core-count systems. See [packetparser performance considerations](../03-Metrics/plugins/Linux/packetparser.md#performance-considerations) for details.
+
 Some examlpes of existing Retina plugins:
 
 - Drop Reason - measures the number of packets/bytes dropped and the reason and the direction of the drop.
diff --git a/docs/03-Metrics/plugins/Linux/packetparser.md b/docs/03-Metrics/plugins/Linux/packetparser.md
@@ -15,6 +15,39 @@ The `packetparser` plugin requires the `CAP_NET_ADMIN` and `CAP_SYS_ADMIN` capab
 
 `packetparser` does not produce Basic metrics. In Advanced mode (refer to [Metric Modes](../../modes/modes.md)), the plugin transforms an eBPF result into an enriched `Flow` by adding Pod information based on IP. It then sends the `Flow` to an external channel, enabling *several modules* to generate Pod-Level metrics.
 
+## Performance Considerations
+
+### Reported Performance Impact on High-Core-Count Systems
+
+Community users have reported performance considerations when running the `packetparser` plugin on systems with high CPU core counts (32+ cores) under sustained network load. While these reports have not been independently verified by the Retina maintainers, we document them here for awareness.
+
+**User-Reported Observations:**
+
+A detailed analysis by a Retina user (see [this blog post](https://blog.zmalik.dev/p/who-will-observe-the-observability)) and [KubeCon 2025 talk](https://www.youtube.com/watch?v=J-Zx64mJzVk) documented performance degradation that scaled non-linearly with CPU core count on nodes running network-intensive, multi-threaded workloads.
+
+**Current Implementation:**
+
+The `packetparser` plugin currently uses **BPF_MAP_TYPE_PERF_EVENT_ARRAY** for kernel-to-userspace data transfer. This architecture creates per-CPU buffers that must be polled by a single reader thread. On systems with many CPU cores, this can lead to:
+
+- Increased context switching overhead
+- Memory access patterns that may not scale linearly
+- Potential NUMA-related penalties on multi-socket systems
+
+**Alternative Approaches:**
+
+Alternative data transfer mechanisms like BPF ring buffers (BPF_MAP_TYPE_RINGBUF, available in Linux kernel 5.8+) use a shared buffer architecture that may perform better on high-core-count systems. However, **Retina does not currently support ring buffers for packetparser**. Future versions may provide configurable data transfer mechanisms.
+
+#### If You Experience Performance Issues
+
+If you observe performance degradation on high-core-count nodes:
+
+1. **Disable `packetparser`**: Use Basic metrics mode which doesn't require this plugin
+2. **Enable Sampling**: Use the `dataSamplingRate` configuration option (see [Sampling](#sampling) section)
+3. **Use High Data Aggregation**: Configure `high` [data aggregation](../../../05-Concepts/data-aggregation.md)
+4. **Monitor Impact**: Watch for elevated CPU usage, context switches, or throughput changes
+
+**Note:** The Retina team is evaluating options for addressing reported performance concerns, including potential support for alternative data transfer mechanisms. Community feedback and contributions are welcome.
+
 ## Sampling
 
 Since `packetparser` produces many enriched `Flow` objects it can be quite expensive for user space to process.  Thus, when operating in `high` [data aggregation](../../../05-Concepts/data-aggregation.md) level optional sampling for reported packets is available via the `dataSamplingRate` configuration option.
diff --git a/docs/06-Troubleshooting/performance.md b/docs/06-Troubleshooting/performance.md
@@ -0,0 +1,176 @@
+# Performance Troubleshooting
+
+This guide helps diagnose and address potential performance issues when running Retina, particularly the `packetparser` plugin on high-core-count systems.
+
+## Background
+
+Community users have reported performance considerations when running the `packetparser` plugin (used in Advanced metrics mode) on systems with high CPU core counts under sustained network load. For detailed background, see the [`packetparser` performance considerations](../03-Metrics/plugins/Linux/packetparser.md#performance-considerations).
+
+## Symptoms to Monitor
+
+Watch for these indicators after deploying Retina:
+
+- **Decreased network throughput** compared to baseline
+- **High CPU usage** by Retina agent pods
+- **Elevated context switches** on nodes running Retina
+- **Increased latency** in network-intensive applications
+
+## Diagnostic Steps
+
+### Step 1: Identify Your Configuration
+
+Check which plugins are enabled:
+
+```bash
+kubectl get configmap retina-config -n kube-system -o yaml | grep enabledPlugin
+```
+
+If `packetparser` is enabled, you're running Advanced metrics mode which is more resource-intensive.
+
+### Step 2: Check Node Specifications
+
+```bash
+# Check core count on nodes
+kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu
+
+# Identify nodes with high core counts (32+)
+kubectl get nodes -o json | jq '.items[] | select((.status.capacity.cpu | tonumber) >= 32) | {name: .metadata.name, cpu: .status.capacity.cpu}'
+```
+
+### Step 3: Monitor Retina Resource Usage
+
+```bash
+# Check CPU and memory usage of Retina pods
+kubectl top pods -n kube-system -l app=retina
+
+# For more detailed analysis, check specific pod on a node
+RETINA_POD=$(kubectl get pods -n kube-system -l app=retina -o jsonpath='{.items[0].metadata.name}')
+kubectl top pod $RETINA_POD -n kube-system
+```
+
+### Step 4: Establish Performance Baseline
+
+Before and after Retina deployment, measure:
+
+- Network throughput (using your application's metrics or tools like iperf3)
+- Application response times
+- CPU utilization on nodes
+
+## Mitigation Options
+
+If you observe performance impact, consider these approaches:
+
+### Option 1: Use Basic Metrics Mode (Recommended)
+
+Basic metrics mode provides node-level observability without the `packetparser` plugin:
+
+```bash
+# Reinstall or upgrade Retina without packetparser
+helm upgrade retina oci://ghcr.io/microsoft/retina/charts/retina \
+    --set enabledPlugin_linux="\[dropreason\,packetforward\,linuxutil\,dns\]" \
+    --reuse-values
+```
+
+**Trade-off:** You'll have node-level metrics only, not pod-level metrics.
+
+### Option 2: Enable Data Sampling
+
+Reduce event volume by sampling packets:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: retina-config
+  namespace: kube-system
+data:
+  config.yaml: |
+    dataSamplingRate: 10  # Sample 1 out of every 10 packets
+```
+
+**Trade-off:** Reduced data granularity, but lower overhead.
+
+### Option 3: Use High Data Aggregation Level
+
+Reduce events at the eBPF level:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: retina-config
+  namespace: kube-system
+data:
+  config.yaml: |
+    dataAggregationLevel: "high"
+```
+
+**Trade-off:** Disables host interface monitoring; API server latency metrics may be less reliable.
+
+### Option 4: Selective Deployment
+
+Deploy Retina only on nodes where you need detailed observability:
+
+```yaml
+# Use node selectors or taints/tolerations
+apiVersion: apps/v1
+kind: DaemonSet
+spec:
+  template:
+    spec:
+      nodeSelector:
+        retina-enabled: "true"
+```
+
+## Advanced Diagnostics
+
+### Inspecting eBPF Maps
+
+To see what data structures Retina is using:
+
+```bash
+# Access the node
+kubectl debug node/<node-name> -it --image=ubuntu
+
+# In the debug container, enter the host namespace
+chroot /host
+
+# List BPF maps (requires bpftool)
+bpftool map list | grep retina
+
+# Check the packetparser map type
+bpftool map show name retina_packetparser_events
+```
+
+Currently, `packetparser` uses `BPF_MAP_TYPE_PERF_EVENT_ARRAY`.
+
+### Monitoring Event Rates (Advanced)
+
+If you have bpftrace available on nodes:
+
+```bash
+# Monitor perf_event activity
+sudo bpftrace -e '
+  kprobe:perf_event_output { @events = count(); }
+  interval:s:5 { print(@events); clear(@events); }
+'
+```
+
+High event rates may correlate with increased CPU usage.
+
+## Reporting Issues
+
+If you experience performance issues, please report them with:
+
+1. **Node specifications**: CPU count, memory, kernel version
+2. **Retina configuration**: Version, enabled plugins, configuration settings
+3. **Workload characteristics**: Network throughput, number of pods, traffic patterns
+4. **Performance metrics**: CPU usage, network throughput before/after, specific observations
+
+Open an issue at: <https://github.com/microsoft/retina/issues>
+
+## Further Resources
+
+- [Packetparser Performance Considerations](../03-Metrics/plugins/Linux/packetparser.md#performance-considerations)
+- [Data Aggregation Levels](../05-Concepts/data-aggregation.md)
+- [Configuration Options](../02-Installation/03-Config.md)
diff --git a/site/package-lock.json b/site/package-lock.json
