[mcache] Support memcached built-in proxy mode (proxy binding protocol)

[mstuparu]

Summary

The mcache integration fails against memcached instances running in built-in proxy mode.

Example memcached 1.6.43 command line:

memcached -m 896 -c 130000 -I 2m -R 250 \
  -o modern,proxy_config=routelib,proxy_arg=/etc/memcached/proxy_routes.lua -v

The check connects via bmemcached (binary protocol) but receives an empty stats payload, causing a hard failure.

We'd like native mcache support so proxy instances can be monitored with the same integration as standard memcached nodes.

Environment

• Integration: mcache (datadog_checks/mcache/mcache.py)
• Client library: python-binary-memcached (bmemcached)
• Memcached version: 1.6.43 (official Docker image)
• Proxy config: modern,proxy_config=routelib,proxy_arg=/etc/memcached/proxy_routes.lua
• Datadog Agent: 7.x (Kubernetes autodiscovery via pod annotations)
• Deployment: Kubernetes StatefulSet — one memcached container per pod acting as a frontend proxy to backend memcached pods

Current behavior

Autodiscovery config:

{
  "mcache": {
    "init_config": {},
    "instances": [
      {
        "url": "%%host%%",
        "port": 11211,
        "options": {
          "items": true,
          "slabs": true
        }
      }
    ]
  }
}

Agent error:

ConfigurationError: Unable to retrieve stats from memcache instance: <host>:11211.
Please check your configuration. (Malformed response for host: {})

Traceback (abbreviated):

File ".../mcache/mcache.py", line 290, in check
    self._get_metrics(client, tags, service_check_tags)
File ".../mcache/mcache.py", line 126, in _get_metrics
    stats = self._process_response(client.stats())
File ".../mcache/mcache.py", line 116, in _process_response
    raise BadResponseError("Malformed response for host: {}".format(stats))

bmemcached.Client.stats() returns {host: {}} — the connection succeeds, but the stats dict is empty.

Root cause

When memcached runs with built-in proxy enabled:

• The listener binding protocol becomes proxy, not binary / ascii / negotiating
• The proxy frontend supports basic and meta protocols for client traffic, but mcache uses the binary protocol via bmemcached
• Binary STAT against a proxy listener does not return the standard stats dict the check expects
• Even if basic stats worked, stats items / stats slabs are relative to the proxy process (which typically stores no items in routelib route_direct setups). Proxy-specific telemetry lives under stats proxy

References:

• https://docs.memcached.org/features/proxy/
• How do you setup a replicated memcached proxy? memcached/memcached#1152
• Some issues with internal APIs memcached/memcached#1073

Suggested fix

Explicit instance option:

instances:
  - url: localhost
    port: 11211
    options:
      proxy_mode: true
      proxy_stats: true

When enabled:

• Collect proxy-specific metrics (from stats proxy) and standard connection vitals
• Do not collect cache storage metrics (curr_items, fill_percent, slab metrics, etc.)

Current workaround

We scrape prometheus/memcached-exporter v0.16.0 via the OpenMetrics check:

{
  "openmetrics": {
    "instances": [{
      "openmetrics_endpoint": "http://%%host%%:%%port%%/metrics",
      "namespace": "memcached",
      "metrics": [
        "memcached_proxy_backend_failed",
        "memcached_proxy_conn_errors",
        "memcached_current_connections"
      ]
    }]
  }
}

This works, but all metrics are billed as custom metrics (vs standard integration metrics from mcache), and requires a sidecar exporter in every proxy pod.

Acceptance criteria

• mcache check succeeds against memcached 1.6.x with proxy_config=routelib
• memcache.can_connect service check reports OK
• Proxy-specific metrics collected from stats proxy
• options.items / options.slabs gracefully skipped on proxy instances
• Documentation on proxy vs backend monitoring