Custom actuator metric with request counter

[dennisvang]

For #661 we are interested in counting the requests to individual resources.

However, the default Spring MVC actuator metric http.server.requests does not count requests for individual resources. Instead, it only shows counts for the path patterns defined for the controllers, such as

FAIRDataPoint/src/main/java/org/fairdatapoint/api/controller/metadata/GenericController.java

Line 150 in f65cd35

path = {"", "{oUrlPrefix:[^.]+}/{oRecordId:[^.]+}"},

To count requests to individual resource URIs, I added a custom micrometer ObservationFilter implementation that replaces the default uri path pattern by the full path. Also see spring observability docs.

Note

Although it is relatively quick and easy to use Spring actuator metrics for this, I'm not convinced this is the best approach. I think it would make more sense to do this at the deployment level, e.g. based on proxy logs, instead of the app level.

fixes #661

todo

• Exclude default metrics, only expose the custom one.
  So, figure out why the following lines are ignored by the fdp, whereas they do work in a fresh spring project, and they also work without issue if we apply them to the develop branch...
  

  FAIRDataPoint/src/main/resources/application.yml

  Lines 35 to 39 in 4c69720

  	metrics:
  	enable:
  	all: false
  	http:
  	server: true

• add tests to check metrics filtering and content of server metrics

usage example

To see totals and list available endpoints (they only appear in the list after they have been visited at least once):

curl http://localhost:8080/actuator/metrics/http.server.requests | jq

{
  "name": "http.server.requests",
  "baseUnit": "seconds",
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 67.0
    },
    {
      "statistic": "TOTAL_TIME",
      "value": 1.3833911090000002
    },
    {
      "statistic": "MAX",
      "value": 0.089165968
    }
  ],
  "availableTags": [
    {
      "tag": "exception",
      "values": [
        "none"
      ]
    },
    {
      "tag": "method",
      "values": [
        "POST",
        "PUT",
        "GET"
      ]
    },
    {
      "tag": "http.url",
      "values": [
        "/index/admin/trigger",
        "/configs/bootstrap",
        "/index/entries",
        "/index/entries/info",
        "/index/entries/654b5878-fc8a-44db-bb47-81754617f5a5",
        "/spec",
        "/actuator/info",
        "/actuator/metrics/",
        "/actuator/metrics/http.server.requests",
        "/meta",
        "/"
      ]
    },
    {
      "tag": "error",
      "values": [
        "none"
      ]
    },
    {
      "tag": "outcome",
      "values": [
        "CLIENT_ERROR",
        "SUCCESS"
      ]
    },
    {
      "tag": "status",
      "values": [
        "404",
        "200",
        "204"
      ]
    }
  ]
}

To see details for a specific endpoint:

http://localhost:8080/actuator/metrics/http.server.requests?tag=http.url:/index/entries/654b5878-fc8a-44db-bb47-81754617f5a5 | jq

{
  "name": "http.server.requests",
  "baseUnit": "seconds",
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 3.0
    },
    {
      "statistic": "TOTAL_TIME",
      "value": 0.138951876
    },
    {
      "statistic": "MAX",
      "value": 0.05697992
    }
  ],
  "availableTags": [
    {
      "tag": "exception",
      "values": [
        "none"
      ]
    },
    {
      "tag": "method",
      "values": [
        "PUT",
        "GET"
      ]
    },
    {
      "tag": "error",
      "values": [
        "none"
      ]
    },
    {
      "tag": "outcome",
      "values": [
        "SUCCESS"
      ]
    },
    {
      "tag": "status",
      "values": [
        "200"
      ]
    }
  ]
}

[dennisvang]

Initially I implemented this by extending OncePerRequestFilter and creating a custom metric.

However, a micrometer dev suggested another approach in this comment from 2018.
Do note that the classes mentioned in that comment have been deprecated in favor of the micrometer observation APIs, as mentioned in spring boot 3 release notes.

The latest and greatest approach is probably to override either DefaultServerRequestObservationConvention, or micrometer ObservationFilter, as mentioned in the observability docs.

[dennisvang]

ran into #839

[dennisvang]

The current implementation still shows all metrics, so the filter in application.yml is ignored for some reason.

This can be checked as follows:

curl http://localhost:8080/actuator/metrics | jq

which still returns a long list, instead of just http.server.requests:

Details

{
  "names": [
    "application.ready.time",
    "application.started.time",
    "executor.active",
    "executor.completed",
    "executor.pool.core",
    "executor.pool.max",
    "executor.pool.size",
    "executor.queue.remaining",
    "executor.queued",
    "http.client.requests",
    "http.client.requests.active",
    "http.server.requests",
    "http.server.requests.active",
    "mongodb.driver.commands",
    "mongodb.driver.pool.checkedout",
    "mongodb.driver.pool.checkoutfailed",
    "mongodb.driver.pool.size",
    "mongodb.driver.pool.waitqueuesize",
    "spring.data.repository.invocations",
    "spring.security.authorizations",
    "spring.security.authorizations.active",
    "spring.security.filterchains",
    "spring.security.filterchains.CORSFilter.after",
    "spring.security.filterchains.CORSFilter.before",
    "spring.security.filterchains.JwtTokenFilter.after",
    "spring.security.filterchains.JwtTokenFilter.before",
    "spring.security.filterchains.LoggingFilter.after",
    "spring.security.filterchains.LoggingFilter.before",
    "spring.security.filterchains.access.exceptions.after",
    "spring.security.filterchains.access.exceptions.before",
    "spring.security.filterchains.active",
    "spring.security.filterchains.authentication.anonymous.after",
    "spring.security.filterchains.authentication.anonymous.before",
    "spring.security.filterchains.authorization.after",
    "spring.security.filterchains.authorization.before",
    "spring.security.filterchains.context.async.after",
    "spring.security.filterchains.context.async.before",
    "spring.security.filterchains.context.holder.after",
    "spring.security.filterchains.context.holder.before",
    "spring.security.filterchains.context.servlet.after",
    "spring.security.filterchains.context.servlet.before",
    "spring.security.filterchains.header.after",
    "spring.security.filterchains.header.before",
    "spring.security.filterchains.logout.after",
    "spring.security.filterchains.logout.before",
    "spring.security.filterchains.requestcache.after",
    "spring.security.filterchains.requestcache.before",
    "spring.security.filterchains.session.management.after",
    "spring.security.filterchains.session.management.before",
    "spring.security.filterchains.session.urlencoding.after",
    "spring.security.filterchains.session.urlencoding.before",
    "spring.security.http.secured.requests",
    "spring.security.http.secured.requests.active",
    "tomcat.sessions.active.current",
    "tomcat.sessions.active.max",
    "tomcat.sessions.alive.max",
    "tomcat.sessions.created",
    "tomcat.sessions.expired",
    "tomcat.sessions.rejected"
  ]
}

[dennisvang]

Debugging shows that the management.metrics.enable.all: false filter has no effect here because the metrics registry is not auto-configured properly.

This is caused by conflicting annotations on the Application class and misconfiguration in the AclConfig and AclMethodSecurityConfiguration

UPDATE: Now fixed by #906