Tackling Path Parameters with Service Profiles and Prometheus

To truly understand service behavior in Kubernetes, you must also understand their requests and responses, including the HTTP path. With a nearly unlimited number of unique potential paths, time series capture tends to be overwhelmed, making it difficult to get a clear picture of service behaviors.

In his KubeCon 2019 talk, Alex Leong introduces the Service Profile concept, a custom Kubernetes resource used by Linkerd. It allows operators to define permitted routes for the service with regular expressions allowing Prometheus to scrape and aggregate service behaviors. Operators can also set detailed service behavior rules such as retries and timeouts that can be easily reproduced and monitored.