How to handle High Cardinality Metrics

High cardinality metrics are metrics that have a large number of unique values. This can occur when the metric is tracking a large number of entities, or when the metric itself has a high number of possible values. For example, a metric that tracks the number of requests per second for a web service would be high cardinality if the web service receives requests from a large number of different users or if the web service has a large number of different endpoints. High-cardinality metrics can be challenging to store and process efficiently. Traditional time-series databases are often not designed to handle large numbers of unique values and can become slow or inefficient when used with high-cardinality metrics. 

Examples of High Cardinality Metrics from Prometheus

1. HTTP Request Paths: The path of an HTTP request can have a high cardinality, especially in large-scale web applications. Each unique request path represents a different endpoint or resource on the server, and the number of unique paths can easily grow into the thousands or even millions.

2. User IDs or Session IDs: Metrics related to individual users or sessions, such as the number of active users or the duration of a user session, can have high cardinality. Each unique user or session ID represents a different entity, and the number of unique IDs can quickly become very large in large-scale applications.

3. Timestamps: Metrics that are collected at very high frequencies, such as system metrics or application logs, can result in high cardinality in the timestamp values. Each unique timestamp represents a different point in time, and the number of unique timestamps can grow rapidly over time.

4. Labels: In Prometheus, metrics can be tagged with labels to provide additional context or dimensions. If there are a large number of unique label values, this can introduce high cardinality to the metric. For example, a metric that tracks the number of requests to a web service, with labels for the HTTP method, status code, and server region, could potentially have a very high cardinality if there are many different combinations of these labels.

5. Error Codes: Metrics related to errors or exceptions can have high cardinality, especially in complex systems with many different types of errors. Each unique error code or message represents a different type of error, and the number of unique error codes can grow significantly over time.

Challenges of High Cardinality Metrics 

1. Storage: High-cardinality metrics can require a significant amount of storage space. This is because each unique value must be stored separately, rather than being aggregated into a single value. 

2. Processing: High-cardinality metrics can also be challenging to process. This is because each unique value must be processed separately, which can slow down queries and other operations. 

3. Aggregation: Aggregating high-cardinality metrics can be difficult. This is because it is not always clear how to aggregate the values in a meaningful way. For example, if you are tracking the number of requests per second for a web service, you might not be able to simply sum the values for all of the different endpoints. 

Solutions for High Cardinality Metrics 

There are several different solutions that can be used to address the challenges of high-cardinality metrics. 

1. Downsampling: Downsampling is a technique that reduces the number of data points that are stored by aggregating multiple data points into a single value. This can be done regularly, such as hourly or daily. Downsampling can help to reduce the storage space that is required for high-cardinality metrics, and it can also improve the performance of queries and other operations. 

2. Aggregation: Aggregation is a technique that combines multiple values into a single value. This can be done using a variety of different functions, such as sum, average, and min. Aggregation can help to reduce the number of unique values that are stored, and it can also make it easier to understand the data.

3. Time-series databases: There are several time-series databases that are designed to handle high-cardinality metrics. These databases typically use a distributed storage system to store the data, and they often use specialized techniques to optimize the storage and processing of high-cardinality metrics. 

Prometheus and High Cardinality Metrics 

Prometheus is a popular open-source monitoring system that can be used to collect and store high-cardinality metrics. Prometheus uses a distributed storage system called TSDB, which is designed to handle large volumes of data. Prometheus also uses several techniques to reduce the amount of data that needs to be stored and processed, such as downsampling and aggregation. Prometheus provides several features that make it well-suited for monitoring high-cardinality metrics, including: 

1. A flexible data model: Prometheus allows you to define your metrics, and you can specify the cardinality of each metric. 

2. Efficient storage and processing: Prometheus uses a distributed storage system and several optimization techniques to store and process high-cardinality metrics efficiently

3. Powerful query language: Prometheus provides a powerful query language that allows you to easily query and analyze your data. 

Conclusion 

High cardinality metrics can be a challenge to store and process efficiently. However, several solutions can be used to address this challenge. Prometheus is a popular open-source monitoring system that provides several features that make it well-suited for monitoring high-cardinality metrics. 

Thanks for reading my technical blog!

I'm glad you found my content informative and helpful. If you enjoyed this post, you can find more like it by following me on social media:

Twitter: @adityaajoshi12

LinkedIn: adityajoshi12

GitHub: adityajoshi12

References 

1. Prometheus documentation on high cardinality metrics 

2. Thanos documentation on high cardinality metrics