This content originally appeared on DEV Community and was authored by Amanda Guan
The Thanos Query UI provides robust querying capabilities across multiple Prometheus instances, offering a comprehensive analysis of your metrics. Using the Thanos Interactive example dataset, I experimented with various PromQL queries to explore metrics. Here’s a summary based on frequent use cases:
Available Metrics in the Metrics Explorer:
- continuous_app_metric0
- continuous_app_metric1
- continuous_app_metric2
- continuous_app_metric3
- continuous_app_metric4
PromQL Query Examples:
-
Sum of Metrics:
sum(continuous_app_metric0)– Sum across all clusters and replicas. -
Sum by Cluster:
sum(continuous_app_metric0) by (cluster)– Grouped by cluster. -
Sum by Replica:
sum(continuous_app_metric0) by (replica)– Grouped by replica. -
Average by Cluster:
avg(continuous_app_metric0) by (cluster)– Average grouped by cluster. -
Maximum Value by Replica:
max(continuous_app_metric0) by (replica)– Max value grouped by replica. -
Rate of Change:
rate(continuous_app_metric0[5m])– Rate of change over 5 minutes. -
Increase Over Time:
increase(continuous_app_metric0[1h])– Increase over the past hour. -
Histogram Quantile:
histogram_quantile(0.95, sum(rate(continuous_app_metric0_bucket[5m])) by (le))– 95th percentile if part of a histogram. -
Count of Instances:
count(continuous_app_metric0)– Count of reporting instances. -
Specific Cluster and Replica:
continuous_app_metric0{cluster="eu-1", replica="0"}– Current value for a specific cluster and replica. -
Comparison Between Clusters:
continuous_app_metric0{cluster="eu-1"} / continuous_app_metric0{cluster="us-1"}– Metric comparison between clusters. -
Alerting:
continuous_app_metric0 > 100– Create an alert if the metric exceeds a threshold.
Understanding Empty Query Results:
-
Histogram Quantile Query: May return empty if
_bucketsuffix is missing, required forhistogram_quantile. - Comparison Between Clusters: Returns empty if data points are missing or unmatched in timestamps.
PromQL Info Messages:
-
Rate and Increase Queries: Indicate that
continuous_app_metric0might not be a counter as it does not end in_total,_sum,_count, or_bucket.
Metric Types Explained:
- Counters: Metrics that increase or reset to zero. Used for tracking quantities like server requests or encountered errors.
- Gauges: Metrics that can both increase and decrease. Used for measuring fluctuating values like memory usage or temperature.
- Non-Counter Metrics: Includes gauges and histograms, useful for tracking non-monotonically increasing values.
This content originally appeared on DEV Community and was authored by Amanda Guan