Understanding missing metrics
The absence of metrics doesn’t necessarily indicate a collector failure. Missing metrics are often caused by normal service behavior or collection timing rather than a configuration or collection issue.
Common reasons why metrics may be unavailable even when the collector is functioning correctly include:
Sparse metrics. Not all metrics are emitted continuously. Some metrics are only recorded when specific events occur, such as bucket reads or writes, connection activity, or query execution. If no such activity occurred during the query time range, no data points will exist and the response will be empty.
Bucket inactivity. For Object Storage, metrics are tied to bucket-level activity. A bucket that has had no reads, writes, or deletes during the query time range won’t produce activity-based metrics. The absence of data accurately reflects the absence of activity, and doesn’t indicate a collection failure.
Scrape interval settings. The collector collects metrics at a configured polling interval (polling_interval). Data points are only written at the end of each scrape cycle. If the query time range falls entirely between two scrape cycles, or the collector has only recently started, there may not yet be any data to return. Expanding the query time range to span at least two or three scrape intervals usually resolves this issue.
Query delay settings. Query delay (query_delay) introduces a deliberate offset between when data is scraped and when it becomes queryable. Query delay is used to account for ingestion latency in the backend. If query delay is configured and the query time range doesn’t account for this offset, the query may return no data even though data exists.
Collector uptime
The collector only collects metric data while it’s running. If the collector is stopped, restarted frequently, or runs as a short-lived process, metric data may be missed during periods when collection is not active. This is especially important for:
- Object storage metrics
- Metrics with longer scrape intervals, such as one hour
- Spare workloads that generate metric data infrequently
- Long-term monitoring and trend analysis