Your dashboard refreshed a few seconds ago. The health indicator is still green. Did the application report that it was healthy a few seconds ago—or did the monitoring system just read the same old answer?
Those are different events. If you collect health into a file and let an exporter serve it, the exporter can keep answering after the collection job has stopped. A successful scrape doesn't prove that the report inside it is recent.
The useful question isn't just “is it green?” It's “what was checked, when was it checked, and can I still see the path that delivered the answer?”
A fresh scrape can carry an old report
Prometheus records up=1 when a target scrape succeeds and up=0 when it fails. For a node_exporter target, that describes the scrape—not the health of every application whose metrics it exposes.
Prometheus: jobs and instances
The node_exporter textfile collector reads files ending in .prom. An unchanged file can therefore keep supplying the same health value on successive scrapes. Store the collection time as a gauge value alongside the report; textfile collection doesn't support explicit sample timestamps.
node_exporter: textfile collector
This is not the same as Prometheus's own time-series staleness. Prometheus can stop returning a series when its source stops supplying it. Here, the exporter is still supplying it: the underlying observation is old, even though the series keeps receiving samples.
We use this distinction in Eggcracker's reported-health integration. A collection attempt exports its completion time and whether the selected doctor fields were valid. A failed query can publish a fresh “query invalid” result without health gauges. But if publication fails before replacement, the previous file can remain. Its age—not a newly written failure value—is then the clue.
Released collector implementation
Follow one failure through the clock
Here's a worked example, not an incident report. Assume one configured, opted-in target; aligned clocks; collection every minute; and rule evaluations every 30 seconds. For simplicity, each changed file is scraped before the next listed evaluation. Real scrape timing can shift when a condition is first seen.
The released rules consider a collection old only when its age is greater than 180 seconds. A matching condition must then remain active for two minutes before the alert fires. A pending alert is not yet a firing alert.
- 10:00 — A valid collection reports healthy. Its file is published and scraped. — A recent healthy report—not proof of protection or containment.
- 10:01 — The next collection never publishes. The exporter continues serving the 10:00 file. — Scrapes still succeed. A panel using only the health value can stay green.
- 10:03 — The report is exactly 180 seconds old. — The strict > 180 stale condition is not yet true.
- 10:03:30 — The report is 210 seconds old. — The stale alert enters its two-minute pending period. Health alerts are suppressed because the report is too old.
- 10:05:30 — No new file has arrived. — The stale alert fires. It identifies stopped collection or failed publication, not the application's present health.
- 10:06 — Collection resumes, but the doctor query fails. A query-invalid file is successfully published and scraped. — The file is fresh, so the stale condition clears. The query-failure condition starts pending. This is not recovery to healthy.
- 10:08 — Fresh query-invalid collections have continued each minute. — The query-failure alert fires after two continuous minutes.
- 10:09 — A valid healthy report is published and scraped. — The query-failure condition clears at the next evaluation. A recent healthy report is available again.
The important transition is at 10:06. Collection has recovered; the health query hasn't. One warning disappearing doesn't mean the whole path recovered. Equally, if a collector publishes a valid but unhealthy report, the appropriate signal is reported-unhealthy—not query failure.
These times describe rule state, not when a person receives a notification. Notification routing and delivery are separate from evaluating the rule.
Missing is not unhealthy—and neither is unreachable
If the exporter scrape fails, start with that failure. In this rule set, up=0 starts the exporter-down alert's two-minute pending period and suppresses the other Eggcracker alerts. A previously firing stale or unhealthy alert may disappear because its exporter gate is no longer satisfied. Don't read that disappearance as a healthy result.
If the exporter is reachable but the expected collection timestamp is absent, that's a different investigation: check whether the file was ever created, whether the collector directory is correct and readable, and whether collection or parsing failed. The alert identifies missing evidence; it doesn't identify the root cause.
Keep the expected target in your monitoring inventory. Otherwise, removing the target can also remove the signal you needed to notice its absence. Eggcracker's rules match opted-in targets by job and instance, so one reporting host doesn't stand in for another. They also flag timestamps more than 60 seconds in the future and suppress health/query alerts when freshness is uncertain. They do not monitor Prometheus's own availability.
Make the next check small
Before trusting a green panel, put three facts next to it: whether the exporter scrape succeeded, the age of the collection, and whether that collection produced a valid health report. Treat missing or unusable evidence as unknown, not as a healthy default.
If you already operate a supported Eggcracker installation and an existing same-UID node_exporter, the pinned monitoring guide provides a concrete path. Read its directory, ownership and locking requirements before collecting once; it doesn't install a scheduler or change permissions. New output files are mode 0600, so a differently owned exporter isn't covered by this integration.
This is a read-only report of supervisor health, not a measurement of containment effectiveness, independent authentication or evidence of adoption. A recovered collection tells you that the reporting path is working again. It doesn't tell you what happened while you couldn't see it.
Start with one panel you already use: can you tell a new healthy report from a newly scraped old one?