Observability

Prometheus Alert Rule Debugging Guide

Debug Prometheus alert rules by separating query results, evaluation timing, pending duration, labels and notification delivery. Last updated September 17, 2026.

Debug Prometheus alert rules by separating query results, evaluation timing, pending duration, labels and notification delivery. This reference is written for developers who need practical validation behavior, reviewable rules and safe examples rather than copied snippets with no explanation.

Recommended workflow

Step	Why it matters
Run the expression	Evaluate the exact PromQL at the rule timestamp and inspect returned labels.
Check rule state	A true query remains pending until the for duration completes.
Inspect label stability	Changing labels create new alert identities and reset pending time.
Trace notification routing	After firing, verify inhibition, silence, grouping and receiver delivery.

Starter snippet

PromQL result -> rule evaluation -> pending for -> firing -> Alertmanager route

Review checks

Test rules against recorded fixtures.
Include runbook and owner annotations.
Use severity labels consistently.
Alert on user impact rather than raw noise.

Common mistakes

Changing the query while investigating historical behavior.
Using volatile labels in alert identity.
Testing Alertmanager before proving the rule is firing.

Validation should help users correct input while protecting systems from bad data. Keep syntax checks, product policy, security review and deliverability checks separate.

Related Formalint references

Continue with Prometheus High Cardinality, Error Budget Slo Guide, Application Health Check Guide.