ALL-TEST0002 · flaky_test_quarantine_debt
Count quarantined flaky tests without timely owned remediation.
This is a deterministic rule for all languages. Read its implementation.
Definition
Section titled “Definition”Count quarantined tests that exceed the age limit, lack a required owner, recur after claimed repair, or have no remediation evidence. This complements flaky-test rate without treating reruns or quarantine as a fix.
Evidence
Section titled “Evidence”Findings retain test identity, quarantine date, owner, outcomes, recurrence, and repair status. The value is the number of quarantined tests without timely owned remediation.
Exceptions
Section titled “Exceptions”A bounded quarantine may remain during an active incident when ownership and next action exist.
maximum_age_days is how long a quarantine may last before it counts as debt, and setting
require_owner to false accepts a quarantine nobody has been assigned, which is worth doing
only while another record carries the ownership. Python evidence recognizes explicit
pytest.mark.flaky, pytest.mark.quarantine, and pytest.mark.quarantined decorators. Their
since, owner, remediation, and recurred_after_repair keywords carry the lifecycle
evidence. A missing since is unknown age and therefore debt rather than a new quarantine.
Examples
Section titled “Examples”Two old quarantines and one ownerless quarantine produce 3. A three-day quarantine with an
owner and active repair does not count.
References
Section titled “References”- Cites “The Google Testing Blog”, flaky tests
- Cites “Flaky Test Detection and Management at Microsoft”, arXiv 2212.00908
- Cites “pytest-rerunfailures documentation”