Key takeaway: A detection that fires on nothing is indistinguishable from a detection that is broken. Without deliberate testing, coverage decays invisibly while the dashboard looks healthy.
The Silent Failure Mode
Someone writes a rule matching a suspicious process pattern. It works. Eighteen months later an agent upgrade renames a field, the rule matches nothing, and no alert is generated.
Nothing indicates a problem. The rule is enabled, the query is valid, and zero results is the normal state for a good detection. The gap persists until an incident review discovers that the technique was used and nothing fired.
This is the central operational risk in detection work, and it is why treating rules as configuration rather than as code fails.
Applying Software Practice
Detections have the properties of code and benefit from the same handling.
Version control. Rules in a repository with review history make changes auditable and revertible. A rule modified during an incident without record is a rule nobody can reason about later.
Automated testing. Every rule needs a sample event that must match and a benign event that must not. Running those on every change catches field renames, logic errors and over-broad conditions before deployment.
Continuous validation. Periodically emit synthetic activity matching each rule and confirm the alert fires end to end. This is the only mechanism that detects a pipeline break, because it exercises collection, parsing, storage and alerting together rather than testing the query in isolation.
Documented intent. Each rule should state which technique it detects, what a true positive looks like, and what the analyst should do. A rule that fires with no context wastes the responder’s time reconstructing the author’s reasoning.
Measuring Quality Honestly
| Metric | What it reveals |
|---|---|
| True positive rate | Whether the rule is worth its noise |
| Alerts per day per rule | Whether it is operationally sustainable |
| Time from event to alert | Whether detection is fast enough to matter |
| Technique coverage | Where the gaps are |
| Rules with passing tests | Whether the suite still works |
A rule generating forty alerts daily with no true positives is negative value — it consumes attention and trains analysts to dismiss its category. Deleting it improves the programme.
Coverage mapped to a technique framework is the strategic view. It answers where to invest next, which raw rule counts cannot.
Building Rules That Age Well
Prefer behavioural conditions over specific artefacts. A rule matching a particular filename breaks when the attacker renames the file. A rule matching an office application spawning a shell survives, because the behaviour is inherent to the technique rather than to one implementation.
Alert on aggregate anomalies where individual events are unremarkable. A single failed authentication means nothing; a shift in the endpoint-wide failure ratio means something. These rules are more durable because they describe outcomes rather than signatures.
Tune by allowlisting specific known-good sources rather than by broadening the condition. Widening the exclusion to silence noise removes coverage; naming the legitimate management server preserves it.
The Bottom Line
Keep detections in version control with tests that run on every change, validate the full pipeline with synthetic activity on a schedule, measure true positive rate per rule and delete what fails, and prefer behavioural conditions that survive an attacker changing filenames.
