active
Defender detection bench
Write the KQL, run the simulated attack, count what else it caught. A detection doesn't ship here until both numbers exist.
- status
- Active — new detections added as scenarios are built
- telemetry
- Microsoft Defender for Endpoint, Defender for Identity, Microsoft Sentinel
- simulation
- Atomic Red Team techniques plus scripted benign activity
- language
- KQL, written and tuned against real schema, not sample data
The question
Most published detection content shows a query catching a technique. It rarely shows what that same query does against a normal week of Defender telemetry — sign-ins from a new device, an admin running a script they run every month, a service account authenticating somewhere slightly unusual. A detection that fires on all of that gets muted within a week, which is worse than not having it, because someone believes it's still watching.
So every detection on this bench gets two numbers: does it catch the technique, and how often does it catch something else.
Setup
The bench has two halves. A scenario generator runs a known technique — via Atomic Red Team where it fits — against the lab tenant's endpoints and identities. A separate noise generator runs plausible benign activity that looks structurally similar: the admin who genuinely does run PowerShell outside business hours, the account that authenticates from a new location because someone travelled. The KQL runs across both windows, and a PowerShell harness records exactly what matched and why.
The benign traffic generator is the part worth the most effort. Writing a detection that catches the attack is the easy half; the near-misses decide whether it survives contact with a real environment.
What gets measured
- True positives against the simulated technique, by variant and by evasion attempt.
- False positives against the benign window, with the specific benign action that tripped each one.
- Query cost across the Sentinel and Defender schemas — a detection too expensive to run on schedule gets turned off regardless of accuracy.
- How the detection degrades when a field is missing, since production telemetry is rarely as clean as a lab's.
Results pending. Detection-by-detection true/false-positive tables land here as each scenario finishes.
Notes so far
- Most of the real work is in the exclusions, not the base query — which means the exclusions need to be documented as carefully as the detection logic itself.
- Identity-based detections age better than endpoint process-name detections. A renamed binary breaks the second kind immediately; a genuine behavior change is much rarer.
- A detection that can't be explained to whoever triages it won't get triaged correctly, regardless of how well-tuned the query is.
Lab telemetry only. Every query here is written and tested against activity generated in the lab tenant. Nothing from any employer or customer environment appears on this site.
Where it goes next
Extending the noise generator to run continuously over days rather than hours — the only honest way to measure a detection that keys on rarity.
Related work in the lab
Every access-package and hybrid-identity change in the sandbox becomes telemetry too — which makes it fair game for this bench.