How it works

This repository treats threat-detection rules like software: written in version control, reviewed, tested against real attacker telemetry, scored for quality, and blocked from merging when they fall below a bar they declare for themselves. This page is the method behind every number on the coverage dashboard — what is measured, how it is measured, and what it deliberately does not claim.

It is written on two levels at once. The blue blocks explain each idea in ordinary language with no security background assumed; the text around them is the precise version. Read either one, or both.

The problem Vocabulary Where this started The pipeline Two test layers The metrics Thresholds CI gates Design decisions Honest limits Repo map FAQ

01The problem this solves

In plain English

Windows computers write down almost everything that happens on them, in files called event logs. When an attacker breaks in, they leave traces there. A detection rule is a written recipe that says: if you ever see this pattern in the logs, shout.

Anyone can write such a recipe. What almost nobody does is prove theirs works — that it catches the attack and stays quiet during ordinary Tuesday-afternoon admin work. This repository is a set of rules plus the machinery that produces that proof, automatically, every time anything changes.

In most organisations detection rules live inside a SIEM's web console: edited in a text box, deployed by whoever was on shift, with no history, no tests and no measurement. Nobody can answer the two questions that matter — does this rule still fire on the attack it was written for, and how much noise does it cost the analyst reading the queue.

Detection-as-Code is the answer software engineering already found: put the logic in git, make every change reviewable, and gate it on automated tests. This repo is a working implementation of that idea over 20 rules I wrote myself, plus the tooling that keeps them honest.

The claim being made

Not "I can write a Sigma rule." Anyone can. The claim is: every rule here is mapped to adversary behaviour, proven to fire on a real attack capture, measured for false positives against benign look-alikes, and continuously re-checked — and where a rule is bad, the number saying so is published rather than hidden.

02Vocabulary

Nothing below this point needs any term that is not defined here.

Detection rule
A written condition over log data that raises an alert when it matches. "Shout if a program starts with a scrambled command line" is a detection rule.
Sigma
A vendor-neutral YAML format for writing those rules. You write the logic once and compile it to whichever platform you run — Splunk, Sentinel, Elastic. Like writing a recipe in a universal language and having it auto-translated into any kitchen's local terms.
EVTX
The binary file format of Windows event logs. Public repositories publish EVTX captured while real attacker tooling ran, which is what makes it possible to test a rule against genuine telemetry rather than something I invented.
Event ID
Windows numbers every kind of event. 4688 = a program started. 1102 = the security log was cleared. 4697 = a service was installed. Attackers trigger these too, which is why they are worth watching — and why matching on the number alone is not a detection.
MITRE ATT&CK
The industry catalogue of adversary behaviour — tactics (the goal, e.g. Credential Access) and techniques (the method, e.g. T1003.001 LSASS Memory). Tagging rules with technique IDs is what turns a pile of rules into a coverage map. This repo validates tags against release v19.2.
False positive
The rule shouted and nothing bad had happened. This is the problem in security operations: a rule with too many false positives gets ignored, and an ignored rule might as well not exist.
Hayabusa
A fast Sigma engine that runs rules directly against EVTX files. Here it is the test harness: pytest hands it a rule and a captured attack and asserts on whether it hit. Pinned at v3.9.0.
pySigma
The Python library that parses Sigma and compiles it to backends. It does the conversion to Splunk and KQL, and — through its SQLite backend — the scoring of rules against labelled events.
CI
Continuous integration: a robot on GitHub's machines that re-runs every check on every change. If anything breaks it goes red. It is what stops "it worked on my laptop".

Four counts describe every rule's performance, and everything else is derived from them: TP (malicious event, rule fired), FP (benign event, rule fired anyway), FN (malicious event, rule stayed silent — a miss), TN (benign event, rule correctly silent).

03Where this started

In plain English

This repo did not begin in the state described on this page. It began as a normal portfolio project that looked fine from the outside, and was then audited on purpose, without mercy. Six things were wrong, and every one of them is the kind of thing that quietly makes a security project a decoration rather than a control.

All six are fixed. They are written down here rather than deleted from the history, because the interesting part of the project is not that it is tidy now — it is what it took to find out that it wasn't.

  1. The tests could pass while testing nothing. The suite needed an external engine. When that engine was missing the tests did not fail — they skipped, and the run reported success. The repository's single biggest claim was guarded by nothing at all.
    A smoke alarm that reports "all clear" when its battery has been removed.
  2. Nobody was measuring whether the rules were any good. There were 16 test cases. 15 asked "here is an attack, does it fire?". One asked "here is something harmless, does it stay quiet?" — and that one used a different kind of log entirely, so it proved nothing. There was zero evidence about false positives. For detection work that is a chef who never tastes the food.
  3. Four of the fifteen rules were not detections. They alerted whenever a given event ID appeared, with no further logic. That is not detecting anything; it is forwarding an event and making a human decide.
  4. The quality checker was switched off. Sigma's own linter was running, reporting 20 problems (two serious), and the build was configured to ignore all of them and pass.
  5. The supporting code had no safety net. ~800 lines of Python with no tests, no type checking, no dependency locking — in a repository whose entire pitch is treating security content like professional software.
  6. The compiled output would not have run anywhere. Rules were auto-translated to Splunk and checked for grammar, but not for whether the result was usable: wrong field names, and no statement of which logs to search. Grammatically valid nonsense.

What changed

 at the auditnow
Test suite15 tests, all silently skippable 119 tests; a skipped detection test fails the build
Rule qualityunmeasured 172 labelled events, every rule scored, thresholds enforced
Rules alerting on
100% of benign traffic		4, unnoticed 2, both with a written justification
Sigma linter20 problems, ignored 0 problems, and it blocks the build
Python toolingno tests, no types, unlocked deps packaged CLI, strict typing, locked and reproducible
Compiled querieswould not have run all 20 bound to a real source, checked in CI
Published numberstyped by hand generated; the build fails if they go stale

Things measurement found that reading could not

Two examples, both invisible until there were numbers.

A real evasion. PowerShell accepts -e as an abbreviation of -EncodedCommand. The rule looked for the longer spellings only, so powershell -e <blob> walked straight past it. Fixing it moved recall from 0.80 to 1.00 and cost one false positive out of seven benign cases — and the harness blocked the change until that trade was written down.

A rule that had silently stopped running. Writing a rule's condition across multiple YAML lines is accepted by two of the three engines involved and rejected by the third, which reports only "1 parsing error" without naming the rule. The rule stops running and nothing says so. There is now a check for exactly that.

04The pipeline

In plain English

Six stages take a rule from an idea to the published map. Each one can stop the whole thing, and one command runs all of them the same way the robot does — so "it works on my machine" and "the build is green" mean the same sentence.

01Authordetections/*.yml

A Sigma rule with required metadata and ATT&CK tags. detkit probe runs the draft against real samples before it is finalised.

02Gatedetkit validate

Metadata, tag vocabulary pinned to an ATT&CK release, and a fixture or a declared exemption. Structural problems die here.

03Prove it firespytest

Hayabusa runs the rule against pinned public EVTX. Real channel, real field names, real attacker artefacts.

04Score itdetkit eval

The rule is compiled to SQL and run over hand-labelled malicious and benign events. Precision, recall and FP rate fall out of the result.

05Compiledetkit convert

Splunk SPL and Microsoft XDR KQL, with every query checked for a bound source=.

06Publishdetkit dashboard

Coverage map, Navigator layer, this site and the README table — all regenerated and diffed, so published numbers cannot go stale.

Why the order matters

Each stage assumes the previous one held. There is no point measuring the false-positive rate of a rule that does not parse, and no point publishing a coverage map built from rules that failed their tests. detkit ci stops at the first failure for the same reason.

05Two test layers, two different questions

In plain English

There are two completely separate ways a rule is checked here, because there are two different ways a rule can be wrong.

The first replays a real recorded attack and demands the rule fire. That catches a rule written against logs the author imagined rather than the ones Windows actually produces — the classic beginner mistake, and one that never shows up in your own tests.

The second feeds the rule hand-written examples, some attacks and some deliberately similar harmless activity, and counts what it does. A real attack recording cannot do this: it has no harmless counterpart inside it to stay quiet on.

 EVTX fixture testsLabelled-event scoring
Question Does the rule fire on a real recording of this attack? Does the rule tell malicious apart from benign?
EngineHayabusa v3.9.0, over real EVTX pySigma SQLite backend, over JSON events
Data 18 public captures pinned by URL and SHA-256 172 events hand-written and labelled, each with a stated reason
Catches Wrong channel, wrong field name, a rule that cannot run at all Logic too broad, logic too narrow, a rule that alerts on everything
Misses A rule that fires on the sample and on everything else A rule matching a field Windows never actually emits
Coverage 16 of 20 rules (0 declared exempt) 20 of 20 rules

The two engines are also a cross-check on each other. Two independent implementations of Sigma semantics agreeing on the same rule is worth more than trusting either alone; where they disagreed during development, the disagreement was the bug.

The cloud rules carry a third, weaker tier — and say so

In plain English

The Entra ID (Microsoft cloud identity) rules cannot get the first test. Nobody publishes recordings of cloud sign-in logs, because those logs are real people's names, addresses and devices — it is not a gap somebody forgot to fill, it is what privacy means for that data.

So those rules get the scoring, which is compulsory for them rather than optional, plus a check that every field they mention actually exists in Microsoft's published table layout. A rule that invents a column name cannot be built. What that still does not prove is that the rule fires in a real tenant, and the dashboard labels them schema-verified instead of tested so the difference is visible at a glance rather than buried here.

Concretely: detkit validate fails a non-EVTX rule with no case set, and detkit convert compiles each Entra rule to its real Log Analytics table (SigninLogs, AuditLogs) through the repo's own Azure Monitor pipeline, where the backend rejects any column Microsoft does not publish. ADR 0005 has the argument and the two rejected alternatives — not writing cloud rules at all, and writing them while quietly letting the fixture gate not apply.

06The metrics, and what each one hides

In plain English

Three numbers describe a rule, and they answer different questions:

Recall — of the real attacks, how many did it catch? Missing one is the expensive failure.
FP rate — of the harmless activity, how often did it shout anyway?
Precision — of everything it shouted about, how much was real?

The trap is that precision can be improved without touching the rule, just by writing more attack examples and fewer harmless ones. Recall and FP rate cannot be gamed that way, so they lead and precision trails.

Every rule card on the dashboard carries four figures. Hovering any of them shows this same arithmetic with that rule's own counts filled in.

False-positive rate — the number to read first

FP ÷ (FP + TN) — of the benign events, the share that alerted

Stable against the shape of the dataset: adding or removing malicious cases cannot change it. That is what makes it the honest headline. A rule at 1.00 fires on every benign look-alike it was given, which usually means it is matching an event ID and nothing else.

Recall — did it catch what it was written for

TP ÷ (TP + FN) — of the malicious events, the share that alerted

Also stable, for the mirror reason: it only looks at malicious events. A miss is the expensive failure mode, because an alert that never fires is indistinguishable from a quiet network. Almost every rule here declares a recall floor of 1.00 — missing a true positive is not negotiable, and the trade is paid in false positives instead.

Precision — how much of the alert queue is real

TP ÷ (TP + FP) — of everything that alerted, the share that was real

This is what an analyst actually feels on shift, and it is the one figure here that must be read with suspicion. Precision depends on the ratio of malicious to benign events, and in a case set that ratio is authored, not observed. Real estates run millions of benign events for every malicious one, so a rule scoring 0.80 here would score far lower in production. It is published because it is informative, and ranked third because it is flattering.

Why there is no accuracy figure

(TP + TN) ÷ total — not published anywhere in this repo

Accuracy is dominated by whichever malicious-to-benign ratio was authored. Write nine benign cases and one malicious one, and a rule that never fires at all scores 0.90. It would measure the dataset rather than the rule, which is precisely the failure this harness was built to avoid.

Worked example — posh_ps_obfuscated_payload_execution

5 malicious and 9 benign events are labelled for this rule. It fired on 5 of the malicious ones and 1 of the benign ones.

FP rate  = 1 ÷ (1 + 8) = 0.11
recall   = 5 ÷ (5 + 0) = 1.00
precision = 5 ÷ (5 + 1) = 0.83

How the labelled events are built

Each rule has an evals/<rule>/cases.yml holding events written by hand. Every case carries a why justifying its label, and the loader refuses a case without one — an unexplained label is an assertion, not evidence.

Benign cases must be genuine look-alikes: the same event type as the malicious ones, differing only where the rule is supposed to discriminate. A benign case whose event ID no malicious case uses is rejected, because it cannot exercise the rule's logic, and a set with no benign cases at all is refused outright — precision would be 1.00 by construction. An earlier version of this repo had exactly that problem: 15 of 16 fixtures were "expect fire", and the single negative was a different log channel, so it proved nothing.

07Every rule declares its own bar

In plain English

Rules are not equally hard. Detecting a specific hacking tool should be near perfect; detecting "someone added an antivirus exclusion" cannot be, because a system administrator does exactly the same thing for legitimate reasons and the log entry looks identical.

So each rule states, in its own file, the score it has to reach — and the build fails if it drops below it. Setting an easier bar is allowed. Setting one silently is not: without a written reason, the file is rejected. Every one of those reasons is printed below.

The defaults are strict — precision ≥ 0.90, recall = 1.00, FP rate ≤ 0.10. Anything more lenient is rejected at load time unless it carries a written justification. 16 of 20 rules currently run on a justified lenient threshold.

ruleprecisionrecallFP ratedeclared bar
entra_audit_app_credential_added0.601.000.40p≥0.55 · r≥1.00 · fp≤0.40
Why the bar is where it is: Two benign cases are unwinnable at the Sigma level. One is a provisioning service principal doing legitimately what an attacker does maliciously, which is an allow-list problem. The other is the InitiatedBy serialisation shape described in the rule: separating it needs `tostring(InitiatedBy.app.appId)`, which is platform syntax Sigma cannot express portably. Both are measured here rather than assumed away.
entra_audit_illicit_consent_grant0.801.000.25p≥0.75 · r≥1.00 · fp≤0.25
Why the bar is where it is: One benign case is an approved mail-integration onboarding, which requests exactly the scopes an attacker's app requests. Nothing in the event separates a sanctioned Mail.Read grant from a phished one — that needs the application object ID checked against an approved list, which is enrichment the rule cannot perform on its own.
entra_audit_privileged_role_assigned1.001.000.00p≥0.90 · r≥1.00 · fp≤0.10
entra_signin_legacy_auth_success0.801.000.20p≥0.80 · r≥1.00 · fp≤0.20
Why the bar is where it is: One benign case is deliberately unwinnable: a multifunction printer relaying over Authenticated SMTP is indistinguishable from an attacker using the same protocol with a stolen service-account password. That is an allow-list problem, not a logic problem, and the rule says so in its falsepositives.
posh_ps_defender_tamper0.711.000.40p≥0.65 · r≥1.00 · fp≤0.45
Why the bar is where it is: Sanctioned exclusion management is genuinely indistinguishable from an attacker carving out a staging directory when all you have is the 4104 event. Separating them needs the author identity and the excluded path checked against an approved list, which is enrichment the rule cannot do alone.
posh_ps_obfuscated_payload_execution0.831.000.11p≥0.80 · r≥1.00 · fp≤0.15
Why the bar is where it is: One benign case fires and is expected to: a packaging script that decodes a URL from base64 and then fetches it is, at the level of the script block, the same behaviour as a stager. Separating them needs context this telemetry does not carry (script path reputation, destination host), so the cost is one declared false positive per baselined bootstrap script rather than a weaker rule. Measured FP rate is 0.11 against the sibling rule's 0.67 on the same benign events.
posh_ps_susp_encoded_powershell_scriptblock0.501.000.67p≥0.45 · r≥1.00 · fp≤0.70
Why the bar is where it is: A token-frequency heuristic, not a detection. -join and [Convert]::ToInt are ordinary PowerShell, so two thirds of benign administrative script blocks fire. Demoted to informational and kept as a hunting query: useful for ranking script blocks by suspiciousness, unusable as a page. The alerting tier is posh_ps_obfuscated_payload_execution, whose benign half is this rule's four false alarms — it scores 0.11 FP rate on them by requiring an execution or ingress sink alongside the primitive.
proc_creation_win_certutil_download_decode0.671.000.33p≥0.60 · r≥1.00 · fp≤0.40
Why the bar is where it is: -encode and -decode are legitimate certutil verbs used by PKI enrolment scripts and management agents. Dropping them would raise precision but lose payload-staging coverage, which is the more valuable half of this rule. Keeping them and allow-listing the known callers is the right trade.
proc_creation_win_encoded_powershell0.831.000.14p≥0.80 · r≥1.00 · fp≤0.20
Why the bar is where it is: A deliberate trade, measured both ways. The previous substring version scored precision 1.00 / FP 0.00 but recall 0.80, because it listed -enc/-encodedcommand/-ec and PowerShell accepts any unambiguous abbreviation of -EncodedCommand — so `powershell -e <base64>` walked straight past it. The regex closes that evasion and takes recall to 1.00; the cost is one false positive, a PowerShell command line passing a long base64-shaped value to an inner tool's -e switch. Missing a known evasion is worse than an occasional alert on `docker run -e`, and the residual FP is fixable by baselining the parent process, whereas the recall gap was only fixable by widening the switch match — which is what this is.
proc_creation_win_firewall_disabled_netsh0.751.000.14p≥0.70 · r≥1.00 · fp≤0.20
Why the bar is where it is: "netsh advfirewall firewall set rule ... disable" disables a single rule, not the firewall, but shares both keywords with the malicious pattern. Excluding it needs a negative condition on "set rule", which risks an attacker including that phrase to evade. The false positive is the safer trade.
proc_creation_win_impacket_wmiexec_output1.001.000.00p≥0.90 · r≥1.00 · fp≤0.10
security_win_asrep_roasting0.751.000.25p≥0.70 · r≥1.00 · fp≤0.30
Why the bar is where it is: Accounts intentionally configured without pre-authentication are a real configuration state, not a detection failure. The residual false positive is a known legacy account; the fix is an inventory of those accounts fed in as an exclusion list, not a change to the rule logic.
security_win_dcsync_replication1.001.000.00p≥0.90 · r≥1.00 · fp≤0.10
security_win_eventlog_cleared0.601.001.00p≥0.55 · r≥1.00 · fp≤1.00
Why the bar is where it is: Accepts a 100% false-positive rate deliberately. This rule has no discriminating logic and cannot acquire any: a cleared log carries no evidence of why it was cleared. It stays a high-severity alert because the base rate is near zero — 1102 does not occur on a healthy host — so the absolute alert volume is negligible even at FP rate 1.0. That is a base-rate argument, not a quality one, and it is the only reason this bar is set here.
security_win_kerberoasting_rc4_request0.751.000.17p≥0.70 · r≥1.00 · fp≤0.25
Why the bar is where it is: Measured against a modern AES-default domain, where RC4 is anomalous. The one residual false positive is a legacy appliance still negotiating RC4. In an estate that has not moved off RC4 these numbers do not transfer at all — the bar would have to be re-derived against that environment's own case set.
security_win_local_user_created0.431.001.00p≥0.40 · r≥1.00 · fp≤1.00
Why the bar is where it is: Unlike the 1102 rule, this one cannot lean on a low base rate: account creation is routine on a managed estate, so FP rate 1.0 here means real, daily noise. The rule is therefore demoted to informational and treated as an audit trail rather than an alert. Closing this properly needs correlation (creation followed by privileged-group addition, or creation outside the helpdesk workflow), which is stateful and out of scope for a single-event rule.
security_win_scheduled_task_created0.751.000.25p≥0.70 · r≥1.00 · fp≤0.30
Why the bar is where it is: Scheduling a .cmd or .ps1 script is legitimate administrative automation and is indistinguishable from persistence at the event level. This is already a large improvement on the bare EventID match it replaced (precision 0.43 -> 0.75, FP rate 1.00 -> 0.25); closing the rest needs the task author checked against an approved automation identity.
security_win_service_installed1.001.000.00p≥0.90 · r≥1.00 · fp≤0.10
security_win_user_added_to_privileged_group0.601.000.50p≥0.55 · r≥1.00 · fp≤0.55
Why the bar is where it is: Privileged-group additions are performed legitimately by helpdesk and provisioning workflows, and the event carries nothing that separates a sanctioned grant from an attacker's. The discriminator that would work is the actor: additions by accounts outside the helpdesk group. That is enrichment against directory data, not something a single-event rule can express.
sysmon_proc_access_lsass_dump0.801.000.20p≥0.75 · r≥1.00 · fp≤0.25
Why the bar is where it is: Defender's scanning engine opens lsass with a read mask as normal operation, and any AV or EDR will do the same. This rule is not deployable without a per-estate allow-list of legitimate accessors; the threshold reflects the rule as written, before that allow-list exists.

Thresholds live in the case files, not in the Sigma rules, so the rules stay portable — a threshold is a property of this harness, not of the detection.

Some rules are hunting queries, not alerts

In plain English

Attackers scramble their PowerShell so it does not read as an attack. The obvious rule is "alert when a script looks scrambled" — and measuring it showed that two thirds of ordinary admin scripts look scrambled too, because joining text together and converting numbers are everyday operations.

Deleting the rule would be wrong: scrambling really is a signal. So it was demoted. It now ranks script blocks for a human to look through — a hunting query — while a second rule pages someone only when scrambling appears together with something that runs or downloads the result. Same signal, two products, two severities.

ruleasksseverityFP rate
posh_ps_susp_encoded_powershell_scriptblock which script blocks are worth a human's attention? informational0.67
posh_ps_obfuscated_payload_execution which ones should wake someone up? high0.11

The second rule's benign cases are the first rule's false alarms, so the pair is held together by its own gate: if the alerting tier ever starts firing on -join building a file path, CI fails. On the real Invoke-Obfuscation capture both rules see the same 66 script blocks; the hunting tier flags most of them, the alerting tier flags the one that also calls iex.

A false-positive rate of 0.67 is a failure for an alert and perfectly reasonable for a hunt, which is why the threshold is a property of the rule rather than a constant. Severity follows what is being obfuscated, not the fact that something is.

08What CI actually checks

In plain English

Every one of the checks below runs on every change, on GitHub's machines, and any single one of them can stop the change from landing. The last row is the unusual one: the robot rebuilds every published chart, table and page from scratch and compares them to what is committed. If they differ, someone changed a rule without refreshing the numbers, and the build fails.

One command runs all of it: uv sync --all-extras && uv run detkit ci, about twenty seconds from a clean checkout. The workflow runs the same steps split across parallel jobs.

gatewhat it enforces
ruffLint over every Python file in the repo, tooling and tests alike.
mypyStrict type checking. No implicit Any, no untyped defs.
yamllintFormatting discipline on the rule files themselves.
sigma checkpySigma's own rule linter, run with --fail-on-issues. It sat at 20 issues while it was advisory; it is only enforced now that the count is 0.
rule disciplinedetkit validate: required metadata on every rule, at least one ATT&CK technique tag, and either a test fixture or a declared exemption. It also pins the ATT&CK release and rejects a folded scalar in a condition: — Hayabusa cannot parse one and silently stops running the rule.
pytestUnit tests over the tooling, plus the EVTX detection tests. A skipped detection test fails the run: the suite used to pass by skipping everything.
detection qualitydetkit eval scores every rule against its labelled events and fails if any rule drops below the bar it declares.
convert + bindingCompiles the corpus to Splunk SPL and Microsoft XDR KQL, and fails if any Splunk query lacks a source=. Syntactic validity is not deployability.
vendored corpusBatch-converts the pinned third-party SigmaHQ corpus and rebuilds its report.
navigator layerRebuilds the ATT&CK Navigator layer JSON.
coverage matrixRebuilds the coverage heat map.
dashboardRebuilds this site.
readme numbersRewrites the measured table in the README from evals/results.json. The README's numbers are the harness's numbers because the harness writes them.
artifact driftEvery generated artifact is rebuilt and diffed against what is committed (coverage/navigator_layer.json, vendored/report.json, evals/results.json, site/index.html, site/about.html, README.md). Generator output is a pure function of the corpus — no timestamps — so a diff means someone changed the rules without regenerating, and the published numbers no longer describe the repo.

The gate that mattered most

The suite originally passed while testing nothing: with no Hayabusa binary present every detection test skipped, and pytest exits 0 on a fully skipped run. CI now installs a checksum-verified pinned binary and treats any skip as a failure. A test suite that cannot fail is worse than no test suite, because it manufactures confidence.

09Design decisions on record

In plain English

Every significant choice is written down with the options that were considered and the reason one won — including the ones that were rejected. Knowing when not to add something is half of engineering, and it is invisible unless you record it.

Other choices worth stating

10Honest limits

In plain English

Everything below is a genuine weakness, written here on purpose. A limitation somebody else discovers reads very differently from one you documented yourself.

What this does not prove

  • The labelled events are synthetic. They are written to be realistic look-alikes, but they are mine. They establish that a rule's logic discriminates; they cannot establish what its volume would be on a real estate.
  • Case sets are small. 172 events across 20 rules. Enough to catch a rule that matches an event ID and nothing else — not enough for a confidence interval.
  • Some rules are irreducibly noisy, and stay. Clearing the event log is one event with no distinguishing fields; its case set scores an FP rate of 1.00 and it is still shipped at high, on a base-rate argument — the behaviour is rare enough in practice that the alert is worth the noise. Others were demoted to informational instead, so they remain available for hunting without paging anyone. Both calls are argued in the case files rather than hidden by deleting the rule.
  • Sentinel's SecurityEvent schema has no PreAuthType column, so the Kerberos rules cannot bind there without EventData parsing. KQL conversion therefore targets Microsoft XDR on the process-creation subset. That is a platform limit, documented rather than pretended away.
  • The cloud rules are not proven against a real tenant. Entra ID telemetry has no public captures, so those four rules carry the schema-verified tier described above: measured logic and a validated query, but no replay. That is a real difference from the Windows rules and it is labelled as one everywhere it appears.
  • Deployment is a static site. Nothing here provisions a SIEM. The rules compile to deployable, source-bound queries; actually running them in an estate is out of scope, and Terraform with nothing behind it would be set dressing.

11Repo map

detections/
The 20 authored Sigma rules, split by telemetry: windows/ and identity/entra/. Everything is gated on these, though not every rule can be gated the same way — see the evidence tiers above.
evals/
Labelled events and declared thresholds, one directory per rule, plus the generated results.json.
tests/
pytest: unit tests over the tooling, the Hayabusa harness, and the per-rule fixture manifests pinning public EVTX by hash.
src/detkit/
The tooling package behind the detkit command — validation, evaluation, conversion, coverage, and the generators for this site.
pipelines/
pySigma processing pipelines: the field maps that bind generic Sigma categories to a platform's real channels and column names.
vendored/
The pinned third-party SigmaHQ corpus and its report. Not authored here, not gated.
coverage/
The ATT&CK Navigator layer and the rendered heat map.
docs/
The detection lifecycle and the ADRs.

Run it yourself: uv sync --all-extras then uv run detkit ci. The pinned test engine is downloaded and checksum-verified on first run; nothing else is needed.

12Questions this invites

Why measure detections at all — isn't a rule either right or wrong?
No. Every detection is a trade between missing the attack and drowning the analyst, and the trade is different for every behaviour. Without measurement that trade is made by whoever wrote the rule, silently, and re-made every time someone edits it. Numbers make it a decision.
Why is FP rate the headline instead of precision?
Because precision moves with the malicious-to-benign ratio of the dataset, and in a hand-authored case set I choose that ratio. FP rate looks only at benign events, so I cannot flatter it by writing more attacks. When a metric can be improved by editing the test data rather than the rule, it should not lead.
You have rules with an FP rate of 1.00. Why ship them?
Because the alternative is worse. Clearing the security event log produces one event with nothing in it to discriminate on — any rule for it fires on every instance, benign or not. What matters is the base rate: it almost never happens legitimately, so the alert is cheap in practice even though it is undiscriminating in the lab. That argument is written into the case file, and rules where the argument does not hold were demoted to informational instead.
Why two engines rather than one?
They answer different questions and check each other. Hayabusa proves a rule works against real Windows telemetry — right channel, right field names. The SQL evaluator scores the logic over labelled events, which real captures cannot do, because a capture has no benign counterfactual in it. Where two independent implementations of Sigma disagree about the same rule, something is wrong and I want the build to say so.
What stops the published numbers going stale?
CI regenerates every artifact and diffs it against what is committed. Generator output is a pure function of the corpus, so a diff means the repo changed and the published numbers no longer describe it. That includes this page.
What was actually hard?
Making failure possible. The first version of this repo passed its build while testing nothing, reported coverage it did not have, and had exactly one negative test case — which was for a different log channel and could never have failed. Most of the engineering since has been removing ways for the build to be green without being true.
Only 20 rules? SigmaHQ has thousands.
A pinned copy of SigmaHQ's Windows corpus is in this repository too, and it is reported separately on purpose. The authored rules are the ones held to every gate: a pinned capture or a mandatory case set, a declared threshold with a written justification, a bound query, an ATT&CK tag that resolves. 20 rules that each carry that evidence say more about how someone works than a thousand copied ones. The corpus is deliberately the smaller, true number.
Is this finished, or abandoned?
Finished, and the boundary is argued rather than implied — ADR 0007 names the two things that were on the list and declines both: more cloud rules of a shape already proven four times over, and a Splunk container replaying the same pinned captures the existing gates already check. What a live SIEM would genuinely add is behaviour under production volume, which is not available here, so shipping one would produce a screenshot rather than evidence. The gates keep running; the corpus is closed.