Back to Blog
    purple teamdetection engineering

    Campaign 18: What Our Own Detection Stack Missed

    ·5 min read

    Campaign 18: What Our Own Detection Stack Missed

    We ran 16 new attack techniques against our own detection stack last month. One slipped through.

    That's the part most security vendors don't tell you. They publish coverage numbers and MITRE matrices without mentioning how they know those numbers are accurate. The answer, usually, is that they don't. They've mapped their rules to technique IDs on a spreadsheet and called it coverage.

    We know our numbers are accurate because we broke them. Then we fixed them.

    How Campaign 18 Works

    Every few weeks, we run a fresh batch of attack techniques through our lab environment. The setup is an isolated Windows domain with a domain controller, file server, and workstation, all running Sysmon with our custom configuration, all reporting to our Wazuh deployment and flowing into Graylog. On the other side, we have a Kali machine on a segmented VLAN with the full attack toolkit.

    Our attack lab captures everything: every process creation, network connection, registry write, and file operation that Sysmon produces during the attack. That telemetry gets packaged as reproducible attack logs in standard Wazuh format. When we run a campaign, we're injecting those logs into our SIEM pipeline and checking whether our Sigma rules fire.

    Campaign 18 covered 16 new techniques, focusing on LOLBins we hadn't fully tested: living off the land binaries that are signed by Microsoft and present on every Windows installation, making them nearly invisible against a background of legitimate administrative activity.

    Most fired immediately. One didn't.

    The Gap: mmc.exe Proxy Execution

    T1218.014 is Microsoft Management Console being used as a signed binary proxy. The attack works like this: an attacker crafts a malicious .msc file, which is the same file type that opens tools like Active Directory Users and Computers or Device Manager. When mmc.exe opens the crafted file, it can execute arbitrary code while appearing in process trees as a routine administrative action.

    Our lab environment confirmed the technique works. mmc.exe opened the crafted file, executed the payload, and established a connection back to our Sliver C2 server. The Wazuh agent captured everything: the process creation event, the network connection, the child process spawn.

    Not a single Sigma rule fired.

    The reason isn't complicated. We had rules watching for mmc.exe spawning suspicious child processes, but they used an exclusion list for common legitimate children (things like dllhost.exe, conhost.exe, and other standard MMC dependencies). The payload we used happened to spawn through a path that matched one of those exclusions, and the rule silently passed it.

    More importantly, we had nothing watching for mmc.exe initiating outbound network connections. Administrators don't use MMC remotely. The binary has no legitimate business making connections to external IPs. That signal was sitting there, uncovered.

    The Fix

    Two rules, both running in our Graylog Sigma pipeline.

    The first closes the child process gap. Instead of an exclusion list of known good children (which grows stale and can be bypassed by choosing a different execution path), we flipped the logic: alert when mmc.exe spawns a child process that isn't one of a small set of explicitly expected Microsoft Management Console binaries. If something unusual comes out of mmc.exe, we want to know.

    rule "Sigma: MMC Suspicious Child Process (T1218.014)"
    when
      has_field("filebeat_data_win_eventdata_parentImage") AND
      contains(to_string($message.filebeat_data_win_eventdata_parentImage), "mmc.exe", true) AND
      NOT (
        contains(to_string($message.filebeat_data_win_eventdata_image), "WerFault.exe", true) OR
        contains(to_string($message.filebeat_data_win_eventdata_image), "conhost.exe", true)
      )
    then
      set_field("sigma_rule_title", "MMC Suspicious Child Process");
      set_field("sigma_rule_level", "high");
      set_field("sigma_mitre_id", "T1218.014");
      set_field("sigma_mitre_tactic", "Defense Evasion");
      set_field("alert", true);
    end
    

    The second rule covers the network connection angle. mmc.exe making any outbound connection is a finding. We pull Sysmon Event 3 (network connection) filtered to source process matching mmc.exe and correlate against our internal IP ranges. Anything going to an external destination is immediately high severity.

    We reran the Campaign 18 logs after deploying both rules. Both fired. The technique is now covered.

    The Part That Doesn't Scale Without Automation

    Running a 16 technique campaign manually takes a few hours: execute the attacks, capture the telemetry, inject it, query the results, document what fired and what didn't, write new rules for the gaps, deploy them, validate. That cycle is manageable monthly.

    It doesn't stay manageable as coverage grows. We're at 244 techniques. A full regression run across all of them after a rule change, a Sysmon config update, or a SIEM migration would take days to do by hand. And it has to happen regularly, not just when someone remembers to schedule it.

    The automation we run handles this. Our recorded attack logs are in a standard format that can be injected in batch. The consensus orchestrator coordinates the validation runs, logs results to an audit trail, and flags any technique where expected detections fail to fire. When we update a rule, we can run a targeted regression against the techniques that rule covers within minutes. When we add a new technique, it immediately enters the rotation and gets validated alongside everything else.

    This is what lets a small team maintain real coverage. Not headcount. The system validates itself.

    What This Means in Practice

    Two things worth understanding about how this affects what we deliver.

    The first is that our detection coverage numbers come with receipts. When we say 244 techniques, that's 244 techniques we've executed in the lab, captured as telemetry, and confirmed produce detections in our stack. The coverage matrix has a test result behind every green cell, not a checkbox next to a rule file.

    The second is that finding gaps is not a failure state. Finding a gap in Campaign 18 and closing it before any client environment was exposed is exactly what the validation loop is supposed to produce. The teams running security programs without this loop don't find their gaps until an attacker finds them first.

    The mmc.exe gap was in our stack for months before Campaign 18. We didn't know because we hadn't tested it. Now we know, and it's closed. That's the cycle.

    The question for any security program isn't whether gaps exist. They do, in every stack, from every vendor, including ours. The question is whether you have a process for finding and closing them systematically, or whether you're waiting for an attacker to run your gap analysis for you.