Back to Blog
    purple teamdetection engineering

    37 Campaigns of Attacking Our Own SOC

    ·13 min read

    37 Campaigns of Attacking Our Own SOC

    We started running attack campaigns against our own stack in late January. Thirty four campaigns later, we have something most security vendors don't: a longitudinal record of what our detection stack actually catches, what it misses, and how those numbers change over time when you keep swinging at yourself.

    This isn't a product announcement or a case study about one campaign. We've published those already. This is what happens when you attack your own SOC continuously for a month straight and track every result.

    The Detection Curve Nobody Publishes

    Security vendors publish MITRE ATT&CK heatmaps. Green cells everywhere. The implication is clear: we detect everything.

    Nobody publishes the graph of how they got there. Because the graph is embarrassing at the start.

    Here's ours.

    Our early campaigns were infrastructure validation. Campaign 2 tested 6 Kali tools against Windows targets. Detection rate: 50%. Three out of six. We had world class endpoint telemetry, hundreds of Sigma rules deployed, and we were missing half the attacks a basic Kali install could throw at us.

    Campaigns 3 through 5 expanded into impacket, payload variants, and the broader tool suite. The detection rate didn't improve much. We were finding the same structural problem over and over: we had rules, but they were written for textbook versions of attacks. Real tool output looks different. Field values don't match what the rule author assumed. Log formats shift between Sysmon versions. Rules that "should" work on paper silently fail in production.

    Campaign 10 was the turning point. We made a deliberate push for full MITRE coverage and got there: 25+ techniques tested, 100% detected. But that number hides something important. We achieved 100% by writing rules for the specific attack variations we tested. We hadn't yet learned that coverage is only as good as the variation set behind it.

    The Middle Campaigns: Sysmon, Kill Chains, and Credential Theft

    Campaigns 11 and 12 broke us in a different way. We ran a full telemetry validation across five event types: Process Creation, Image Load, File Create, Registry, and DNS Query. The goal was to confirm that every event type was flowing through the pipeline correctly and that our rules were matching against the fields they claimed to match.

    Several weren't. Rules that looked correct on paper were matching against field values that didn't exist or had shifted. They'd been deployed for weeks, passing every review, and catching nothing. That's a category of problem that doesn't show up in a coverage spreadsheet. The spreadsheet says "rule exists for T1003.001." It doesn't tell you that the rule hasn't matched a single event since it was deployed.

    One of the outcomes of those campaigns was building our own Sysmon configuration from scratch. The stock configs and community templates we'd been using weren't capturing what we needed, and the field output was inconsistent enough that rules couldn't reliably match against it. So we wrote a custom config tuned to the event types and field mappings our detection rules actually depend on. That config isn't static. Every campaign that exposes a telemetry gap feeds back into it. When a technique goes undetected because the right event type isn't being captured or a field isn't being logged, the Sysmon config gets updated alongside the detection rule. The config and the rules evolve together.

    Campaign 13 was our first full kill chain simulation. Six phases from initial access through impact. It worked. Every phase detected. But it took 6 complete attack scenarios, 466 engagement logs total across our seven engagement templates, to validate something we'd been claiming in a single cell on a heatmap.

    Campaigns 14 through 17 pushed into LOLBins, process injection, infrastructure techniques, and credential hunting. Each campaign tested 12 to 16 new techniques. Detection rates varied between 80% and 100%. The pattern was consistent: well known techniques with community Sigma rules detected fine. Techniques that required custom telemetry correlation or non obvious field matching required custom rules.

    Campaign 17 produced the finding that stuck with us longest. We dumped NTDS.dit from the domain controller. 48 megabytes of every credential in the domain. The Wazuh agent captured the ntdsutil process creation, the volume shadow copy, the file access. We had the telemetry. But our rules were looking for ntdsutil.exe as the image name, and the attack used a renamed copy. Missed it completely.

    The fix took 20 minutes. The gap existed for months.

    Campaign 18: The One We Published

    We wrote a separate post about Campaign 18. Sixteen LOLBin and evasion techniques. Initial detection: 43.75%. Seven out of sixteen. We built nine new Sigma rules, reran the campaign, and brought it to 93.75% in six days.

    That's the version we published because it's clean and illustrative. What we didn't emphasize is that Campaign 18 was the seventh campaign in a row where we found gaps we didn't know about. Not because our stack is bad. Because the attack surface is genuinely large and the only way to find the edges of your coverage is to keep probing them.

    The Linux Expansion

    Campaigns 21 through 24 moved to Linux and immediately exposed a structural assumption in our rule set. Every rule we had used Windows field paths: filebeat_data_win_eventdata_commandLine, filebeat_data_win_eventdata_image. Linux Sysmon uses a different field structure: filebeat_data_eventdata_commandLine, no win segment.

    We had zero Linux coverage. Not because we hadn't thought about Linux. Because our rule templates were built from Windows examples and nobody had tested them against Linux telemetry.

    Campaign 21 tested 9 Linux persistence techniques: systemd services, cron jobs, rc.local modification, SSH authorized keys. Twenty seven attack logs across the variations. We built 8 new rules. Detection rate after the fix: 100%.

    Campaign 22 went deeper: LD_PRELOAD injection, shared library hijacking. Another 9 techniques, 27 logs, no new rules needed because the Campaign 21 rules were broad enough. 100% detection.

    Campaign 23 was the big one. Thirty four privilege escalation techniques across both Windows and Linux. A hundred and two attack logs. Twenty new Sigma rules. 100% detection. This single campaign added more detection coverage than our first five campaigns combined.

    Campaign 24 covered code injection: process hollowing, APC injection, thread hijacking. Twenty techniques, 48 logs, 27 rules. 100%.

    The Linux campaigns taught us something we should have known earlier. Platform coverage isn't a checkbox. It's a separate field mapping, separate rule logic, separate validation cycle. Claiming "we support Linux" while running Windows rules is the same as claiming coverage you haven't tested.

    The Gap Campaigns

    Campaigns 25 and 26 were deliberately different. Instead of running techniques we expected to detect, we targeted areas where we predicted gaps.

    Campaign 25 ran 15 techniques across 7 kill chain phases where our MITRE heatmap showed the thinnest coverage. Detection rate: 60%. Nine out of fifteen. The six misses were spread across less common techniques: resource development, defense evasion variants we hadn't seen in the wild, and collection methods that don't produce obvious process creation events.

    Campaign 26 went harder. Fifteen techniques specifically chosen to exploit known detection weaknesses. Detection rate: 40%. Six out of fifteen. The biggest finding: Registry Event 13 processing had a critical gap. Techniques that relied on registry modification for persistence or configuration changes were passing through our pipeline without triggering any rules. Not because we didn't have rules. Because the Graylog pipeline wasn't parsing Event 13 fields consistently, and the rules were matching against empty values.

    That 40% number is the worst detection rate in our entire campaign history. It's also the most valuable campaign we've run. Finding the Registry Event 13 gap led to a pipeline fix that improved detection across dozens of techniques simultaneously.

    The Perfect Streak

    Campaigns 32 through 35 hit something we hadn't seen before: four consecutive campaigns with 100% detection.

    Campaign 32 tested 15 lateral movement techniques. Campaign 33 hit command and control. Campaign 34 covered exfiltration, including cloud exfiltration paths. Campaign 35 tested persistence across multiple platforms.

    Each campaign added 15 to 16 new techniques. Each achieved 100% detection. The difference between these campaigns and our early ones wasn't that we chose easier techniques. Campaign 34 included DNS exfiltration, HTTPS data staging to cloud storage providers, and covert channel techniques that require Zeek network correlation on top of endpoint telemetry. These are hard detection problems.

    The difference was compound improvement. By Campaign 32 we had:

    • Three detection layers working in parallel (Wazuh endpoint rules, Graylog Sigma pipeline, Zeek network correlation)
    • Linux and Windows field mappings both validated and maintained
    • A regression testing system that caught rule breakage within hours instead of months
    • 932 Sigma rules, each one written because a campaign exposed the gap it fills

    That last point matters. We didn't import a community rule pack and call it done. Every one of those 932 rules exists because a specific campaign demonstrated that without it, a specific attack technique goes undetected. Every rule has a test case. Every test case has a campaign number attached to it.

    Where the Time Actually Goes

    People assume most of the work is executing attacks. It's not. Attack execution and telemetry capture are the smallest slice. Recorded attack logs are reusable. Once a technique is captured, it enters the library permanently and gets replayed in every future regression run. The upfront capture costs time. Replaying it costs seconds.

    The bulk of the effort is rule development and pipeline work. Roughly half of total campaign hours go to writing and tuning detection rules. Another quarter goes to pipeline fixes: field mapping corrections, parser adjustments, stage ordering in the Graylog pipeline. The remaining quarter splits between attack execution, telemetry validation, and documentation.

    The honest breakdown of those 932 rules would disappoint anyone expecting 932 pieces of sophisticated behavioral correlation. Most of them are straightforward. A field match on a process name combined with a command line substring. A parent/child process relationship check. A registry path pattern. Individually, each rule is simple. The value isn't in the complexity of any single rule. It's in the coverage of the set, and the fact that every rule has a validated test case proving it fires when the attack it covers actually happens.

    The rules that require genuine multi event correlation, things like detecting C2 beaconing by combining network connection frequency with process lineage, or catching lateral movement by correlating authentication events across hosts, make up maybe 10 to 15 percent of the total. Those are harder to write, harder to tune, and more prone to false positives. They're also the rules that catch the techniques simple field matching can't reach.

    We'd rather have 800 simple rules that work than 200 complex rules that look impressive in a demo and break in production.

    Where Coverage Plateaued

    After 37 campaigns, some categories of technique are structurally harder to detect than others. Not impossible. Harder.

    Resource development is the weakest tactic in any detection stack, including ours. Techniques like acquiring infrastructure, compromising accounts for later use, and developing capabilities happen outside your network. No amount of endpoint telemetry catches an attacker registering a domain. Detection here depends on threat intelligence integration, and threat intel has latency. The IOC doesn't exist until someone reports it.

    Collection techniques that don't involve file system operations are difficult. Clipboard monitoring, screen capture through legitimate APIs, and input capture through accessibility features produce minimal telemetry. You can detect the tools that implement them, but a custom implementation using standard OS APIs looks identical to legitimate software.

    Defense evasion is an arms race by definition. Every campaign finds new evasion variants. Campaign 37 tested 15 defense evasion techniques and caught 14. The one miss was a process injection variant using a syscall method that bypasses the usermode hooks Sysmon relies on. The detection required adding a Zeek based network correlation rule for the C2 callback that followed the injection. Endpoint alone wasn't enough.

    These aren't failures. They're the natural boundaries of what each detection layer can see. Knowing where those boundaries are is the entire point of continuous testing.

    What Comes Next

    Thirty seven campaigns in, we have a clear picture of where the ceiling is and what it would take to raise it.

    Kernel level visibility is the biggest unlock available to us. Usermode telemetry has blind spots that attackers can exploit by design: direct syscalls, callback based execution, memory only payloads that never touch disk. Moving instrumentation closer to the kernel closes those gaps. eBPF on Linux hosts and ETW with deeper memory correlation on Windows are the paths forward. Neither is a drop in replacement for what we run today. Both require new rule logic, new field mappings, and new validation campaigns to prove they work. That's fine. We have the campaign infrastructure to validate them the same way we validated everything else.

    Fileless and memory resident techniques are the frontier after that. Detecting an attack that lives entirely in process memory, never writes to disk, and communicates through legitimate channels requires correlating signals across multiple data sources in near real time. Endpoint telemetry alone won't cut it. The detection has to combine memory forensics artifacts, behavioral baselines, and network layer correlation into a single verdict. We're building toward that, but it's a harder engineering problem than anything a Sigma rule solves.

    There's also a category of ATT&CK that will always depend on external intelligence rather than internal detection. Resource development, supply chain compromise, and pre exploitation reconnaissance happen outside your perimeter. No amount of endpoint or network instrumentation detects an attacker buying a domain or compromising an upstream vendor. Coverage here comes from threat intel feeds, industry sharing, and proactive hunting against known adversary infrastructure. We invest in that layer, but we're honest that it operates on different timelines and confidence levels than real time detection.

    Roughly 15 to 20 percent of the ATT&CK matrix will stay in that harder category for the foreseeable future. The question isn't whether you can detect everything. It's whether you know which 15 to 20 percent you can't detect, and whether you have compensating controls in place for those gaps. After 37 campaigns, we know exactly which techniques fall in that bucket and what we're doing about each one.

    What the Numbers Actually Mean

    After 37 campaigns:

    • 244 MITRE ATT&CK techniques tested with real attack logs
    • 3,278 total attack simulations executed
    • 932 Sigma rules in production, each one validated
    • 4 consecutive 100% detection campaigns in the latest stretch
    • The worst campaign scored 40%, and it was the most useful one we ran

    The detection curve across 37 campaigns looks like a hockey stick drawn by someone with shaky hands. It doesn't go cleanly up and to the right. It dips when we test hard things. It spikes when we fix the gaps those tests expose. It dips again when we deliberately probe weak spots. Then it climbs again.

    That uneven line is what real detection coverage looks like. The smooth green heatmap your current vendor shows you is what marketing looks like.

    What Continuous Self Attack Changes

    Running 37 campaigns changed three things about how we operate.

    First, we stopped trusting rule counts. Having 3,000 rules deployed means nothing. Having 932 rules that each fire correctly against a documented attack variation means something. We'd rather have the smaller number with the test cases.

    Second, gaps became routine. Finding a gap in Campaign 2 felt like a problem. Finding a gap in Campaign 26 felt like Tuesday. The process of test, find gap, build rule, retest is the process. It's not a sign that something is broken. It's the system working.

    Third, regression became automatic. When we change a Sysmon configuration, update a pipeline rule, or migrate a SIEM component, we can replay the full attack library and know within minutes whether anything broke. Before this, configuration changes were followed by hope. Now they're followed by data.

    This is the advantage of attacking yourself continuously. Not that you achieve perfect detection. Nobody does. The advantage is that you know exactly where your detection boundaries are, you have a measured plan for pushing them outward, and you can prove it to anyone who asks.

    Every green cell on our MITRE matrix has a campaign number next to it. Every campaign number has attack logs behind it. Every attack log was generated from a real technique executed in a real lab environment.

    That's 37 campaigns of receipts. We're not stopping.