Back to Blog
    detection engineeringexposure management

    Building a CTEM Program That Actually Proves Its Own Detections

    ·10 min read

    Most CTEM implementations stop at vulnerability counts. They aggregate CVEs across your fleet, assign severity scores, and hand you a prioritized list. That is useful, but it only answers one question: what can an attacker exploit? It says nothing about whether you would actually detect the attack.

    We built our CTEM program to answer a harder question: if an attacker uses technique T1059.001 against your environment right now, will your detection stack catch it? And if it does, how fast?

    The Problem with Vulnerability Only Exposure Scores

    A traditional exposure score might tell you that you have 412 vulnerabilities across 7 agents, 12 of them critical. That number matters. But it does not tell you that your Sigma rules have no coverage for container escape techniques, or that your credential theft detections have not been tested since January, or that Scattered Spider's preferred TTPs overlap with three of your coverage gaps.

    Vulnerability scanners measure the attack surface. Detection coverage measures whether you would know about the breach. Threat alignment measures whether your gaps are the ones that matter. You need all three.

    Three Component Exposure Index

    Our CTEM system computes a composite Exposure Index on a 0 to 100 scale, where lower is better. It combines three weighted components:

    Vulnerability Score (35% weight) measures the raw vulnerability burden across your fleet, weighted by exploitation evidence. We pull severity data from the Wazuh Indexer's wazuh-states-vulnerabilities index, then enrich every CVE against two sources: FIRST's Exploit Prediction Scoring System (EPSS) for exploitation probability and the CISA Known Exploited Vulnerabilities (KEV) catalog for confirmed active exploitation.

    Base severity weights (critical 10x, high 5x, medium 2x, low 0.5x) are multiplied by an exploitation factor. A CVE on the CISA KEV list gets a 2.0x multiplier because active exploitation is confirmed. A CVE with an EPSS score above 0.5 gets 1.5x. Above 0.1 gets 1.2x. Everything else stays at 1.0x. Multipliers do not stack; KEV takes priority as the strongest signal.

    The practical effect: a critical CVE on the KEV list contributes 20 weighted points per instance instead of 10. A critical CVE with no known exploit stays at 10. This means two environments with identical severity distributions will have different vulnerability scores based on how many of their CVEs have active exploitation in the wild. The enrichment data refreshes daily, with EPSS scores fetched in batch from the FIRST API and the KEV catalog pulled from CISA's static JSON feed.

    Coverage Gap Score (35% weight) measures the percentage of MITRE ATT&CK techniques that lack full detection and testing coverage. A technique counts as fully covered only when it has a deployed Sigma detection rule AND a attack simulation template AND a validated detection result. Partial coverage (rule exists but untested, or test exists but no rule) counts as a gap. The score is the percentage of tracked techniques without full validated coverage, scaled to 0 to 100. If you track 471 techniques and 244 are fully validated, your coverage gap score is ((471 - 244) / 471) * 100 = 48.2.

    Threat Alignment Score (30% weight) measures how well your coverage maps to what active threat groups actually use. We pull data on known threat groups from public threat intelligence, extract their documented TTPs, and weight each uncovered technique by threat group relevance.

    The relevance weights are manually curated based on how active a group is and how relevant they are to the industries we serve. Scattered Spider gets a weight of 1.0 because they actively target the types of environments we defend. Lazarus Group gets 0.5 because their campaigns are real but less likely to hit our customer base. These are judgment calls informed by operational experience, not ML derived scores. We review and adjust them when the threat landscape shifts.

    When a technique is used by multiple tracked groups, the weights accumulate. If both Scattered Spider (1.0) and FIN7 (0.9) use a technique and you have no detection for it, that gap carries a combined weight of 1.9. The score is the sum of weighted uncovered techniques divided by the sum of all weighted techniques, scaled to 0 to 100. This means two environments with identical coverage percentages can have different exposure scores based on which threats target their industry.

    The formula:

    Exposure Index = (Vuln Score x 0.35) + (Coverage Gap x 0.35) + (Alignment x 0.30)
    

    Validated Coverage, Not Theoretical Coverage

    This is where our implementation diverges from every CTEM tool we evaluated. Most platforms determine detection coverage by checking whether a rule exists for a given technique. If you have a Sigma rule that references T1059.001, the technique is marked as covered.

    We do not accept that. A rule existing is not the same as a rule working. Rules break silently. Field mappings change. Pipeline stages get reordered. A rule that worked in January might fail in March because someone updated a pipeline stage.

    So we validate every technique by injecting real attack logs through the full detection pipeline.

    How Attack Simulation Validation Works

    Our in-house attack simulation library is the core of this. It contains over 3,000 log templates that replicate the telemetry real attacks produce: Sysmon process creation events, registry modifications, network connections, file writes, CloudTrail API calls. Each template maps to a specific MITRE ATT&CK technique and produces logs indistinguishable from actual attacker activity flowing through the detection stack.

    A daily validation cycle selects techniques from a round robin queue and injects their attack simulation templates into the detection pipeline. The logs flow through the complete stack, the same path a real attack would take.

    Then it queries OpenSearch for the injected logs and checks whether a Sigma rule fired. The technique is marked validated only if the detection pipeline actually produced an alert. The schema tracks detection latency per technique, and we are building out the instrumentation to measure the full end to end time from log injection to alert creation. That measurement is straightforward in concept but requires timestamp correlation across multiple pipeline stages, so we are rolling it out incrementally as we instrument each hop.

    A technique has three possible states:

    StatusMeaningRequirements
    Full (green)Proven detection capabilitySigma rule deployed + attack template exists + last validation passed
    Partial (yellow)Incomplete coverageHas one of: detection rule OR attack template, but not validated
    Gap (red)No detection capabilityNeither detection rule nor attack template

    As of today we track 471 techniques. 244 have full validated coverage. 227 have partial coverage, meaning attack simulation templates exist but Sigma rules are still being developed or have not yet passed validation. Zero techniques have no coverage at all, meaning every tracked technique has at least an attack simulation template or a detection rule in the pipeline.

    To be clear about what partial means: those 227 techniques are not defended the way the 244 are. We can simulate the attack but we cannot yet prove the detection fires. Partial techniques have zero proven alert fidelity in simulation, so they contribute fully to the coverage gap score until they pass validation. The dashboard shows partial in yellow specifically so no one mistakes it for full coverage. The gap between 244 and 471 is the backlog, and it drives our weekly detection engineering priorities.

    Post Deploy Validation

    When new Sigma rules are deployed, they are automatically queued for validation with elevated priority. A new rule gets tested within hours, not weeks. If the validation fails, we know the rule is broken before a real attacker tests it for us.

    The Collection Pipeline

    Three automated jobs feed the CTEM system:

    Validation Cycle runs daily. Round robins through the technique queue, injects attack simulations, and measures detection capability. Results persist for trend analysis and gap identification. The resource cost of validation is low by design. Each cycle injects a small batch of technique logs in native Wazuh alert format, appended to the same log path real alerts use. The injected volume is a rounding error compared to normal alert throughput. We chose this approach specifically because it scales without requiring a separate simulation infrastructure or dedicated compute. The validation pipeline is the production pipeline.

    Exposure Snapshot Collection runs daily. Pulls vulnerability data from the Wazuh Indexer, agent status from the Wazuh Manager API, and coverage metrics from the MITRE tracking system. Computes the composite Exposure Index and stores a snapshot. These snapshots power the trend charts on the dashboard, showing whether your exposure is improving or degrading over time.

    Threat Alignment Engine runs on a recurring schedule. Checks threat intelligence for recently active threat groups, extracts their attack patterns, and computes a weighted gap analysis. The output is a prioritized list of the highest weight coverage gaps, ranked by which threat groups use them and how relevant those groups are to your environment.

    What the Dashboard Shows

    The exposure dashboard presents five views:

    The hero section displays the current Exposure Index with a 30 day sparkline, the delta from the previous snapshot, and a breakdown of the three component scores. A score of 32 is green. A score of 65 needs attention.

    Four KPI cards show coverage percentage (full vs partial vs gap breakdown), total vulnerabilities with critical count highlighted, seven day validation pass rate, and total detection rules plus attack test count.

    The trend chart shows the Exposure Index over 30, 60, or 90 days with component scores as separate lines. You can see whether a spike in the index came from new vulnerabilities, detection gaps, or a change in the threat landscape.

    The technique drill down table lists every tracked technique with its coverage status, whether a Sigma rule exists, whether a attack test exists, and when it was last tested. You can filter by tactic, coverage status, or search by technique ID.

    Multi Tenant Architecture

    Every data point is scoped by customer. Exposure snapshots, validation results, and alignment scores are all stored with a customer ID. An MSP can view aggregate exposure across all their customers or drill into a single customer's coverage. Customer admins see only their own data. API key authentication allows automated collection scripts to POST data for any customer.

    The global view (no customer filter) shows the aggregate across all environments, which is what we use internally to track our overall detection posture.

    What This Actually Means in Practice

    Last month we deployed 43 new CloudTrail Sigma rules to detect AWS API abuse. Within 24 hours, the post deploy validation cycle had tested every one of them. 38 passed. Five failed because the field mappings in our CloudTrail pipeline used a different prefix than the rules expected. We fixed the five rules and re-validated the same day.

    Without automated validation, those five broken rules would have sat in the pipeline doing nothing until a real CloudTrail attack happened and we missed it.

    Our seven day validation pass rate runs around 82%. The 18% that fail are not wasted. They are detection gaps caught before they matter. Each failure generates a work item: fix the rule, update the field mapping, adjust the pipeline stage. The pass rate trends upward over time because failures drive immediate remediation.

    The threat alignment component changed how we prioritize detection development. Before, we built rules based on MITRE coverage gaps from top to bottom. Now we build rules based on which gaps overlap with active threat groups targeting the industries we serve' industries. A technique used by Scattered Spider with no detection coverage gets priority over a technique in the MITRE matrix that no active group has used in the last six months.

    The Numbers

    MetricValue
    Techniques tracked471
    Full validated coverage244
    Partial coverage (in progress)227
    Coverage gaps0
    Sigma detection rules3,209
    attack simulation templates3,278
    Snapshot collectionAutomated, daily
    Validation cycleDaily rotation, continuous coverage
    Detection latency trackingPer technique, instrumentation in progress
    Threat alignment refreshWeekly

    Every number in that table is computed from actual detection pipeline behavior, not from static rule counts or theoretical mapping documents. When we say 244 techniques have full coverage, we mean 244 techniques were attacked in simulation and the detection stack caught them.

    That is the difference between claiming coverage and proving it.