Skip to main content
{ Reporting and Compliance / Security Operations }

Splunk Detection Posture Report

Delivers a weekly report on Splunk ingestion gaps, correlation searches that are failing, skipped, or blind, and anything in the deployment that needs an admin, from silent forwarders to license use.

What this agent does

This read-only agent checks the health of a Splunk Enterprise Security deployment every week. It finds indexes, sourcetypes, and hosts that stopped sending data. It flags forwarders with no connection to the indexers, from the tcpin metrics in _internal. It finds correlation searches that are disabled, skipping, failing, or searching sourcetypes and data models with no data, and data models whose acceleration has fallen behind. It summarizes the notable events and risk events raised since last week. It flags anything that needs an admin, such as license use close to the entitlement.

The challenge

A correlation search fires only when its data arrives and its scheduled run completes. A forwarder loses its outputs config and a whole host class goes dark. A sourcetype changes name after an upgrade and every correlation search that used the old name returns nothing. Scheduled searches skip under load and nobody reads the scheduler log. Incident Review gets quieter and nobody asks why.

The solution

The agent checks ingestion, search health, and detection activity in one pass each week and compares them with the previous week. It names the data that stopped, the searches that cannot fire, and the fix for each. Provides a weekly briefing a detection engineer can read in a few minutes.

Workflow

  1. 01

    Check ingestion

    Compare the latest event time per index, sourcetype, and host with the window, and check forwarder connections and license use.

  2. 02

    Check search health

    Read each correlation search's enabled state, schedule, skipped and failed runs, and the sourcetypes and data models it depends on.

  3. 03

    Check activity

    Summarize notable events raised, closed, and suppressed, and risk events raised, since last week, by correlation search.

  4. 04

    Report

    Publish what changed and what needs action, with the fix for each item.

Agent template 

# Splunk Detection Posture Report

## Measurable outcomes

Every week, the detection engineer knows which data stopped arriving, which correlation searches cannot fire, and what needs action before a detection is missed. Track the silent sources, the unhealthy searches, and the open action items on every run.

## Procedure

Once a week, for the indexes I set, read the latest event time per index, sourcetype, and host. Flag each one with no events in the last 24 hours, unless I set another window or mark it as expected to be quiet. Compare with the previous week to catch sources that fell silent and sources that are new. Flag forwarders with no connection to the indexers in the window, from the tcpin metrics in _internal. Compare daily license use with the entitlement and flag anything above 80 percent or any violation. For each enabled correlation search in Enterprise Security, read its schedule and the indexes, sourcetypes, and data models its search references. On Enterprise Security 8, read detections and findings in place of correlation searches and notable events. Read skipped runs and their skip reason from scheduler.log, and failed runs from the job's error messages. Flag a search as blind when a sourcetype or index it depends on is silent. For a search that uses a data model, flag it as blind when the data model holds no events from a sourcetype it held last week, even when the index is not silent. Flag a search that skipped or failed in more than 5 percent of its scheduled runs, unless I set another threshold. Flag data models whose acceleration is behind by more than an hour or is disabled. List correlation searches that were disabled or changed since last week, with who changed them. Summarize the notable events raised, closed, and suppressed since the last report, by correlation search. For a search whose only action is a risk modifier, count risk events, not notable events. Flag searches that raised nothing in the last 30 days and at least one in the 90 days before. Compare everything with last week's report, and lead with what changed. Give every action item the specific fix. Report a check it cannot run as not checked, never as fine.

## Requirements

It needs read-only Splunk access to search the indexes in scope, the internal, audit, and risk indexes, the saved searches and correlation searches in Enterprise Security, data model status, forwarder status, and license use, and nothing more. It never changes searches, inputs, indexes, or notable events.