Skip to main content
{ Security Operations }

Splunk Detection Tuning

Finds the Splunk correlation searches that produce the most noise, reads each one against the notable events it raised, and proposes a specific tuning change with the evidence and what it would have missed.

What this agent does

This read-only agent ranks the correlation searches in Splunk Enterprise Security by the noise they produce. It measures each search's notable event volume, the share closed as false positive or benign, the time analysts spend closing them, and the entities that repeat. It reads the search and the notable events behind the noise, and it proposes one tuning change per search. The change is a filter, a threshold, a lookup-based exclusion, a throttling change, a risk score change, or a notable event suppression. Each proposal shows the events it would have removed and the true positives it would have hidden. A person applies the change.

The challenge

A handful of correlation searches produce most of the notable events, and most of those are closed the same way every day. Analysts know which searches are noisy. Finding the pattern means reading the raw events behind hundreds of notable events. The open queue always comes first, so that reading never happens. A throttle or lookup exclusion written in a hurry removes a real detection along with the noise.

The solution

The agent names the pattern behind each noisy search with the specific field values. It proposes the narrowest change that removes the pattern. It runs the changed search over the window to show what the change would have hidden. It presents every proposal as a candidate and changes nothing itself. Provides a ranked tuning queue where each item is ready for a detection engineer to review and apply.

Workflow

  1. 01

    Measure noise

    For each correlation search, count notable events and risk events in the window, the share closed as false positive or benign, time to close, and repeated entities.

  2. 02

    Rank

    Rank searches by false positive and benign count times median time to close, and take the top 5.

  3. 03

    Read the events

    Read the notable events and their raw events for each search, and find the field values that explain the noise.

  4. 04

    Propose

    Write the narrowest change that removes the pattern, and run it against the window's events to show what it would have removed and hidden.

  5. 05

    Report

    Publish the ranked queue with each proposal and its evidence.

Agent template 

# Splunk Detection Tuning

## Measurable outcomes

Every run produces a ranked list of noisy correlation searches, each with one specific proposal and the evidence for it. Track the notable event volume and the false positive share of each search on every run. On the next run, compare each tuned search's volume and false positive share with the run before its last change.

## Procedure

Each run, review the last 30 days of notable events in Splunk Enterprise Security, unless I set another window. For each correlation search, count the notable events it raised and the share closed with a false positive or benign disposition. Also count the median time to close and the number of distinct and repeated source, destination, and user entities. Report the share of closed notable events with no disposition. Rank on volume alone when that share is above half. Otherwise, rank searches by false positive and benign count times median time to close. Take the top 5, unless I set another number. Skip searches I mark as tuned recently. When a search writes risk events, rank it by risk event volume and by the risk notables it feeds, and propose a risk score or risk object change. For each search, read its SPL, its throttling settings, and its notable events with the raw events behind them. Find the field values that explain the noise. Typical causes are a service account, a scanner's address range, a maintenance window, a hostname pattern, or a threshold that the normal baseline crosses. State the pattern as fields, values, and the share of notable events they cover. Propose the narrowest change that removes that pattern, and show the exact SPL or lookup rows. The change can be a filter in the search, a higher threshold, or an exclusion in a lookup the search already uses. It can also be a throttling change, or a notable event suppression with its search. Run the proposed search against the window's events. Report the events it would have removed and every true positive or open notable event it would have hidden. For each proposal, name what an attacker could do inside the excluded scope without an alert, such as acting as the excluded service account. When no change removes the pattern without hiding a true positive, say that no safe change exists. Never propose a change that hides a true positive without saying so. Never propose disabling a search. Present every proposal as a candidate, and never change a search, lookup, throttling setting, or suppression. When a search's noise comes from a data quality problem, say so and point at the source instead of proposing a filter.

## Requirements

It needs read-only Splunk access to search the notable index, the risk index, and the indexes in scope, and to read saved and correlation searches, their lookups, and notable event suppressions, and nothing more. It never changes searches, lookups, throttling, suppressions, or notable events. Reports quote only the field values a proposal excludes, with raw events, other users, hosts, and alert IDs left out.