How SIEM/XDR Tuning Reduces Alert Fatigue

Here’s the hard truth about monitoring solutions: Most companies haven’t properly configured their SIEM/XDR system. Logging millions of events per day may seem productive. But what good does it do if an IT team is overwhelmed with alert fatigue and learns to ignore most of notifications they get? 

“The basic rules in your SIEM may be functioning, but they often aren’t functioning well,” says Chief Technology Officer Steve Healey. Read on to learn how trained SOC analysts leverage SIEM/XDR tuning to turn out-of-the-box rules into meaningful tools for reducing noise and alert fatigue while stopping attacks before they gain a foothold. 

The Problem with Out-of-the-Box SIEM Rules 

All SIEM solutions come pre-loaded with a large number of rules. Alert fatigue happens because standard rules can’t possibly work equally well in every environment. “The idea behind those rules is solid, but they’re generic,” Steve says. “The execution will lead to an enormous number of false positives and alert fatigue. You’ll have to tune the rules with additional logic specific to your business to create exceptions without impeding the rule’s original intent.” 

Beyond SIEM vendors, many other tech vendors regularly issue new detection rules to close gaps discovered in their own products. Many of those rules also generate a flood of false positives. HBS’s SOC analysts (who have managed multi-tenant SIEM/XDR solutions for more than a decade) review each new rule’s goal and customize it for every customer’s environment. “We don’t just disable ineffective rules,” Steve says. “We take the core intent of the rule and build it out to get high-fidelity results.” With this kind of tuning, HBS recently turned 266 million monthly security events in one client’s environment into just 41 alerts sent to the client’s IT team. 

Reducing Alert Fatigue 

The real art of creating SIEM/XDR rules lies in finding the sweet spot of writing rules sensitive enough to detect real threats but not so sensitive that they cause constant false positives. Nobody wants to get an alert every time someone logs in from a coffee shop using a different IP address. But if a legitimate user who normally uses an iPhone suddenly logs in through an Android device in a new geographic location, that’s worth an alert. 

The solution is a team of SOC analysts trained to create models of normal activity. By identifying patterns of typical activity, analysts help the system recognize a scenario that checks all the boxes to be suspicious—but actually isn’t. “We can create threat models based on baseline behavior so we know what’s normal and only send an alert when the pattern changes,” Steve says. “Machine learning can figure that out over time.” 

(This blog provides a summary of the logic used to eliminate false positives.) 

The following real-world scenarios illustrate how SIEM tuning modified standard rules into more accurate reporting tools that stop the alert fatigue. 

Use Case #1: 

Fighting Business Email Compromise 

HBS recently revised one rule intended to deal with the growing threat of business email compromise (BEC) attacks. In these situations, hackers take over a legitimate user account. Then they often create email forwarding rules that let them intercept a user’s messages and conceal the fact that the account has been compromised. Many SIEM solutions now include a stock alert designed to watch for the creation of suspicious forwarding rules. But HBS’s analysts recognized that the stock rule wasn’t catching the forwarding rule hackers are using most right now. So HBS’s SOC team wrote a new rule, had the HBS penetration testing team attempt an exploit to validate the rule, then rolled the rule out to HBS’s entire client base. The new rule not only identifies the activity, but can also automatically orchestrate a response to contain the threat. 

Use Case #2: 

Eliminating False Positives 

“The intent of most rules is terrific. A lot of rules would be amazing if they were accurate 100% of the time. But they aren’t,” Steve says. The HBS SOC team noticed that one stock rule started generating 50 tickets a day for every organization HBS manages. Less than 5% of the alerts were legitimate threats because the rule kept triggering when normal software operations took place. 

The analysts disabled the rule to stop the flood of unactionable data, then rewrote it with complex logic that cut the false positives to almost zero. “Within 72 hours of enabling the new rule, it saved one of our customers from an intrusion that the stock rule missed,” Steve says. 

Use Case #3: 

Tailoring Rules for SMBs 

SIEM developers rightfully talk a lot about their solutions’ machine learning capabilities. But the developers tend to focus their machine learning work on big customers, which means some of the tools don’t do much for small organizations generating a limited amount of monthly data. So HBS’s analysts devote a lot of attention to modifying rule logic so that companies with, say, 30 employees benefit from the next-gen tools as much as companies with 1,000 employees. 

For more information on how HBS’s custom SIEM/XDR rules could make your organization more secure and efficient, contact us today.