Is AI the Solution to Alert Fatigue? The Problem Can Only Be Solved With a Deep Understanding of How it Was Created.

Recent serious work on the problem of false positives and their effects [1] has measured the rate of false positive alerts, in security operations centers, as ranging between 75% to an astounding 99%. Rates in the nineties have been reported by researchers and data scientists doing work on the problem and I observe similar rates in my own measurements across a wide variety of tools and technologies. At the time of this writing, so-called “agentic” or AI-based products promising a solution to alert fatigue are numerous and increasing by the day. Extensive debate is taking place as to the veracity of these claims. Is AI the magical solution to the problem? To understand that, you need to know a little about how we got here, and about how security threat detections are developed, and some mistakes we made along the way. We know a lot about alert fatigue because we actually worked on manufacturing it. Have you ever wondered about how the people who created the problem would approach solving it, if they had a free hand? This is our answer, an open source project named PROTOSTAR that is the work of eight security researchers and data scientists over the past year plus.

Is AI the answer? We think it has a place, but not exactly the way it is being sold today. We know a lot about alert fatigue because we actually worked on manufacturing it. Have you ever wondered about how the people who created the problem would approach solving it, if they had a free hand? One of the fascinating things you see, inside software manufacturers, is what goes on during “build weeks.” Twice a year, or on some interval, most teams are free to ignore product schedules and build whatever they choose, often in self-organized teams. What comes out of those periods - tools we build for ourselves vs for demos - tends to look quite different from the products we ship during the other 50 weeks of the year. PROTOSTAR is our approach to alert signal processing, using ML and AI, in a different direction. It is the result of over a years’ work by seven security researchers, developers, and data scientists while on walkabout, we could never have taken the time to do this while at our day jobs. It differs from the current crop of solutions to alert fatigue in several ways; 1) It generates data structures that enable intelligent processing of all detections, by an LLM or by a human, at reasonable cost. 2) is asymptotically efficient, decisioning more than ten thousand alerts per minute, and accuracy increases with volume. 3) It applies ML differently, and thoughtfully, in order to boost signal rather than increase noise.

[1] 99% False Positives: A Qualitative Study of SOC Analysts’ Perspectives on Security Alarms
Authors: Bushra A. Alahmadi, Louise Axon, and Ivan Martinovic, University of Oxford