Triaging the AI Horde
On sorting real security bugs from AI noise, and why I've learned to push back harder on false positives.
Hey Everyone!
This edition is a bit different. I want to talk about something that takes up a surprising amount of my week: triaging security vulnerabilities.
I receive 20-40 reports to triage a week. That's work that must be done, and there are plenty of other people doing it too, but I try to pull my fair share.
Here's the thing that's changed: almost all of those reports are now AI-written. On top of that, we usually receive 3-5 duplicates. Linus Torvalds has an opinion on the matter, too:
"We're making it clear that AI detected bugs are pretty much by definition not secret, and treating them on some private list is a waste of time for everybody involved – and only makes that duplication worse because the reporters can't even see each other's reports." May 20, 2026, Open Source Summit North America keynote panel
Sometimes there is a competent human behind the AI doing the research. Other times you're just talking to a bot. The industry has spent years cultivating a culture where security research is a noble calling, and it genuinely is, but that culture has a side effect. We're now forced to deal with the horde. You can't escape it. If you don't engage, obnoxious researchers, and increasingly obnoxious AIs, will try to escalate until you budge. Even if it would require a cursory read of the threat model to notice that the reported behavior is clearly outside it. Note that the majority of the recently released vulnerabilities have extremely low practical applications.
So the goal of triage, more often than not, is to explain to an AI that something is simply not within the threat model of your software. Not to fix a bug. To explain why there's no CVE worth chasing to protect the maintainer time and avoid burnout.
My flow involves pairing with an Agent, running on a VM. Today I use Pi with an extension that pulls the report straight from HackerOne, working alongside Sol or Grok. Other times I use Claude Code with Opus with a similar setup.
The typical prompt goes through a lot of back-and-forth. Whatever the agent is convinced of, I challenge it, repeatedly, until it can craft a LONG explanation for the AI on the other side. Or, more rarely, it convinces me that it's actually a real vulnerability.
My threshold is WAY higher than it used to be. I've made mistakes in the past. I've been too trusting, waving through things I should have pushed back on, and I paid for them. So now I'd rather push back too hard than let another false positive through. That threshold is earned, and it's expensive to learn the hard way.
One thing I've noticed: as soon as I realize there's a human on the other side of that report, I dedicate much more attention. AI vs AI is one thing. Human vs human is another: that's when the real conversation starts, and it's worth showing up for.
I'd love to hear how other maintainers handle their triage flows these days. Are you seeing the same AI-written flood? How do you push back without burning the genuine researchers? Let me know.
Thanks to everyone doing the unglamorous triage work: you're the reason the ecosystem stays standing!