AI slop and fake reports are coming for your bug bounty programs

So-called AI slop, which means LLM-generated low-quality photos, movies, and textual content, has taken over the web within the final couple of years, polluting web sites, social media platforms, not less than one newspaper, and even real-world occasions. 

The world of cybersecurity just isn’t resistant to this downside, both. Within the final yr, individuals throughout the cybersecurity business have raised considerations about AI slop bug bounty experiences, which means experiences that declare to have discovered vulnerabilities that don’t truly exist, as a result of they had been created with a big language mannequin that merely made up the vulnerability, after which packaged it right into a professional-looking writeup. 

“Persons are receiving experiences that sound affordable, they give the impression of being technically right. After which you find yourself digging into them, attempting to determine, ‘oh no, the place is that this vulnerability?’,” Vlad Ionescu, the co-founder and CTO of RunSybil, a startup that develops AI-powered bug hunters, instructed Trendster. 

“It seems it was only a hallucination all alongside. The technical particulars had been simply made up by the LLM,” stated Ionescu. 

Ionescu, who used to work at Meta’s purple crew tasked with hacking the corporate from the within, defined that one of many points is that LLMs are designed to be useful and provides optimistic responses. “In the event you ask it for a report, it’s going to provide you a report. After which individuals will copy and paste these into the bug bounty platforms and overwhelm the platforms themselves, overwhelm the shoppers, and also you get into this irritating scenario,” stated Ionescu. 

“That’s the issue persons are working into, is we’re getting lots of stuff that appears like gold, however it’s truly simply crap,” stated Ionescu. 

Simply within the final yr, there have been real-world examples of this. Harry Sintonen, a safety researcher, revealed that the open supply safety undertaking Curl obtained a pretend report. “The attacker miscalculated badly,” Sintonen wrote in a publish on Mastodon. “Curl can scent AI slop from miles away.”

In response to Sintonen’s publish, Benjamin Piouffle of Open Collective, a tech platform for nonprofits, stated that they’ve the identical downside: that their inbox is “flooded with AI rubbish.” 

One open supply developer, who maintains the CycloneDX undertaking on GitHub, pulled their bug bounty down solely earlier this yr after receiving “nearly solely AI slop experiences.”

The main bug bounty platforms, which basically work as intermediaries between bug bounty hackers and firms who’re keen to pay and reward them for locating flaws of their merchandise and software program, are additionally seeing a spike in AI-generated experiences, Trendster has realized. 

Michiel Prins, the co-founder and senior director of product administration at HackerOne, instructed Trendster that the corporate has encountered some AI slop. 

“We’ve additionally seen an increase in false positives — vulnerabilities that seem actual however are generated by LLMs and lack real-world affect,” stated Prins. “These low-signal submissions can create noise that undermines the effectivity of safety applications.”

Prins added that experiences that comprise “hallucinated vulnerabilities, imprecise technical content material, or different types of low-effort noise are handled as spam.”

Casey Ellis, the founding father of Bugcrowd, stated that there are positively researchers who use AI to seek out bugs and write the experiences that they then undergo the corporate. Ellis stated they’re seeing an general enhance of 500 submissions per week. 

“AI is extensively utilized in most submissions, however it hasn’t but brought on a big spike in low-quality ‘slop’ experiences,” Ellis instructed Trendster. “This’ll most likely escalate sooner or later, however it’s not right here but.”

Ellis stated that the Bugcrowd crew that analyzes submissions critiques the experiences manually utilizing established playbooks and workflows, in addition to with machine studying and AI “help.”

To see if different firms, together with people who run their very own bug bounty applications, are additionally receiving a rise in invalid experiences or experiences containing non-existent vulnerabilities hallucinated by LLMs, Trendster contacted Google, Meta, Microsoft, and Mozilla. 

Damiano DeMonte, a spokesperson for Mozilla, which develops the Firefox browser, stated that the corporate has “not seen a considerable enhance in invalid or low-quality bug experiences that may seem like AI-generated,” and the rejection charge of experiences — which means what number of experiences get flagged as invalid — has remained regular at 5 or 6 experiences monthly, or lower than 10% of all month-to-month experiences.

Mozilla’s staff who assessment bug experiences for Firefox don’t use AI to filter experiences, as it could possible be tough to take action with out the chance of rejecting a legit bug report,” DeMonte stated in an e mail.

Microsoft and Meta, firms which have each wager closely on AI, declined to remark. Google didn’t reply to a request for remark. 

Ionescu predicts that one of many options to the issue of rising AI slop can be to maintain investing in AI-powered methods that may not less than carry out a preliminary assessment and filter submissions for accuracy. 

In reality, on Tuesday, HackerOne launched Hai Triage, a brand new triaging system that mixes people and AI. In line with HackerOne, this new system is leveraging “AI safety brokers to chop via noise, flag duplicates, and prioritize actual threats.” Human analysts then step in to validate the bug experiences and escalate as wanted.

As hackers more and more use LLMs and firms depend on AI to triage these experiences, it stays to be seen which of the 2 AIs will prevail.