By Jennifer Lawinski
An artificial intelligence and machine learning engineer with no previous cybersecurity background won the SANS Institute’s Find Evil! hackathon, a competition that asked participants to build autonomous agents capable of conducting digital forensic investigations and show the evidence behind their conclusions.
Hackathon participants had to build the controls, verification and audit capabilities needed to make autonomous forensic analysis more trustworthy. SANS said the contest was an effort to move beyond the frequently repeated assertion that defenders need AI to counter AI-powered attacks.
“I’m kind of sick of everyone saying AI versus AI,” said chief AI officer and chief of research at SANS, Rob T. Lee,in an interview with ISMG. “Let’s go do something.”
The 60-day competition drew 4,413 registrants. Of the 291 projects submitted, 123 met the technical and documentation requirements needed to advance to judging.
A team of 90 practicing incident responders then conducted 1,775 evaluations, testing the agents’ accuracy, safeguards, audit trails and ability to recognize and correct their own errors.
The hackathon was free to enter and awarded more than $22,000 in cash and prizes. The top three projects received $10,000 for first place, $7,500 for second and $4,500 for third.
Mulder, a “fully autonomous forensic investigator” developed by Caleb Evans, took first place. It uses a five-stage process to plan an investigation, execute tools, analyze results, challenge its conclusions and prepare a report. In one investigation, it made 773 logged tool calls across 11 systems and 120 gigabytes of evidence.
The AI model performed the analysis, while an AI harness controlled its access to evidence and tools, checked its work and recorded how it reached each conclusion. An evidence-reference validator rejected a conclusion if the agent couldn’t connect it to an actual artifact, so hallucinated findings cannot be written.
A judge who ran Mulder against his own test disk image reported that it identified 119 of 143 artifacts of compromise.
Commenting on Mulder, SANS fellow and Find Evil! judge Heather Barnhart said, “The hallucination precautions and sources being required and challenged are fantastic. He challenged the tool to work harder before simply jumping to conclusions.”
TRUDI, developed by incident responder Trinity Harrison, placed second. It distinguished itself by rejecting its own starting assumptions after finding that indicators planted in an incident briefing were not supported by the evidence.
Camel, developed by a team led by Allister Beharry, placed third with a sandboxed architecture that traced findings to the scripts and commands that produced them.
Judges tested the finalists with evidence-spoliation attempts, path traversal, command injection and other attacks against their safeguards. Every finalist underwent a separate verification round using evidence the teams had not previously seen.
SANS said projects that documented their errors and testing limitations generally performed better than projects that claimed perfect accuracy. Nearly every serious finalist enforced evidence protections through its architecture rather than relying on prompts instructing the model to behave.
Lee said he had expected the strongest teams to combine AI engineers with experienced incident responders. Instead, the top two projects came from solo participants who had to learn the discipline.
Evans entered with AI engineering experience but no cybersecurity background. Harrison, a student in the SANS Technology Institute programme, had incident response experience but little AI or machine learning expertise.
Hackathons give participants a defined problem, real tools, deadlines and feedback that are difficult to replicate through self-directed study, Lee said. Even people who didn’t complete their submissions told him they had learned new skills and discovered they could build more sophisticated systems than they had expected.
“You just need to know enough about how to get the stuff going, and then you just start building from there,” he said.
In addition to cash prizes to the top three projects, SANS recognized two additional finalists, FindEvil and Protocol SIFT++. All five harnesses are open source, and SANS said it is incorporating them into the SIFT Workstation.
