人工智能驱动的漏洞搜寻-演变和基准测试.docx

1

AI-PoweredBugHunting-Evolutionand

benchmarking

AlfredoOrtega-ortegaalfredo@X:@ortegaalfredo

Neuroengine.aiJune27,2024

WhileAIholdspromiseforassistingwithbughunting,itsactualimpactre-mainsunclear.ThispresentationaddressesthisdoubtbyintroducingCrash-Benchmark,astandardizedevaluationframeworkforAI-drivenstaticanaly-sistools.We’llshareresultsfromasimplebug-huntingAIagent,AutoKaker,anddiscusstheimplicationsforoptimizingAI-basedbughuntinginC/C++codebases.

1Introduction

Opiniononautomaticbugfindingiscontroversial.Atthedateofthisarticle’spubli-cation,thereisnoconsensusaboutwhetherthisispossibleornot,ortowhatextent.ThisispartlyduetotherapidadvancementofLLMmodels;upuntilmonthsago,open-sourcemodelswerenotadvancedenoughtobeeffectiveatbugfinding.ThereexistsathresholdinthecomplexityofLLMsbeyondwhichbugfindingbecomespossible,andinthisarticlewebenchmarkvariousmodelsandfoundthatthisthresholdhasbeenreachedforsomevulnerabilities.

2CrashBench

Crashbench[1]isasimpleautomatictest-casebasedbenchmarktool.ItconnectstoserveralLLMservicesofferingopensourceandprivateAImodels,sendatestcaseandthenmeasureresults.IftheLLMfoundthebuginthecorrectline,thenscoreisincreasedbyone.

2.1Design

Mostofthetestcasesforthev1versionarebasedonGera’sInsecureProgrammingexercises[2],plus3realvulnerabilityexamples.TheLLMisassignedascorebased

2

onthenumberofvulnerabilitiesthatwerereported,withrealvulnerabilitieshaving10timesthescore.

TheconfigurationofCrashbenchisasingle.inifilecontainingtheprompt,testcasefilesandexpectedlineswherethebugisfound.

[SETTINGS]

SystemPrompt=Youareanexpertsecurityresearcher,programmerandbugfinder.

Prompt=Checkthiscodeforanyout-of-boundsor

integer-overflowvul