高通：2024多模态AI的感官融合-视觉、听觉与交互技术白皮书(英文版).docx

are

are

Integratingsenses:

HowAIislearningto

see,hear,andinteract

RolandMemisevic

SeniorDirectorofEngineeringatQualcommAIResearch

JointworkwithSunnyPanchal,ApratimBhattacharyya,

GuillaumeBerger,AntoineMercier,RezaPourreza,SanjayHaresh,andothers

September24,2024

SnapdragonandQualcommbrandedproductsareproductsofQualcommTechnologies,Inc.and/oritssubsidiaries.

Agenda

?Keyconcept:streamingarchitecture

?Importanceofdatasetsforend-to-endtraining

?Efficienthuman-AIinteractionandvideo-basedreasoning

?ImprovingstreamingvideoLLMsusingauxiliarytasks

?QA

2

MODALITYANDUSECASECAPABILITYANDKPI

LongercontextwindowAllows

Longercontextwindow

Allowsin-depthconversations

Voiceisanaturalandintuitiveinterfaceforconversation

GenerativeAI

Largemultimodalmodelso

Largemultimodalmodelso

Utilizingmoresensinginputmodalitiestobetterunderstandtheworld

oPersonalization

Fine-tunedmodelscustomizedtoconsumers,enterprises,orindustries(e.g.,LoRA)

continueto

increase

HigherresolutionProcesshigherfidelityimagesforbetter

Higherresolution

Processhigherfidelityimagesforbetteraccuracy

Generatingcontentforaricherandmorerealisticexperience

Agents

Executemulti-steptaskswithreasoningautonomouslytoachieveagoal

3LoRA:low-rankadaptation

3

Full-stackAIoptimization

forLMs

Runscompletely

onthedevice

Significantlyreduces

runtimelatencyandpowerconsumption

Continuouslyimproves

theQualcomm?AIStack

LM:Languagevisionmodel

Designinganefficientdiffusionmodelthroughknowledge

distillationforhighaccuracy

Knowledgedistillationforpruningandremovingofattentionblocks,resultinginaccuratemodelwithimproved

performanceandpowerefficiency

Qualcomm?AIEnginedirect

forimprovedperformanceandminimizedmemoryspillage

AIaccelerationontheQualcomm?HexagonmNPUoftheSnapdragon?8Gen3MobileProc