生成未必理解：基于扩散模型能否实现视觉世界模型.docx

VN

V

88

8

生成未必理解：

基于扩散模型能否实现视觉世界模型？

袁粒LiYuan

SchoolofECE,PekingUniversityShenzhenGraduateSchool

“WhatIcannotcreate,Idonotunderstand”—RichardFeynman

“WhatIcangenerate,Idounderstand”

“WhatIcanunderstand,Idogenerate”

?

基于扩散的生成真的理解了吗？

跑步机上的人反着跑步，不符合逻辑

Prompt:Step-printingsceneofapersonrunning,cinematicfilmshotin35mm.

杯子水先撒出来，后破碎，不符合事实

Prompt:Glassshatteringwithredliquidandicecubes

小狼的数量时而五只，时而三只或四只

Prompt:Fivegraywolfpupsfrolickingandchasingeachotheraroundaremotegravelroad

DiffusionsBeatGANs

V.S.

V.S.

Dhariwal,Prafulla,andAlexanderNichol.DiffusionModelsbeatGansonImageSynthesis.NeurIPS34(2021):8780-8794.4

基于Diffusion的视觉生成发展

HuaweiproposedtheT2ImodelPixArt-αbasedonDiTGoogle

HuaweiproposedtheT2ImodelPixArt-αbasedonDiT

GoogleproposedV1oftheVideoDiffusionModel

Proposed

DDIM

AcademicProgress

ShanghaiAI

Labproposed

Latte,aT2V

modelbasedon

DiT

T2I-Adapter(PKU)andControlNet

(Stanford)were

proposedfor

preciseT2Icontrol

CLIP:Aligningtextandimagespaces,laterwidelyusedforT2I

DDPMwas

proposedin

June

LoRAforDiffusion

wasproposed,

quicklyadoptedforvariousapplications

LatentDiffusion

Model(LDM)was

processed

Metaproposed

DiffusionTransformer,

replacingU-NetwithTransformer

20242023202220202021

2024

2023

2022

2020

2021

StabilityAIopen-sourcedStableDiffusionV1andV2OpenAIreleasedSora

StabilityAI

open-sourced

StableDiffusionV1andV2

OpenAIreleasedSora,a

T2Vmodel,butnoAPI

accessyet

OpenAIproposed

DALL-E,basedon

Transformernot

Diffusion

THUproposed

CogView,atext-

to-imagemodel

basedon

Transformer,

followingDALL-E

Videogenerationapps

PikaV1,RunwayGen1

andGen2,andStable

VideoDiffusionemerged

KelingModelby

KuaishouViduby

ShengshuOpenSora

planbyPKU

ApplicationProgress

OpenAIproposed

DALL-E2,based

onDi