Mamba2 ssd层的一些疑问.pdf

Mamba2ssd层的一些疑问

图1：mamba2ssd层框图

参考代码mamba/mamba_ssm/modules/ssd_minimal.pyatmain·state-spaces/mamba

我稍微在test_correctness()函数做了修改跟mamba2-130M设置匹配了一下

#Copyright(c)2024,AlbertGuandTriDao.

MinimalimplementationofSSD.

ThisisthesameasListing1fromthepaper.

importtorch

importtorch.nn.functionalasF

fromeinopsimportrearrange,repeat

frommamba_ssm.ops.triton.ssd_combinedimport

mamba_chunk_scan_combined

defsegsum_unstable(x):

Naivesegmentsumcalculation.

Tx.size(-1)

x_cumsumtorch.cumsum(x,dim-1)

x_segsumx_cumsum[...,:,None]-x_cumsum[...,None,:]

masktorch.tril(torch.ones(T,T,devicex.device,

dtypebool),diagonal0)

x_segsumx_segsum.masked_fill(~mask,-torch.inf)

returnx_segsum

defsegsum(x):

Morestablesegmentsumcalculation.

Tx.size(-1)

xrepeat(x,...d-...de,eT)

masktorch.tril(torch.ones(T,T,devicex.device,

dtypebool),diagonal-1)

xx.masked_fill(~mask,0)

x_segsumtorch.cumsum(x,dim-2)

masktorch.tril(torch.ones(T,T,devicex.device,

dtypebool),diagonal0)

x_segsumx_segsum.masked_fill(~mask,-torch.inf)

returnx_segsum

defssd_minimal_discrete(X,A,B,C,block_len,

initial_statesNone):

Arguments:

X:(batch,length,n_heads,d_head)

A:(batch,length,n_heads)

B:(batch,length,n_heads,d_state)

C:(batch,length,n_heads,d_state)

Return:

Y:(batch,length,n_heads,d_head)

assertX.dtypeA.dtypeB.dtypeC.dtype

assertX.shape[1]%block_len0

#Rearrangeintoblocks/chunks

X,A,B,C[rearrange(x,b(cl)...-bcl...,

lblock