2024高性能大语言模型推理.pptx

高性能大语言模型推理2024/12/61

LLM大语言模型的基本结构LayernormAttentionSiluMatMulRotaryEmbeddingKVCache2024/12/62

LLM大语言模型的基本结构LayernormAttentionSiluMatMulRotaryEmbeddingKVCache2024/12/63OpnameTypeGputimeMatmulForQGemm30MatmulForKGemm31MatmulForVGemm30MatMulForOGemm31SkipRMSNormANormalization5FeedForwardGemmAGemm70FeedForwardGemmBGemm71FeedForwardGemmCGemm73SkipRMSNormBNormalization5AttentionAttention9(会根据输入长度变化哦)

LLMInferencePPL.LLM将推理分成Prefill阶段Decoding阶段2024/12/64

LLMInferencePPL.LLM将推理分成Prefill阶段Decoding阶段2024/12/65

LLMBenchmarkThoughtput(吞吐量)FirstTokenLatency(首字延迟)Latency(延迟)QPS(每秒请求数)2024/12/66

LLMBenchmarkThoughtput(吞吐量)FirstTokenLatency(首字延迟)Latency(延迟)QPS(每秒请求数)2024/12/67LLaMAFP16H800PPL.NN.LLMCUDA12.0modelsize(B)gpubatchinputlenoutputlenprefill(ms)decode(ms)throughput(tokens/s)mem(GiB786.83145.8513.26846.91288.3213.34126.95573.6613.50107.051131.1013.82711682567.947.152229.6514.477132825610.417.983992.1815.777164825616.368.557429.8518.3671128825630.0310.4712095.6623.5771256825658.6915.2016597.2333.9771384825683.4222.9016536.8644.36715128256110.8225.4519785.2354.78717688256165.1937.1920298.5275.577111024102430.868.64115.4013.867121024102458.338.75227.0914.5671410241024109.138.88445.3615.9371810241024213.489.23848.0418.70711610241024415.9510.061528.7324.22713210241024830.3513.842184.1735.277164102410241633.8920.182939.0757.377180102410242078.3722.153309.2568.42

LLMBenchmarkThoughtput(吞吐量)FirstTokenLatency(首字延迟)Latency(延迟)QPS(每秒请求数)2024/12/68

/openppl-public/ppl.llm.serving/blob/master/tools/samples_1024.jsonPPLServingThoughtput(吞吐量)FirstTokenLatency(首字延迟)Latency(延迟)QPS(每秒请求数)=K/完成K个请求的时间2024/12/69Dataset:/openppl-public/ppl.llm.serving/blob/master/tools/samples_1024.json

LLMInferenceLLM的推理由一些子过程构成：Tokenize(将文本转换为向量)Computing(模型推理)Detokenize(将向量转换为文本)Sampling(依据推理结果进行采样)2024/12/610在每一个Decoding阶段结束时，我们都返回一次