智能AI方面的文章Transformer Based Learning for BERT and GPT.pdfVIP

TransformerBasedLearningforBERTandGPT

LanguageModels

LectureNotesonDeepLearning

AviKakandCharlesBouman

PurdueUniversity

Wednesday10thJuly,202408:11

©2024AvinashKak,PurdueUniversity

PurdueUniversity1

PreambleToTOCToHowTo

LargeLanguageModeling(LLM)madeitsdebutintheyear2019.OpenAI

officiallyreleasedtheGPT-2modelinFebruary2019andGoogleofficially

releaseditsBERTmodelinOctober2019.

WhatismostdistinctiveaboutLLMsisthattheycaningestterabytesofpublicly

availabletextualdatasets,learnfromthatdatawithoutanysupervision,and

becomeexpertsinword-to-word,clause-to-clause,sentence-to-sentence,and

paragraph-to-paragraphcontinuitypropertiesofnarratives.

MygoalinthislectureistohighlightsomeoftheimportantaspectsofLLMs,the

architecturesoftheirneuralnetworksthatarebasedonTransformers,howthey

carryoutunsupervisedlearning,etc.

I’llstartbyillustratingseveraloftheLLMconceptsthroughmyexplanationsof

BERT.ThemainreasonforthatisthatmyacquaintanceshipwithBERTdates

backalmosttotheyearitwasborn.TheGPTmodelshaveenteredmy

consciousnessratherrecently.

BERTstartedoutasanacronymforBidirectionalEncoderRepresentationsfrom

Transforms.

PurdueUniversity2

Preamble(contd.)

However,fromthewaytheacronymBERTisnowusedinpublicationsandin

generalcommunications,youcouldsaythatBERThasbecomeanoununto

itself.BERTwasfirstpresentedinthe2019paperbyDevlinetel.:“BERT:

Pre-TrainingofDeepBidirectionalTransformersforLanguage