CNNs have been really good for images, but there was nothing comparably good for language tasks
RNNs were typically used to understand text -> sequential
Process one word at a time
Did not do well with large sequences of text
Hard to train since it could not be parallelised
Transformers were developed in 2017 by Google researchers and University of Toronto
Initially designed for translation
Can be parallelised -> so can be trained in huge amount of data
How do transformers work (high-level)?
Positional encoding
Encode word order in the data instead of the structure of network
When training on text data -> learns how to interpret these positional encodings
Attention
Neural network structure that allows a text model to look at every single word in the original sentence when making a decision about translating a word in the output sentence
Knowing which words to ‘attend’ to -> learnt over time from looking at a lot of data
Self-attention
Allows a neural network to understand a word in the context of the words around it -> building up an understanding of the internal representation of language
Tokens are associated with vectors -> list of numbers that encode meaning -> coordinates with similar meaning would be closer to each other in the vector space
Attention Block -> vectors talk to each other and pass information back and forth to update their values. This is responsible for figuring out :
Which words in the context are relevant to update the meanings of which other words
How exactly these meanings should be updated
Multilayer Perceptron / Feedforward layer : Vectors dont talk to each other, they go through the same operation in parallel -> Like asking a list of questions to the vectors and updating them based on answers
Repeats the attention and MLP layers
In the end, all the essential meaning would be baked into the last vector in the sequence -> Produces probability distribution of all possible tokens that might come next
Source : https://youtu.be/wjZofJX0v4M?si=VJ8Q0HtXG3RtV\_N-