From Bag of Words to Transformers: 10 Years of Practical Natural Language Processing

Depending on the context (here, the last word of the sentence), ‘it’ can refer to ‘animal’ or ‘street’. From Google introducing transformers.

Until 2016: the reign of bag of words and TF-IDF

2016–2019: the rise of word embeddings + Keras and Tensorflow

Encoding a sequence of words with a Recurrent Neural Network

2018–2019: the PyTorch breakthrough

Tensorflow (blue) VS PyTorch (red) search trend in Google (source: Google Trend)

2019: the birth of transformers and pretrained language models

The Transformer — model architecture (from Attention Is All You Need paper)

What’s next?



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store