XLNet
Google / CMU · June 2019
● activeOpen Sourcedecoder onlytext
Description
A model from Google and Carnegie Mellon that combined the best of autoregressive (GPT-style) and autoencoding (BERT-style) approaches using permutation-based language modeling. Outperformed BERT on 20 tasks when released.
Key Innovations
permutation-lm
Autoregressive
AutoregressiveGenerates text one token at a time, each prediction based on all previous tokens. The foundation of modern language models.