What Algorithms Can Transformers Learn? A Study in Length Generalization

Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua Susskind, Samy Bengio, Preetum Nakkiran
January 2024
PDF Code
Type
Conference paper
Publication
International Conference on Learning Representations (ICLR), 2024
Transformers Length Generalization Systematic Generalization Language Models