They show that a decoder only transformer (which gpts are) are rnns with infinit... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		joewferrara on Jan 16, 2024 \| parent \| context \| favorite \| on: Transformers Are Multi-State RNNs They show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.

visarga on Jan 17, 2024 [–]

not infinite, just scaling linearly with length

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact