RNN / TRACE

An interactive history of sequence intelligence

RNNs
When AI Learned
to Remember

Follow one signal through time: from the first neural loops, to hidden state and gates, to attention - and finally to the architectural break that made modern generative AI possible.

Choose any chapter - nothing is locked

Signal path complete

The loop changed.
The idea survived.

Modern language models no longer carry context in a classical recurrent hidden state. They still generate by predicting one next token, appending it, and repeating.

The durable contribution

RNNs established the logic of learning from ordered experience.

The dominant machinery moved from recurrence to self-attention, but five foundational ideas crossed the divide:

  • Order and context change what an input means.
  • Prediction pressure can produce useful internal structure.
  • Learned representations replace hand-written feature rules.
  • Encoder-decoder systems can turn one sequence into another.
  • Attention lets a model weight the parts of context that matter now.

The Transformer replaced the recurrent relay. It did not erase the path that led to it.