From RNNs to Mamba: A Mathematical Journey Through Modern Sequence Models: Part 2

Memory, State Representations, and Recurrent Neural Networks 3. Why Sequence Models Need Memory In Part 1, sequential data was represented as an ordered collection of observations, [latex] \begin{equation} X={x_1,x_2,\ldots,x_T}. \tag{3.1}…

Continue ReadingFrom RNNs to Mamba: A Mathematical Journey Through Modern Sequence Models: Part 2

From RNNs to Mamba: A Mathematical Journey Through Modern Sequence Models: Part 1

In this section, we establish the mathematical foundations of sequential data that will be used throughout the remainder of this article. Specifically, we discuss: What is sequential data? — Definition…

Continue ReadingFrom RNNs to Mamba: A Mathematical Journey Through Modern Sequence Models: Part 1