This is part 2 of my new multi-part series 🐍 Towards Mamba State Space Models for Images, Videos and Time Series.
State space models, known to many disciplines of engineering for decades, are now making their debut in deep learning. On our journey towards Mamba selective state space models and its recent achievements in research, understanding the state space model is crucial. And, as is often the case in engineering, it is the details that make theoretical concepts applicable in practice. In addition to state space models, we must also discuss how to apply them to sequence data, how to handle long-range dependencies, and how to efficiently train them by exploiting certain matrix structures.
Structured state space models build the theoretical foundation for Mamba. However, their connection to system theory and advanced algebra might be one of the obstacles to adopt this new framework.
So, let’s break it down, ensure we understand the key concepts and visualize them to shed some light onto this new-old theory.