blechRNN: Denoising and Latent Space Discovery for Multivariate Time Series with Autoregressive Autoencoders | by Abuzar Mahmood | Feb, 2025

Time series data is ubiquitous, spanning fields from finance and healthcare to sensor networks and industrial process monitoring. However, real-world time series data is often noisy, incomplete, and high-dimensional, making analysis and modeling challenging. The blechRNN repository (https://github.com/abuzarmahmood/blechRNN) offers a powerful approach to tackling these challenges by leveraging a cleverly designed autoregressive recurrent autoencoder architecture. This blog post will delve into the inner workings of blechRNN, explaining the theoretical underpinnings and practical implications of its design.

Estimating firing rates from neural spike trains is a fundamental problem in neuroscience. Traditional methods for firing rate estimation, such as calculating spike counts within fixed time bins or using simple smoothing kernels (as described in the provided link: https://pmc.ncbi.nlm.nih.gov/articles/PMC2783748/)), often struggle when dealing with data that exhibits these characteristics, all common in real neural recordings:

• Noise: Spike detection itself can be noisy. Extracellular recordings are susceptible to electrical noise from various sources. Furthermore, even perfectly detected spikes might not perfectly reflect the underlying “true” firing rate due to inherent stochasticity in neuronal firing. This noise makes it difficult to discern the true underlying firing rate patterns. The “noise” here isn’t just measurement error, but also includes the inherent variability of neural spiking around the underlying rate.
• High Dimensionality: Modern neuroscientific experiments often involve recording from large populations of neurons simultaneously (e.g., using multi-electrode arrays or calcium imaging). This results in multivariate time series data, where each neuron represents a dimension. The interactions and dependencies between these neurons (and their firing rates) can be highly complex and non-linear. Understanding these population dynamics is a key goal.
• Missing Data: In practice, it’s common to lose spikes due to recording artifacts, or to have periods where the activity of certain neurons cannot be reliably determined. This creates gaps in the spike trains.
• Irregular Spiking (Not Sampling): It’s not irregular sampling that is an issue, rather, individual neurons don’t fire at perfect, regular intervals. This inherent irregularity, even in perfectly detected spikes, presents challenges to many methods that rely on equal time bins. Even if the data is sampled regularly (e.g. by the recording device), the underlying spikes won’t be regularly timed.

blechRNN, in the context of neural data, aims to address the noise and high-dimensionality problems by providing improved estimates of underlying firing rates. It does this through:

• Denoising: Extracting the underlying “smooth” firing rate signal from the noisy, discrete spike trains. This is akin to finding the “true” time-varying probability of spiking, given the observed, noisy spikes.
• Latent Space Learning: Discovering a lower-dimensional representation (the “latent space”) that captures the essential dynamics of the population firing rate activity. This lower-dimensional representation can reveal underlying neural computations and coordinated activity patterns across the recorded neurons. Instead of looking at, say, 100 individual firing rate estimates, we might learn a 10-dimensional latent representation that captures the most important aspects of the population activity.

The key conceptual shift here is moving from thinking about individual spikes as the primary data to thinking about the underlying, time-varying firing rate as the signal of interest, and the spikes as noisy observations of that signal. blechRNN provides a powerful way to estimate this underlying signal and its population-level structure.

Before examining the specific architecture, let’s review the fundamental concepts that blechRNN builds upon:

2.1 Autoencoders

An autoencoder is a type of neural network trained to reconstruct its input. It consists of two main parts:

• Encoder: Maps the input data to a lower-dimensional representation, often called the “latent code” or “bottleneck.” This forces the network to learn a compressed representation of the essential features of the input.
• Decoder: Reconstructs the original input from the latent code.

The network is trained by minimizing the reconstruction error, the difference between the input and the reconstructed output. A simple autoencoder can be visualized as follows:

2.2 Recurrent Neural Networks (RNNs) and LSTMs

RNNs are specifically designed to handle sequential data like time series. Unlike feedforward networks, RNNs have internal memory (hidden state) that allows them to process information sequentially, taking into account the temporal dependencies between data points.

The key idea is that the hidden state at time t is a function of the input at time t and the hidden state from the previous time step t-1:

h_t = f(x_t, h_t-1)

Where:

• h_t is the hidden state at time t.
• x_t is the input at time t.
• f is a non-linear activation function.

Long Short-Term Memory (LSTM) networks are a special type of RNN that are particularly effective at handling long-range dependencies in sequences. LSTMs use a “gating” mechanism to control the flow of information into and out of the hidden state, preventing the vanishing gradient problem that can plague standard RNNs. The gates (input, forget, and output gates) are themselves learned functions of the input and previous hidden state.

2.3 Autoregressive Models

An autoregressive (AR) model predicts future values based on a linear combination of past values. In the simplest case, an AR(p) model uses the p previous values to predict the current value:

x_t = c + φ_1 * x_t-1 + φ_2 * x_t-2 + ... + φ_p * x_t-p + ε_t

Where:

• x_t is the value at time t.
• c is a constant.
• φ_1, φ_2, …, φ_p are the autoregressive coefficients.
• ε_t is a white noise error term.

Crucially, autoregressive models explicitly model the temporal dependencies in the data. (https://aws.amazon.com/what-is/autoregressive-models/)

2.4 Latent Variable Models

A latent variable model assumes that the observed data is generated by an underlying, unobserved (latent) variable. The goal is to infer the distribution of the latent variable given the observed data. This is often done using a probabilistic framework.

Conceptual Diagram:

The mapping between the latent variable z and the observed data is learned during the model training. (https://en.wikipedia.org/wiki/Latent_variable_model)

blechRNN combines the power of autoencoders, RNNs (specifically LSTMs), and autoregressive modeling to create a robust architecture for time series denoising and latent space learning. Here’s how it works:

3.1 Encoder:

The encoder is an LSTM-based RNN. It takes the noisy, multivariate time series as input and processes it sequentially. The hidden state of the LSTM at each time step captures information about the past inputs. The final hidden state of the encoder LSTM serves as the latent code (z). This latent code represents a compressed, lower-dimensional representation of the entire input sequence.

3.2 Decoder:

The decoder is also an LSTM-based RNN, but it operates in an autoregressive manner. This is the key innovation of blechRNN.

• Initialization: The decoder’s initial hidden state is initialized using the latent code (z) from the encoder. This sets the “context” for the decoder.
• Autoregressive Prediction: At each time step t, the decoder:

1. Takes as input the previously reconstructed value (x’_t-1). Initially (at t=0), this might be a special start-of-sequence token or a zero vector.

2. Uses its hidden state (which incorporates information from the latent code and previous reconstructed values) to predict the current value (x’_t).

3. The predicted value (x’_t) is then used as input for the next time step (t+1).

This autoregressive process forces the decoder to learn the temporal dependencies in the data. It’s not just reconstructing the input; it’s generating the sequence one step at a time, conditioned on the latent code and its own previous outputs.

Conceptual Diagram:

+---------------+       +-----------------+      +----------------------+      +-----------------+       +-----------------+
| Noisy         | --->  |  Encoder        | ---> |  Latent Code (LSTM)  | ---> | Decoder          | ---> | Reconstructed   |
| Input         |       |  (Hidden State) |      |      (z_1...z_T)     |      | (x'_1...x'_T)    |      |    Output       |
| (x_1...x_T)|  |       +-----------------+      +----------------------+      +-----------------+       |  (x'_1...x'_T)  |
+---------------+                                        ^                                               +-----------------+
|               
|               
+---------------------+             
| Previous Output     |
| (z'_t-1)          |
+---------------------+

3.3 Training

The entire network (encoder and decoder) is trained end-to-end by minimizing a reconstruction loss. Common loss functions include Mean Squared Error (MSE) or Mean Absolute Error (MAE) between the original input sequence (x_1…x_T) and the reconstructed sequence (x’_1…x’_T).

Key Advantage of Autoregressive Decoding:

The autoregressive decoder is crucial for several reasons:

• Temporal Consistency: It explicitly enforces temporal dependencies in the reconstructed output. The decoder learns to generate sequences that are consistent with the dynamics of the underlying process.
• Denoising: By conditioning the reconstruction on the latent code and previous outputs, the decoder is less susceptible to noise in the input. The autoregressive nature acts as a kind of “smoothing” mechanism.
• Latent Disentanglement: The decoder is incentivized to make maximal use of the information encoded in the latent representation, leading to latents which are more informative.
• Handling Missing Data: The autoregressive decoder can be used to impute missing values in the time series. By feeding in the available data and letting the decoder predict the missing values, you can obtain a complete sequence.

Let’s now examine the blechRNN repository in more detail, linking the code structure to the concepts discussed above.

4.1. Repository Structure

The repository is organized into several key directories and files:

• blech_rnn/: Contains the core code for the blechRNN model.
• models.py: Defines the AutoregressiveLSTM class, which implements the core autoencoder architecture.
• train.py: Contains the training loop and logic for optimizing the model.
• preprocess.py: Provides utilities for data preprocessing, including loading, cleaning, and windowing time series data.
• utils.py: Includes helper functions for various tasks.

4.2. models.py: The AutoregressiveLSTM Class

This is the heart of the repository. The AutoregressiveLSTM class implements the autoregressive recurrent autoencoder. Let’s break down the key components:

• __init__: The constructor initializes the encoder and decoder LSTMs, along with any necessary linear layers for transforming the hidden states. Hyperparameters like the number of LSTM layers, hidden state size, and input/output dimensions are defined here.
• encode(input): This method takes the input time series and passes it through the encoder LSTM. It returns the final hidden state of the encoder, which serves as the latent code.
• decode(latent, rollout_steps): This is the core autoregressive decoding logic.
• — It takes the latent code (from the encoder) and the number of rollout steps (the length of the sequence to generate).
• — It initializes the decoder LSTM’s hidden state with the latent code.
• — It iteratively generates the output sequence, using the previous output as input to the next time step.
• forward(input): This method combines the encoding and decoding steps. It takes the input, encodes it to get the latent code, and then decodes the latent code to generate the reconstructed output.
• loss(input, output): Calculates the reconstruction loss between the original input and the reconstructed output.
• step(epoch, batch, optimizer, clip_grad): Performs a single training step, which would be part of the training loop implemented elsewhere. This includes forward pass, loss computation, backpropagation, and parameter updates.

4.3. train.py: The Training Loop

This script handles the training process:

• Data Loading and Preprocessing: Loads the time series data, likely using functions from preprocess.py. This may involve splitting the data into training, validation, and test sets, normalizing the data, and creating batches.
• Model Initialization: Creates an instance of the AutoregressiveLSTM class.
• Optimizer Setup: Defines an optimizer (e.g., Adam) to update the model’s parameters.
• Training Loop: Iterates over the training data for a specified number of epochs.
• For each batch of data:
• – Calls the step method to perform a forward pass, calculate the loss, and update the model’s parameters.
• – Optionally, calculates and logs metrics (e.g., training loss, validation loss).
• – Optionally, saves checkpoints of the model.
• Evaluation: After training, evaluates the model’s performance on the validation and/or test sets.

4.4. preprocess.py: Data Handling

This script provides utilities for:

• Loading Data: Reading time series data from files (e.g., CSV, HDF5).
• Cleaning Data: Handling missing values (e.g., imputation), removing outliers.
• Normalization: Scaling the data to a suitable range (e.g., zero mean and unit variance).
• Windowing: Creating overlapping windows of the time series to create input-output pairs for training the autoencoder. This is essential for sequence-to-sequence learning. For example, a window of length 100 might be used to predict the next 100 time steps.
• Batching: Grouping the windows into batches for efficient training.

5.1. Hyperparameter Tuning

The performance of blechRNN, like any deep learning model, is sensitive to the choice of hyperparameters. Key hyperparameters to tune include:

• latent_dim: The dimensionality of the latent space. This is a crucial parameter. A smaller latent_dim forces a more compressed representation, which can be good for denoising but may lose important information. A larger latent_dim allows for a richer representation but may be more prone to overfitting and capturing noise. Finding the right balance is key.
• num_layers: The number of LSTM layers in both the encoder and decoder. Deeper networks (more layers) can learn more hierarchical representations but are harder to train.
• learning_rate: The learning rate for the optimizer. A good starting point is often 1e-3 or 1e-4, but this needs to be tuned.
• window_size: These parameters control the length of the input and output sequences. They should be chosen based on the characteristics of the time series data. For example, if the data has strong daily seasonality, the window_size should be at least one day’s worth of data.
• Regularization: Techniques like dropout or L1/L2 regularization on the LSTM weights can help prevent overfitting, especially with limited data.

Strategies for Hyperparameter Tuning:

• Grid Search: Systematically try different combinations of hyperparameter values. This is computationally expensive but exhaustive.
• Random Search: Randomly sample hyperparameter values from a specified range. Often more efficient than grid search.
• Bayesian Optimization: Use a probabilistic model to guide the search for optimal hyperparameters. This can be very efficient, especially for high-dimensional hyperparameter spaces. Libraries like Optuna or Hyperopt can be used.
• Manual Tuning: Start with a reasonable set of hyperparameters and iteratively adjust them based on the model’s performance on a validation set. This requires experience and intuition.

5.4. Extending blechRNN

The blechRNN architecture provides a solid foundation, but it can be extended in several ways:

• Variational Autoencoder (VAE): Instead of learning a deterministic latent code, you could use a VAE framework. The encoder would output the parameters of a probability distribution (e.g., mean and variance of a Gaussian), and the decoder would sample from this distribution. This can lead to a more robust and expressive latent space. This is the most common extension for an Autoencoder.
• Conditional VAE (CVAE): If you have additional information about the time series (e.g., labels or external covariates), you can incorporate this information into the model using a CVAE.
• Different RNN Architectures: Experiment with different RNN architectures, such as GRUs (Gated Recurrent Units) or transformers.
• Attention Mechanisms: Incorporate attention mechanisms into the encoder and/or decoder to allow the model to focus on the most relevant parts of the input sequence.
• Hybrid Models: Combine blechRNN with other time series models, such as ARIMA or Prophet.

The blechRNN repository offers a powerful and well-designed approach to denoising and latent space learning for multivariate time series data. Its autoregressive recurrent autoencoder architecture combines the strengths of autoencoders, RNNs, and autoregressive modeling to effectively capture temporal dependencies and extract underlying structure from noisy data. By understanding the core concepts and carefully tuning the hyperparameters, blechRNN can be a valuable tool for a wide range of time series analysis tasks. The provided resources and the well-structured codebase make it a great starting point for both research and practical applications involving complex time series data. The ability to handle noise, discover meaningful latent representations, and potentially impute missing data makes it a versatile and robust solution.