Complex Memory — Christopher Housholder

An active research problem about fixed-size persistent memory in sequence models.

Active research. I previously used this page as an implementation manual; current mechanism, source excerpts, equations, and benchmark details have been removed while the project is ongoing.

Background

Attention and memory are adjacent but different operations. Attention decides what is useful for the current computation. Persistent memory decides what should remain available after the immediate context has moved on.

That distinction matters when a model has a short local window but must preserve a small amount of information over a much longer delay. Simply caching everything is one answer, but then state grows with context length.

Problem

What properties should a fixed-size persistent state have if it is expected to retain useful information over long delays while remaining strictly causal? The central tradeoffs are memory capacity, temporal resolution, stability, state size, and computational cost.

Tools

For this project the useful preliminaries are dynamical rather than software-specific. I want causality, fixed state, stability, and delayed retrieval defined before talking about any particular memory architecture.

Causal State

Definition. A persistent-state model has a state \(m_t\in\mathbb R^d\) satisfying \[ m_t=\Phi(m_{t-1},x_t). \] It is causal if \(m_t\) depends only on \(x_1,\ldots,x_t\) and the initial state \(m_0\).

Definition. The memory is fixed-size if the state dimension \(d\) is independent of sequence length.

A Basic Stability Model

Lemma. For the linear recurrence \[ m_t=Am_{t-1}+Bu_t, \] if \(\|A\|\leq\rho<1\) and \(\|u_t\|\leq M\), then \[ \|m_t\|\leq \rho^t\|m_0\|+\frac{\|B\|M}{1-\rho}. \]

Proof. Iterating gives \[ m_t=A^tm_0+\sum_{j=0}^{t-1}A^jBu_{t-j}, \] and the claim follows from the geometric-series bound.

Delayed Retrieval

Definition. A delayed-retrieval task places a key-value fact at time \(s\), introduces a gap of length \(\Delta\), and later queries the value. Performance can therefore be studied as a function of \(\Delta\).

State Cost

Definition. The recurrent state cost is the number of values that must be carried from one step or chunk to the next, excluding temporary activations that can be discarded after local computation.

These preliminaries are enough to state the public problem. The nonlinear state equations, gating rules, timescale construction, and current experiments remain private while the project is active.

Direction

I am comparing broad classes of persistent-state mechanisms under controlled memory budgets and asking which structural properties survive as retrieval delay increases. The exact state equations, update rules, ablations, and current results are intentionally withheld.