circuits-llm
Safe HaskellNone
LanguageGHC2024

Circuit.LLM.SSM

Description

Linear state-space model as a Process and the associative-scan law.

A scalar linear SSM is the recurrence h_t = a_t h_{t-1} + b_t. Each step is an affine function (a_t, b_t); composition of affine functions is associative, so the whole sequence can be reduced via an associative scan instead of a sequential fold. This is the parallelisation principle behind S4 Mamba linear attention.

Synopsis

Scalar affine SSM

data Aff Source #

An affine function h -> a * h + b.

Constructors

Aff 

Fields

Instances

Instances details
Eq Aff Source # 
Instance details

Defined in Circuit.LLM.SSM

Methods

(==) :: Aff -> Aff -> Bool #

(/=) :: Aff -> Aff -> Bool #

Show Aff Source # 
Instance details

Defined in Circuit.LLM.SSM

Methods

showsPrec :: Int -> Aff -> ShowS #

show :: Aff -> String #

showList :: [Aff] -> ShowS #

affComp :: Aff -> Aff -> Aff Source #

Composition of affine functions: (f2 . f1)(h) = f2(f1(h)).

>>> let f1 = Aff 2 3; f2 = Aff 5 7 in affComp f2 f1
Aff {affA = 10.0, affB = 22.0}

Because (f2 . f1)(h) = 5*(2*h+3)+7 = 10*h + 22.

seqSSM :: Double -> [Aff] -> [Double] Source #

Sequential scan of a scalar SSM from an initial state.

>>> seqSSM 0 [Aff 1 1, Aff 1 2, Aff 1 3]
[1.0,3.0,6.0]

assocScan :: [Aff] -> [Aff] Source #

Associative scan on affine coefficients. Returns the list of composed affine functions [f1, f2 . f1, f3 . f2 . f1, ...].

The operator is reversed composition because scanl' threads the accumulator on the left, but we want the rightmost function applied first.

assocSSM :: Double -> [Aff] -> [Double] Source #

Apply the associatively-scanned affine functions to an initial state.

chunkedScan :: Int -> [Aff] -> [Aff] Source #

Tree-shaped associative scan with a given chunk size.

Each chunk is reduced to a summary affine function, the summaries are prefix-composed, and the result is expanded back to a per-step prefix. If affComp is associative, this must agree with assocScan for every chunk size. This is the genuine parallelisation principle behind S4 Mamba linear attention.

Vector (harpie) affine SSM

data AffVec Source #

Elementwise affine function on a harpie 1-D array: h_i -> a_i * h_i + b_i. This is the diagonal-matrix special case of the full matrix SSM; it already exercises the associative-scan law on arrays.

Constructors

AffVec 

Instances

Instances details
Eq AffVec Source # 
Instance details

Defined in Circuit.LLM.SSM

Methods

(==) :: AffVec -> AffVec -> Bool #

(/=) :: AffVec -> AffVec -> Bool #

Show AffVec Source # 
Instance details

Defined in Circuit.LLM.SSM

affCompVec :: AffVec -> AffVec -> AffVec Source #

Elementwise composition of diagonal affine functions.

seqSSMVec :: Array Double -> [AffVec] -> [Array Double] Source #

Sequential scan of a vector SSM.

assocScanVec :: [AffVec] -> [AffVec] Source #

Associative scan on vector affine coefficients. The shape is inherited from the first input.

assocSSMVec :: Array Double -> [AffVec] -> [Array Double] Source #

Apply the associatively-scanned vector affine functions to an initial state.

System view

runSystem :: System (->) s (Mono i o) -> s -> [i] -> ([o], s) Source #

Run a deterministic System with a monomial interface over a list of inputs. This is the same semantics as scan, but stated directly on System.

ssmSystemVec :: System (->) (Array Double) (Mono AffVec (Array Double)) Source #

Vector SSM as a 'System (->)' with harpie state, input AffVec, and full state observation.

Multi-head System view (Dirichlet tensor)

multiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double))) Source #

Two independent vector SSM heads packaged as a single System over the Dirichlet tensor Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double)).

The two heads share the same input direction type (AffVec) but receive independent direction values. This is the right polynomial for parallel layers: both heads fire on the same tick, each with its own input. A cartesian Prod would force a choice between heads via Either directions.

runMultiHeadSSMSystem :: (Array Double, Array Double) -> [(AffVec, AffVec)] -> ([(Array Double, Array Double)], (Array Double, Array Double)) Source #

Run the multi-head SSM with independent per-head inputs.

Returns the pair of head outputs at each step and the final pair of states.

runSharedInputMultiHeadSSMSystem :: (Array Double, Array Double) -> [AffVec] -> ([(Array Double, Array Double)], (Array Double, Array Double)) Source #

Run the multi-head SSM with the same input supplied to both heads.

This is the diagonal shared-input case; building it from independent inputs requires an explicit copy of the input direction.

Process / System view

ssmProcess :: Process Aff Double Source #

A Process whose state is the hidden state h and whose output is h. Input is the affine coefficient pair (a_t, b_t).

This is the first-input-seeded presentation with h0 = 0. Use ssmSystem with systemToProcess when you need a non-zero seed.

ssmSystem :: System (->) Double (Mono Aff Double) Source #

A System whose state is the hidden state h and whose output is h. Input is the affine coefficient pair (a_t, b_t); the initial state h0 is supplied when converting to a Process or running directly.

Centrality pair (multi-head coupling)

coupledMultiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double))) Source #

Two-headed SSM with a cross-head coupling: head 1 reads head 2's state.

This breaks premonoidal centrality: the order in which the two heads are threaded through the shared medium matters, because head 1's update depends on head 2's state. It is the "flip" oracle for the multi-head centrality pair.