| Safe Haskell | None |
|---|---|
| Language | GHC2024 |
Circuit.LLM.SSM
Description
Linear state-space model as a Process and the associative-scan law.
A scalar linear SSM is the recurrence h_t = a_t h_{t-1} + b_t. Each step
is an affine function (a_t, b_t); composition of affine functions is
associative, so the whole sequence can be reduced via an associative scan
instead of a sequential fold. This is the parallelisation principle behind
S4 Mamba linear attention.
Synopsis
- data Aff = Aff {}
- affComp :: Aff -> Aff -> Aff
- seqSSM :: Double -> [Aff] -> [Double]
- assocScan :: [Aff] -> [Aff]
- assocSSM :: Double -> [Aff] -> [Double]
- chunkedScan :: Int -> [Aff] -> [Aff]
- data AffVec = AffVec {}
- affCompVec :: AffVec -> AffVec -> AffVec
- seqSSMVec :: Array Double -> [AffVec] -> [Array Double]
- assocScanVec :: [AffVec] -> [AffVec]
- assocSSMVec :: Array Double -> [AffVec] -> [Array Double]
- runSystem :: System (->) s (Mono i o) -> s -> [i] -> ([o], s)
- ssmSystemVec :: System (->) (Array Double) (Mono AffVec (Array Double))
- multiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double)))
- runMultiHeadSSMSystem :: (Array Double, Array Double) -> [(AffVec, AffVec)] -> ([(Array Double, Array Double)], (Array Double, Array Double))
- runSharedInputMultiHeadSSMSystem :: (Array Double, Array Double) -> [AffVec] -> ([(Array Double, Array Double)], (Array Double, Array Double))
- ssmProcess :: Process Aff Double
- ssmSystem :: System (->) Double (Mono Aff Double)
- coupledMultiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double)))
Scalar affine SSM
An affine function h -> a * h + b.
affComp :: Aff -> Aff -> Aff Source #
Composition of affine functions: (f2 . f1)(h) = f2(f1(h)).
>>>let f1 = Aff 2 3; f2 = Aff 5 7 in affComp f2 f1Aff {affA = 10.0, affB = 22.0}
Because (f2 . f1)(h) = 5*(2*h+3)+7 = 10*h + 22.
seqSSM :: Double -> [Aff] -> [Double] Source #
Sequential scan of a scalar SSM from an initial state.
>>>seqSSM 0 [Aff 1 1, Aff 1 2, Aff 1 3][1.0,3.0,6.0]
assocScan :: [Aff] -> [Aff] Source #
Associative scan on affine coefficients. Returns the list of composed
affine functions [f1, f2 . f1, f3 . f2 . f1, ...].
The operator is reversed composition because scanl' threads the accumulator
on the left, but we want the rightmost function applied first.
assocSSM :: Double -> [Aff] -> [Double] Source #
Apply the associatively-scanned affine functions to an initial state.
chunkedScan :: Int -> [Aff] -> [Aff] Source #
Tree-shaped associative scan with a given chunk size.
Each chunk is reduced to a summary affine function, the summaries are
prefix-composed, and the result is expanded back to a per-step prefix. If
affComp is associative, this must agree with assocScan for every chunk
size. This is the genuine parallelisation principle behind S4 Mamba
linear attention.
Vector (harpie) affine SSM
Elementwise affine function on a harpie 1-D array:
h_i -> a_i * h_i + b_i. This is the diagonal-matrix special case of the
full matrix SSM; it already exercises the associative-scan law on arrays.
affCompVec :: AffVec -> AffVec -> AffVec Source #
Elementwise composition of diagonal affine functions.
assocScanVec :: [AffVec] -> [AffVec] Source #
Associative scan on vector affine coefficients. The shape is inherited from the first input.
assocSSMVec :: Array Double -> [AffVec] -> [Array Double] Source #
Apply the associatively-scanned vector affine functions to an initial state.
System view
ssmSystemVec :: System (->) (Array Double) (Mono AffVec (Array Double)) Source #
Vector SSM as a 'System (->)' with harpie state, input AffVec, and full
state observation.
Multi-head System view (Dirichlet tensor)
multiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double))) Source #
Two independent vector SSM heads packaged as a single System over the
Dirichlet tensor Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double)).
The two heads share the same input direction type (AffVec) but receive
independent direction values. This is the right polynomial for parallel
layers: both heads fire on the same tick, each with its own input. A
cartesian Prod would force a choice between heads via Either directions.
runMultiHeadSSMSystem :: (Array Double, Array Double) -> [(AffVec, AffVec)] -> ([(Array Double, Array Double)], (Array Double, Array Double)) Source #
Run the multi-head SSM with independent per-head inputs.
Returns the pair of head outputs at each step and the final pair of states.
runSharedInputMultiHeadSSMSystem :: (Array Double, Array Double) -> [AffVec] -> ([(Array Double, Array Double)], (Array Double, Array Double)) Source #
Run the multi-head SSM with the same input supplied to both heads.
This is the diagonal shared-input case; building it from independent inputs requires an explicit copy of the input direction.
Process / System view
ssmProcess :: Process Aff Double Source #
A Process whose state is the hidden state h and whose output is h.
Input is the affine coefficient pair (a_t, b_t).
This is the first-input-seeded presentation with h0 = 0. Use
ssmSystem with systemToProcess when you need a non-zero seed.
Centrality pair (multi-head coupling)
coupledMultiHeadSSMSystem :: System (->) (Array Double, Array Double) ('Tensor (Mono AffVec (Array Double)) (Mono AffVec (Array Double))) Source #
Two-headed SSM with a cross-head coupling: head 1 reads head 2's state.
This breaks premonoidal centrality: the order in which the two heads are threaded through the shared medium matters, because head 1's update depends on head 2's state. It is the "flip" oracle for the multi-head centrality pair.