Safe Haskell	None
Language	GHC2024

Circuit.Diff.Param

Contents

Parameterised differentiable arrow
Primitive contract
Wiring helpers
Relationship to the phantom-tagged Diff arrow

Description

Parameterised reverse-mode automatic differentiation.

DiffP is a differentiable operation with parameters p, input a and output b. Running forward produces the output; the backward pass, given a cotangent on the output, produces the input cotangent and parameter gradients.

This is the same shape as Diff from circuits-ad, but with an explicit parameter carrier p. When p = () and the parameter gradient is discarded, DiffP reduces to Diff (see toParam and fromParam).

The representation is the denotation p -> a -> (b, b -> (a, p)) rather than a pair of separate forward/backward fields, so it stays aligned with the monomial/lens reading of AD.

Synopsis

newtype DiffP p a b = DiffP {
- runDiffP :: p -> a -> (b, b -> (a, p))
}
data TensorPrim p a b = TensorPrim {
- primForward :: p -> a -> b
- primBackward :: p -> a -> b -> (a, p)
}
fromPrim :: TensorPrim p a b -> DiffP p a b
toPrim :: DiffP p a b -> TensorPrim p a b
residual :: Num a => DiffP p a a -> DiffP p a a
splitP :: DiffP p1 a1 b1 -> DiffP p2 a2 b2 -> DiffP (p1, p2) (a1, a2) (b1, b2)
joinP :: Num a => DiffP p1 a b1 -> DiffP p2 a b2 -> DiffP (p1, p2) a (b1, b2)
toParam :: forall {k} (q :: k) a b. Diff q a b -> DiffP () a b
fromParam :: DiffP () a b -> Diff () a b

Parameterised differentiable arrow

newtype DiffP p a b Source #

A differentiable operation with parameters p, input a and output b.

The forward pass closes over the parameter p and input a, producing an output b and a pullback. The pullback maps an output cotangent db to an input cotangent da and a parameter gradient dp.

This is a point-dependent lens: the parameter is part of the position, not an extra layer.

Constructors

DiffP
Fields runDiffP :: p -> a -> (b, b -> (a, p))

Instances

Instances details

MergeZero (->) p => Channel (,) (DiffP p :: Type -> Type -> Type) Source #	Structural associator for nested pairs. No parameters are touched. The superclass 'Category (DiffP p)' requires 'MergeZero (->) p', so the instance carries the same constraint even though the structural maps themselves ignore the parameter.
Instance details Defined in Circuit.Diff.Param Methods assoc :: DiffP p ((a, b), c) (a, (b, c)) # assoc' :: DiffP p (a, (b, c)) ((a, b), c) # slide :: DiffP p (a, (b, c)) (b, (a, c)) #
MergeZero (->) p => Action (,) (DiffP p :: Type -> Type -> Type) Source #
Instance details Defined in Circuit.Diff.Param Methods braid :: DiffP p (a, b) (b, a) #
MergeZero (->) p => Tensor (,) (DiffP p :: Type -> Type -> Type) Source #
Instance details Defined in Circuit.Diff.Param Methods tensor :: DiffP p a b -> DiffP p c d -> DiffP p (a, c) (b, d) #
MergeZero (->) p => Unital (,) (DiffP p :: Type -> Type -> Type) Source #	Parallel composition runs both arrows with the same parameter `p` and combines their parameter gradients. The parameter is treated as an input value, not a state, so it can be passed to both branches without a comonoid structure. Only the output gradients need `plus`. `>>>` `let inc = DiffP (\() x -> (x + 1, \dy -> (dy, ()))) :: DiffP () Int Int``>>>` *`let dbl = DiffP (\() x -> (2 x, \dy -> (2 * dy, ()))) :: DiffP () Int Int``>>>` `let (y, pb) = runDiffP (tensor inc dbl) () (3, 4)``>>>` `y`(4,8) `>>>` `pb (1, 1)`**((1,2),())
Instance details Defined in Circuit.Diff.Param Methods unitl :: DiffP p (Unit (,), a) a # unitl' :: DiffP p a (Unit (,), a) # unitr :: DiffP p (a, Unit (,)) a # unitr' :: DiffP p a (a, Unit (,)) #
MergeZero (->) p => Category (DiffP p :: Type -> Type -> Type) Source #	Sequential composition threads the same parameter `p` through both arrows and combines the parameter gradients with the monoid structure on `p`. Identity returns a zero parameter gradient. This instance needs `MergeZero (->) p` because the parameter type must supply both `zero` (for `id`) and `plus` (for composing gradients). `>>>` `let inc = DiffP (\() x -> (x + 1, \dy -> (dy, ()))) :: DiffP () Int Int``>>>` *`let dbl = DiffP (\() x -> (2 x, \dy -> (2 * dy, ()))) :: DiffP () Int Int``>>>` `let (y, pb) = runDiffP (dbl . inc) () 3``>>>` `y`8 `>>>` `pb 1`(2,()) Parameter gradients from both composed arrows accumulate (not just the outer one). `addParam` contributes `dpG = dy`; `dblParam` contributes `dpF = dc`; composition must sum them via `plus`. `>>>` `let addParam = DiffP (\p x -> (x + p, \dy -> (dy, dy))) :: DiffP Int Int Int``>>>` `let dblParam = DiffP (\p b -> (2 * b, \dc -> (2 * dc, dc))) :: DiffP Int Int Int``>>>` `let (y, pb) = runDiffP (dblParam . addParam) 10 3``>>>` `y`26 `>>>` `pb 1`**(2,3)
Instance details Defined in Circuit.Diff.Param Methods id :: DiffP p a a # (.) :: DiffP p b c -> DiffP p a b -> DiffP p a c #
(Zero (->) a, Zero (->) p) => Discard (DiffP p :: Type -> Type -> Type) (a :: Type) Source #
Instance details Defined in Circuit.Diff.Param Methods discard :: DiffP p a () #
(Zero (->) a, Zero (->) p) => Zero (DiffP p :: Type -> Type -> Type) (a :: Type) Source #
Instance details Defined in Circuit.Diff.Param Methods zero :: DiffP p () a #
(Merge (->) a, Zero (->) p) => Copy (DiffP p) a Source #	Copy in `DiffP`: forward copy, backward add. The parameter gradient is zero because copy has no parameters. `>>>` `let (_, pb) = runDiffP copy () (5 :: Int)``>>>` `pb (1, 2)`(3,())
Instance details Defined in Circuit.Diff.Param Methods copy :: DiffP p a (a, a) #
(Merge (->) a, Zero (->) p) => Merge (DiffP p) a Source #	Add in `DiffP`: forward add, backward copy. The parameter gradient is zero because addition has no parameters. `>>>` `let (_, pb) = runDiffP plus () ((3, 4) :: (Int, Int))``>>>` `pb 1`((1,1),())
Instance details Defined in Circuit.Diff.Param Methods plus :: DiffP p (a, a) a #

Primitive contract

data TensorPrim p a b Source #

A primitive contract with separate forward and backward fields, easier to read and write for dense linear-algebra primitives.

Constructors

TensorPrim
Fields primForward :: p -> a -> b primBackward :: p -> a -> b -> (a, p)

fromPrim :: TensorPrim p a b -> DiffP p a b Source #

Convert a TensorPrim into a DiffP.

toPrim :: DiffP p a b -> TensorPrim p a b Source #

Convert a DiffP into a TensorPrim.

Wiring helpers

residual :: Num a => DiffP p a a -> DiffP p a a Source #

Add a residual connection around an operation.

forward: y = x + op(x) backward: dx = dy + dOp

splitP :: DiffP p1 a1 b1 -> DiffP p2 a2 b2 -> DiffP (p1, p2) (a1, a2) (b1, b2) Source #

Split a parameter tuple so the left and right halves can be used by independent parallel branches.

This is the parameter-product combinator that the fixed-parameter Category instance cannot express. It keeps the migration semantics identical to the original circuits-llm DiffP.

joinP :: Num a => DiffP p1 a b1 -> DiffP p2 a b2 -> DiffP (p1, p2) a (b1, b2) Source #

Pair two operations that share the same input type but produce independent outputs. Input cotangents are added; parameter gradients are paired.

Relationship to the phantom-tagged Diff arrow

toParam :: forall {k} (q :: k) a b. Diff q a b -> DiffP () a b Source #

Embed a phantom-tagged Diff into 'DiffP ()'. The phantom tag is discarded because DiffP carries parameters at the value level.

>>> let d = Diff (\x -> (x * x, \dy -> 2 * x * dy)) :: Diff () Double Double
>>> let (y, pb) = runDiffP (toParam d) () 3.0
>>> y
9.0
>>> pb 1.0
(6.0,())

fromParam :: DiffP () a b -> Diff () a b Source #

Project a parameter-free 'DiffP ()' back into the phantom-tagged Diff arrow.

This is a section/retract pair with toParam: fromParam . toParam = id for the parameter-free fragment.

>>> let d = Diff (\x -> (x * x, \dy -> 2 * x * dy)) :: Diff () Double Double
>>> let (y, pb) = runDiff (fromParam (toParam d)) 3.0
>>> y
9.0
>>> pb 1.0
6.0