Dual-number scalar: value plus accumulated gradient.

Multiplication follows the dual-number product rule: the gradient component records the cross-derivative. This naturally gives the REINFORCE coupling: when a continuation multiplies a downstream reward (value component) by a score function (gradient component), the result records the reward-weighted score.

>>> Dual (2, 3) * Dual (5, 7)
Dual (10, 29)

Check: 2·5 = 10, 2·7 + 3·5 = 14+15 = 29 ✓

REINFORCE (score-function) gradient estimator for Gaussian policy.

∇J = E[R(a) · ∇_θ log π(a)] where ∇_θ log π(a) = (a-θ)/σ².

The analytical expectation uses Gaussian central moments. Let z = a-θ, δ = θ-a_target:

  R(a) = -(a - a_target)² = -(z+δ)²
  R(a)(a-θ)σ² = -(z+δ)²·zσ² = -(z³ + 2δz² + δ²z)/σ²
  E[R(a)(a-θ)σ²] = -(1σ²)(E[z³] + 2δ·E[z²] + δ²·E[z])
                  = -(1/σ²)(0 + 2δσ² + 0) = -2δ
                  = -2(θ-a_target)

>>> reinforceGrad 3.0 0.5 1.0
-4.0

Pathwise (reparametrization) gradient estimator for Gaussian policy.

a = θ + σε with ε ~ N(0,1). ∂R∂θ = ∂R∂a · ∂a/∂θ = -2(a - a_target) · 1 = -2(θ+σε-a_target). E[∂R/∂θ] = -2(θ-a_target) since E[ε] = 0.

>>> pathwiseGrad 3.0 0.5 1.0
-4.0

Closed-form gradient for Gaussian policy N(θ, σ²) with quadratic reward R(a) = -(a - a_target)².

J(θ) = E[R(a)] = -(σ² + (θ - a_target)²), so ∇J = -2(θ - a_target).

>>> closedFormGrad 3.0 0.5 1.0
-4.0