Skip to content

Fix zero outer-reverse second derivatives of saturated tanh by hoisting the tangent out of its JVP rule - #40319

Open
kodlan wants to merge 1 commit into
jax-ml:mainfrom
kodlan:fix/issue-40315-saturated-tanh-second-grad
Open

Fix zero outer-reverse second derivatives of saturated tanh by hoisting the tangent out of its JVP rule#40319
kodlan wants to merge 1 commit into
jax-ml:mainfrom
kodlan:fix/issue-40315-saturated-tanh-second-grad

Conversation

@kodlan

@kodlan kodlan commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #40315.

For f(v) = tanh(u) * u * u with u = -1e8 * v, the second derivative at v = 1 is -2e16 and jacfwd(jacfwd) computes it exactly, but grad(grad) and any other combination with reverse mode outermost returned -0.0. tanh(u) rounds to exactly -1 there, and the correct value flows through terms that survive that rounding, so this was a real forward/reverse asymmetry, not expected precision loss. Notably clip and erf in the same program shape were fine; only tanh misbehaved.

The cause is the exact form of the default tanh JVP rule, the Herbie rewrite from 2020: (g + gans) * (1 - ans). It distributes the tangent g inside the (1 + ans) factor. Evaluated forward, g(1 + ans) collapses to an exact zero early (ans is exactly -1) and everything stays small. The transpose, which is what an outer grad evaluates, multiplies the cotangent by the large residuals first (intermediates around 1e24 for the repro), so the true answer term at 2e8 scale is absorbed by rounding before the matching pair cancels to zero. The transpose is mathematically exact; the evaluation order it induces is catastrophic.

The fix hoists g out: g * ((1 - ans) * (1 + ans)). The tangent path becomes a single multiplication by a factor computed purely from the primal, so the transpose sees the exact zero immediately, like the logistic rule next to it already does. Same operation count, and the compiled HLO for a tanh-MLP gradient is identical op for op (XLA fuses both forms the same way). The AccuracyMode.HIGHEST variant is untouched.

On first-derivative numerics, being fully upfront for anyone with golden gradient values: forward-mode JVPs with unit tangents are bitwise identical to before (checked over 200k points in f32 and f64), but reverse-mode grad(tanh) transposes to a differently associated expression and shifts in the last bit for many inputs. The shifts stay inside each form's error band, and the worst case strictly improves: the old g + g*ans add cancels at g's magnitude, with unbounded relative error deep in the negative saturation tail (observed errors up to about 1), while the new form stays within about 2 ulp of the value implied by the rounded primal everywhere we sampled, since 1 + ans is exact by the Sterbenz lemma for ans in [-1, -0.5].

The regression test checks grad(grad), grad(jacfwd) and jacfwd(jacfwd) agree on the exact second derivative at a saturated point, eager and jit, at a float32-safe scale; it fails before this change (-0.0) and passes after.

…ng the tangent out of its JVP rule

Fixes jax-ml#40315.

For f(v) = tanh(u) * u * u with u = -1e8 * v, the second derivative at v = 1
is -2e16 and jacfwd(jacfwd) computes it exactly, but grad(grad) and any other
combination with reverse mode outermost returned -0.0. tanh(u) rounds to
exactly -1 there, and the correct value flows through terms that survive that
rounding, so this was a forward/reverse asymmetry rather than expected
precision loss; clip and erf in the same program shape were unaffected.

The cause is the exact form of the default tanh JVP rule, the Herbie rewrite
from de8df3a: (g + g*ans) * (1 - ans). It distributes the tangent g inside
the (1 + ans) factor. Evaluated forward, g*(1 + ans) collapses to an exact
zero early (ans is exactly -1) and everything downstream stays small. The
transpose, which is what an outer grad evaluates, multiplies the cotangent by
the large residuals first (intermediates around 1e24 for the repro), so the
true answer term at 2e8 scale is absorbed by rounding before the matching
pair cancels to zero. The transpose is mathematically exact; the evaluation
order it induces is catastrophic. The bug only fired at negative saturation:
at ans = +1 the zero factor (1 - ans) sits outside the add and transposes
cleanly.

The fix hoists g out: g * ((1 - ans) * (1 + ans)). The tangent path becomes a
single multiplication by a factor computed purely from the primal, so the
transpose sees the exact zero immediately, matching the structure of the
neighboring logistic rule. Same operation count, and the compiled HLO for a
tanh-MLP gradient is identical op for op. The AccuracyMode.HIGHEST variant is
untouched.

First-derivative numerics: forward-mode JVPs with unit tangents are bitwise
identical to before; reverse-mode grad(tanh) transposes to a differently
associated expression and can shift in the last bit, staying within each
form's error band, and its worst case strictly improves - the old g + g*ans
add cancels at g's magnitude with unbounded relative error deep in the
negative saturation tail, while 1 + ans is exact by the Sterbenz lemma for
ans in [-1, -0.5].

The regression test checks grad(grad), grad(jacfwd) and jacfwd(jacfwd) agree
on the exact second derivative at a saturated point, eager and jit, at a
float32-safe scale (tanh(-1e4) is exactly -1 in every float dtype and -2e8 is
exactly representable in f32); it fails before this change and passes after.

Verified: second and third derivatives correct at both saturation signs under
grad/jacfwd combinations, vmap, scan, remat, and jit on CPU and GPU for
f16/bf16/f32/f64; check_grads order 2 passes for real and complex tanh;
lax_autodiff (CPU, CPU+x64, GPU), nn, api, and jet suites green.
@kodlan
kodlan force-pushed the fix/issue-40315-saturated-tanh-second-grad branch from 115b5d0 to 1e8bc3a Compare August 30, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant