Fix zero outer-reverse second derivatives of saturated tanh by hoisting the tangent out of its JVP rule - #40319
Open
kodlan wants to merge 1 commit into
Open
Conversation
…ng the tangent out of its JVP rule Fixes jax-ml#40315. For f(v) = tanh(u) * u * u with u = -1e8 * v, the second derivative at v = 1 is -2e16 and jacfwd(jacfwd) computes it exactly, but grad(grad) and any other combination with reverse mode outermost returned -0.0. tanh(u) rounds to exactly -1 there, and the correct value flows through terms that survive that rounding, so this was a forward/reverse asymmetry rather than expected precision loss; clip and erf in the same program shape were unaffected. The cause is the exact form of the default tanh JVP rule, the Herbie rewrite from de8df3a: (g + g*ans) * (1 - ans). It distributes the tangent g inside the (1 + ans) factor. Evaluated forward, g*(1 + ans) collapses to an exact zero early (ans is exactly -1) and everything downstream stays small. The transpose, which is what an outer grad evaluates, multiplies the cotangent by the large residuals first (intermediates around 1e24 for the repro), so the true answer term at 2e8 scale is absorbed by rounding before the matching pair cancels to zero. The transpose is mathematically exact; the evaluation order it induces is catastrophic. The bug only fired at negative saturation: at ans = +1 the zero factor (1 - ans) sits outside the add and transposes cleanly. The fix hoists g out: g * ((1 - ans) * (1 + ans)). The tangent path becomes a single multiplication by a factor computed purely from the primal, so the transpose sees the exact zero immediately, matching the structure of the neighboring logistic rule. Same operation count, and the compiled HLO for a tanh-MLP gradient is identical op for op. The AccuracyMode.HIGHEST variant is untouched. First-derivative numerics: forward-mode JVPs with unit tangents are bitwise identical to before; reverse-mode grad(tanh) transposes to a differently associated expression and can shift in the last bit, staying within each form's error band, and its worst case strictly improves - the old g + g*ans add cancels at g's magnitude with unbounded relative error deep in the negative saturation tail, while 1 + ans is exact by the Sterbenz lemma for ans in [-1, -0.5]. The regression test checks grad(grad), grad(jacfwd) and jacfwd(jacfwd) agree on the exact second derivative at a saturated point, eager and jit, at a float32-safe scale (tanh(-1e4) is exactly -1 in every float dtype and -2e8 is exactly representable in f32); it fails before this change and passes after. Verified: second and third derivatives correct at both saturation signs under grad/jacfwd combinations, vmap, scan, remat, and jit on CPU and GPU for f16/bf16/f32/f64; check_grads order 2 passes for real and complex tanh; lax_autodiff (CPU, CPU+x64, GPU), nn, api, and jet suites green.
kodlan
force-pushed
the
fix/issue-40315-saturated-tanh-second-grad
branch
from
August 30, 2026 18:50
115b5d0 to
1e8bc3a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40315.
For f(v) = tanh(u) * u * u with u = -1e8 * v, the second derivative at v = 1 is -2e16 and jacfwd(jacfwd) computes it exactly, but grad(grad) and any other combination with reverse mode outermost returned -0.0. tanh(u) rounds to exactly -1 there, and the correct value flows through terms that survive that rounding, so this was a real forward/reverse asymmetry, not expected precision loss. Notably clip and erf in the same program shape were fine; only tanh misbehaved.
The cause is the exact form of the default tanh JVP rule, the Herbie rewrite from 2020: (g + gans) * (1 - ans). It distributes the tangent g inside the (1 + ans) factor. Evaluated forward, g(1 + ans) collapses to an exact zero early (ans is exactly -1) and everything stays small. The transpose, which is what an outer grad evaluates, multiplies the cotangent by the large residuals first (intermediates around 1e24 for the repro), so the true answer term at 2e8 scale is absorbed by rounding before the matching pair cancels to zero. The transpose is mathematically exact; the evaluation order it induces is catastrophic.
The fix hoists g out: g * ((1 - ans) * (1 + ans)). The tangent path becomes a single multiplication by a factor computed purely from the primal, so the transpose sees the exact zero immediately, like the logistic rule next to it already does. Same operation count, and the compiled HLO for a tanh-MLP gradient is identical op for op (XLA fuses both forms the same way). The AccuracyMode.HIGHEST variant is untouched.
On first-derivative numerics, being fully upfront for anyone with golden gradient values: forward-mode JVPs with unit tangents are bitwise identical to before (checked over 200k points in f32 and f64), but reverse-mode grad(tanh) transposes to a differently associated expression and shifts in the last bit for many inputs. The shifts stay inside each form's error band, and the worst case strictly improves: the old g + g*ans add cancels at g's magnitude, with unbounded relative error deep in the negative saturation tail (observed errors up to about 1), while the new form stays within about 2 ulp of the value implied by the rounded primal everywhere we sampled, since 1 + ans is exact by the Sterbenz lemma for ans in [-1, -0.5].
The regression test checks grad(grad), grad(jacfwd) and jacfwd(jacfwd) agree on the exact second derivative at a saturated point, eager and jit, at a float32-safe scale; it fails before this change (-0.0) and passes after.