Automatic Differentiation Can Produce Wrong Gradients

The problem: the sum rule fails for nonsmooth functions

For smooth functions, the sum rule holds as an equality: $(f+g)'(x) = f'(x) + g'(x)$. AD relies on this: it differentiates each subexpression independently and adds the results. For nonsmooth functions, the sum rule for the Clarke subdifferential is only an inclusion:

The right-hand side can be strictly larger than the left. AD effectively picks one element from $\partial_C f(x)$ and one from $\partial_C g(x)$ and adds them, producing an element of the right-hand side. But this element may not belong to $\partial_C(f+g)(x)$ on the left. The following example makes this concrete.

(For the limiting subdifferential $\partial f$, the sum rule $\partial(f+g)(x) \subseteq \partial f(x) + \partial g(x)$ can even fail entirely without qualification conditions for nonconvex functions.)

Example: AD output outside the Clarke subdifferential

Setup. Consider two programs that both compute $\mathrm{relu}(t) = \max(0,t)$:

• $\mathrm{relu}(t) = \max(0,t)$, implemented with branches $f_1=0$, $f_2=t$ and index $s(0)=1$ (selects $f_1$ at $t=0$).
• $\mathrm{relu}_2(t) = \mathrm{relu}(-t) + t$, a different program computing the same function.

As functions, $\mathrm{relu} = \mathrm{relu}_2$ on all of $\mathbb{R}$. But AD gives different outputs at $t=0$:

Both $0$ and $1$ lie in $\partial_C\mathrm{relu}(0) = [0,1]$, so individually there is no problem.

Now subtract. Define $g(t) = \mathrm{relu}_2(t) - \mathrm{relu}(t)$. As a function: $g \equiv 0$ on all of $\mathbb{R}$, so $\partial_C g(0) = \{0\}$.

AD applies the sum rule as an equality and differentiates each piece independently:

What went wrong, precisely. AD computed an element of $\partial_C\mathrm{relu}_2(0) - \partial_C\mathrm{relu}(0) = [0,1] - [0,1] = [-1,1]$. The actual Clarke subdifferential is $\partial_C g(0) = \partial_C(\mathrm{relu}_2 - \mathrm{relu})(0) = \{0\}$. The inclusion is strict:

AD picked $1 \in [0,1]$ and $0 \in [0,1]$, giving $1 - 0 = 1 \in [-1,1]$, but $1 \notin \{0\}$. The sum rule inclusion allowed AD to land in the gap $[-1,1] \setminus \{0\}$.

How this arises in practice

The $\mathrm{relu}_2 - \mathrm{relu}$ construction is not artificial. In modern deep learning architectures:

• Skip connections (ResNets): the output is $\mathrm{layer}(x) + x$. If $\mathrm{layer}(x) = -x$ at some input (which can happen for a linear layer with weight $-I$), the forward pass gives $0$, but backpropagation computes the gradient of each path separately.
• Residual blocks: $h(x) = f(x) + (g(x) - g(x))$ might appear after algebraic simplification. The two copies of $g$ may follow different computational paths in the graph, producing different AD outputs at nonsmooth points.
• Equivalent reformulations: practitioners routinely rewrite $\mathrm{relu}(x)$ as $\tfrac{1}{2}(x + |x|)$ or $x \cdot \mathbf{1}_{x>0}$ for numerical reasons. These are the same function but different programs, and AD treats them differently at $x = 0$.

In all these cases, at the measure-zero set of inputs where a neuron sits exactly at a ReLU kink, the AD gradient depends on the program representation.

Consequence: no subdifferential operator captures AD

One might try to define a "universal" operator $\partial_A$ that contains all possible AD outputs for all programs computing a given function. Bolte & Pauwels (NeurIPS 2020, Theorem 1) prove this is impossible:

Theorem 1. There is no nontrivial operator $\partial_A$ on locally Lipschitz functions satisfying both:

1. $\partial_A$ encompasses AD: for every program $P$ computing $f$, the AD output at $x$ lies in $\partial_A f(x)$;
2. $0 \in \partial_A \mathrm{relu}(0)$ (the operator detects the minimum of ReLU at $0$).

Proof sketch. By (a), $\partial_A$ must contain all possible AD outputs. The example above shows AD can output $1$ for $g \equiv 0$, so $1 \in \partial_A g(0)$. If $\partial_A$ respects the sum rule, then $\partial_A g(0) \supseteq \partial_A \mathrm{relu}_2(0) - \partial_A \mathrm{relu}(0)$. Since $\mathrm{relu}_2 = \mathrm{relu}$ as functions, $\partial_A$ (which acts on functions, not programs) must assign the same set to both: $\partial_A \mathrm{relu}_2(0) = \partial_A \mathrm{relu}(0)$. So $1 \in \partial_A \mathrm{relu}(0) - \partial_A \mathrm{relu}(0)$, which forces $\partial_A \mathrm{relu}(0)$ to contain elements differing by $1$ (e.g., both $0$ and $1$). Repeating with other programs ($\mathrm{relu}_3$, etc.) forces $\partial_A \mathrm{relu}(0)$ to grow without bound, making $\partial_A$ trivially large and useless as a criticality certificate.

The solution: conservative fields

Since no subdifferential operator captures AD, Bolte & Pauwels introduce conservative fields: set-valued maps satisfying a pathwise chain rule rather than pointwise subdifferential inclusions.

Setup. Let $f:\mathbb{R}^n\to\mathbb{R}$ be the cost function to be minimized (e.g., training loss of a neural network). Suppose $f$ is implemented as a program that selects among smooth branches $f_1, \ldots, f_m$ (elementary functions such as affine maps, $\exp$, $\log$, etc.) via an index function $s(x) \in \{1,\ldots,m\}$, so that $f(x) = f_{s(x)}(x)$ for all $x$. At any input $x$, several branches may agree with $f$: define the active index set $I(x) = \{i : f_i(x) = f(x)\}$.

Conservative field for $f$. The representation-minimal operator is the set-valued map

the convex hull of the gradients of all active branches at $x$. This set contains every possible AD output at $x$ (since AD returns $\nabla f_{s(x)}(x)$ for whichever branch $s(x)$ the program selects, and $s(x) \in I(x)$). Bolte & Pauwels prove that $D_f$ is a conservative field for $f$: it satisfies a chain rule along absolutely continuous curves (NeurIPS 2020, Theorem 3).

Hierarchy of generalized derivatives. For $f \in \mathcal{S}$ (elementary selection, e.g., ReLU networks with smooth losses), the four notions are nested:

• $\hat{\partial}f(x)$ — Fréchet subdifferential: local linear lower bounds.
• $\partial f(x)$ — limiting (Mordukhovich) subdifferential: closure of Fréchet subgradients with $f$-value convergence. Used in Theorem 2.9 (H2, KL inequality). See the nonsmooth proof page.
• $\partial_C f(x)$ — Clarke subdifferential: closed convex hull of $\partial f(x)$; equivalently, $\mathrm{conv}\{\lim\nabla f(y^k):y^k\to x,\,y^k\in\mathrm{diff}(f)\}$.
• $D_f(x)$ — conservative field from the program representation: $\mathrm{conv}\{\nabla f_i(x):i\in I(x)\}$. Contains every AD output.

The last inclusion $\partial_C f(x) \subseteq D_f(x)$ can be strict (NeurIPS 2020, Theorem 3; NeurIPS 2021, Theorem 1). The gap $D_f(x) \setminus \partial_C f(x)$ contains the “spurious” AD outputs demonstrated in the example above (e.g., $1 \in D_g(0)$ but $1 \notin \partial_C g(0) = \{0\}$).

Examples at $x = 0$ (scalar functions). $D_f$ depends on the program representation; the column below lists the natural representation.

Verification of the last row. As a function, $g(t)\equiv 0$, so the first three subdifferentials at $0$ equal $\{0\}$. The program $g = \mathrm{relu}_2 - \mathrm{relu}$ has four effective branches at $t=0$ (two branch choices per $\mathrm{relu}$, minus the shared sign): $t\mapsto t$ (derivative $1$), $t\mapsto 0$ (derivative $0$ in two ways), and $t\mapsto -t$ (derivative $-1$). All four branches satisfy $f_j(0)=0=g(0)$, so $I(0)=\{1,2,3,4\}$ and $D_g(0) = \mathrm{conv}\{1,0,0,-1\} = [-1,1]$. This is strictly larger than $\partial_C g(0)=\{0\}$: the gap $[-1,1]\setminus\{0\}$ contains every spurious AD output.

Key results (NeurIPS 2020, Theorems 3–4):

• Lyapunov property: $f$ always decreases along the flow $\dot{x} \in -D_f(x)$. So gradient descent with AD (which uses elements of $D_f$) always decreases the cost function $f$.
• Convergence: accumulation points of SGD with AD are selection critical ($0 \in D_f(\bar{x})$).
• Spurious points are harmless: the "spurious" critical points ($0 \in D_f(x)$ but $0 \notin \partial_C f(x)$) form a measure-zero set and are avoided by almost all initializations. For almost all starting points, accumulation points are Clarke critical ($0 \in \partial_C f(\bar{x})$).

Concrete example: ReLU CNN for classification

Consider training a ReLU convolutional neural network for classification with cross-entropy loss:

Branch structure. Each ReLU node selects between $t\mapsto 0$ (off) and $t\mapsto t$ (on) based on the sign of its pre-activation. A network with $R$ ReLU nodes has up to $2^R$ activation patterns. Each pattern gives a smooth branch $f_j(w)$: a composition of affine maps and cross-entropy (all $C^\infty$), with the ReLU nodes replaced by their selected linear pieces. The index function $s(w)$ records which activation pattern is active at weights $w$, so $f(w) = f_{s(w)}(w)$.

What backpropagation computes. AD (backprop) computes $\nabla f_{s(w)}(w)$, the gradient of the active smooth branch. This is a selection derivative: it belongs to $D_f(w) = \mathrm{conv}\{\nabla f_j(w) : j \in I(w)\}$.

Does Theorem 2.9 (KL) apply? Theorem 2.9 has two kinds of hypotheses: one on the function (proper l.s.c., bounded below, KL property) and one on the iterates produced by the algorithm (H1 sufficient decrease, H2 relative error in $\partial f$, H3 continuity).

On the function: the ReLU loss $f$ is continuous, bounded below, and semi-algebraic (ReLU is semi-algebraic; cross-entropy is log-exp definable; compositions and finite averages preserve definability). Hence $f$ is a KL function (Bolte–Daniilidis–Lewis 2007). So the function-class hypotheses of Theorem 2.9 are satisfied.

On the iterates: the obstruction for ReLU networks is producing iterates that satisfy H1 and H2 with respect to the limiting subdifferential.

So Theorem 2.9 applies to the ReLU loss function (it is a KL function), but no practical algorithm on a ReLU network is known to satisfy H1 and H2 with respect to the limiting subdifferential. The conservative field framework (Theorem 4 below) circumvents this by replacing $\partial f$ with $D_f$ and using a different discrete-to-continuous bridge. See the nonsmooth proof page for a smooth-activation classification example where GD does satisfy all of H1, H2, H3 directly.

Why Theorem 4 (conservative gradients) does apply. The cost function $f$ is definable (cross-entropy is log-exp definable, ReLU is semialgebraic), so $f$ is path differentiable and $D_f$ is a conservative field. Theorem 4 guarantees: SGD with backpropagation converges, and accumulation points are Clarke critical ($0 \in \partial_C f(\bar{w})$) for almost all initializations. No smoothness of the activation function is required.

Bolte & Pauwels, Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning, NeurIPS 2020, §1.1, Theorems 1–4.