Rounding Error and Total Error

Reducing the step size $h$ decreases the truncation error, but there is a lower limit. When $h$ becomes very small, a different source of error dominates: rounding error (also known as loss of significance). Understanding how both errors combine reveals why there is an optimal step size that minimises the total error.

---

Why Small $h$ Amplifies Rounding Error

When $h$ is very small, the values $f(x+h)$ and $f(x-h)$ become nearly identical. For example, if $h = 0.001$:

$f(2 + 0.001) \approx f(2 - 0.001)$

Subtracting two numbers that are almost equal causes catastrophic cancellation: the significant digits cancel out, leaving only the rounding noise in the least significant bits. This noise is then divided by the tiny value $2h$, which amplifies the error severely.

---

The Floating-Point Error Model

Based on the floating-point representation $fl(x) = (1 + \delta)x$ (where $\delta$ is the relative rounding error), evaluating the function at the shifted points introduces small multiplicative errors:

$fl[f(x+h)] = (1 + \delta_1)\,f(x+h)$

$fl[f(x-h)] = (1 + \delta_2)\,f(x-h)$

where $|\delta_1|,\, |\delta_2| \leq \varepsilon_M$ and $\varepsilon_M$ is the machine epsilon: the upper bound on the relative rounding error for the floating-point system in use.

---

Total Error Formula

The total error in the central difference approximation is the sum of the truncation error and the rounding error:

$\text{Total Error} \;\leq\; \left|\frac{f'''(\xi)\,h^2}{6}\right| + \varepsilon_M \,\frac{|f(x+h)| + |f(x-h)|}{2h}$

Component	Scales with $h$	Behaviour as $h \to 0$
Truncation error	proportional to $h^2$	Decreases toward zero
Rounding error	proportional to $1/h$	Increases without bound

Note

For IEEE 754 double precision, $\varepsilon_M \approx 2.22 \times 10^{-16}$. Even though this is tiny, dividing by a sufficiently small $h$ can make the rounding contribution dominate the total error.

---

The Optimal Step Size

Because the two error components move in opposite directions as $h$ changes, the total error graph forms a distinct V-shaped (or U-shaped) curve when plotted on a logarithmic scale.

• For large $h$ (right side): Truncation error dominates, and the total error is large.
• For small $h$ (left side): Rounding error from catastrophic cancellation dominates, and the total error shoots back up.
• At the minimum of the curve: An optimal $h$ exists where the two contributions are perfectly balanced, yielding the lowest possible total error.

Caution

Making $h$ as small as possible is not always better. Using a step size well below the optimal value can actually increase the numerical error rather than reducing it. The standard guideline for central difference is to choose $h \approx \varepsilon_M^{1/3}$, which mathematically balances the two terms.