Weierstrass Theorem & Taylor Series

We just saw that any polynomial $P_n(x)$ approximates a function $f(x)$ from a finite-dimensional subspace. Two fundamental results tell us how good that approximation can be: the Weierstrass Approximation Theorem guarantees it can be made arbitrarily precise, and the Taylor Series gives us a concrete method for constructing such approximations.

---

Weierstrass Approximation Theorem (1885)

Theorem 2.1 (Weierstrass, 1885). For any $f \in C([0,1])$ and any $\varepsilon > 0$, there exists a polynomial $p(x)$ such that:

$\max_{0 \le x \le 1} |f(x) - p(x)| \le \varepsilon$

In plain terms: any continuous function on a closed interval can be approximated to within any desired accuracy by a polynomial of sufficiently high degree.

The Error Trade-off

If we have $f(x) = 2 + 3x + 4x^2 + 5x^3 + 8x^4$ but can only afford $P_2(x) = a_0 + a_1 x + a_2 x^2$, we inevitably carry error from the dropped higher-degree terms.

The key principle is: higher degree → lower error. As $n$ increases, the polynomial can track $f(x)$ more closely:

$\text{as } n \uparrow \qquad |f(x) - P_n(x)| \downarrow$

Note

The Weierstrass theorem is an existence result. It guarantees such a polynomial exists, but does not say how to build it. Taylor series, Lagrange interpolation, and Newton divided differences are the practical tools we use to actually construct these approximations.

---

Taylor Series

The Taylor Series transforms the intuition behind a tangent-line approximation into a precise, high-order polynomial expansion.

Motivation: The Tangent-Line Approximation

Given a point $x_0$ and the value $f(x_0)$, suppose we also know the gradient $f'(x_0)$. The tangent line at $x_0$ gives a first-order approximation:

$f'(x_0) = \frac{f(x) - f(x_0)}{x - x_0} \implies f(x) \approx f(x_0) + f'(x_0)(x - x_0)$

As the figure shows, the tangent line accurately predicts the function’s value near $x_0$. However, as we move to a farther point $x$, the curve bends away from the straight line, introducing an approximation error.

This tangent line is exact only for linear functions. To do better and close that error gap, we must also include curvature $f''(x_0)$, then the rate of change of curvature $f'''(x_0)$, and so on, giving the full Taylor Series.

The Infinite Taylor Series

$f(x) = f(x_0) + f'(x_0)(x - x_0) + \frac{f''(x_0)}{2!}(x - x_0)^2 + \frac{f'''(x_0)}{3!}(x - x_0)^3 + \cdots$

Each successive term captures one more layer of information about $f$ at the point $x_0$.

---

Worked Example

Find the Taylor expansion for $f(x) = \sin(x)$ about $x_0 = 0$.

Step 1: Evaluate derivatives at $x_0 = 0$:

$k$	$f^{(k)}(x)$	$f^{(k)}(0)$
0	$\sin(x)$	$0$
1	$\cos(x)$	$1$
2	$-\sin(x)$	$0$
3	$-\cos(x)$	$-1$
4	$\sin(x)$	$0$
5	$\cos(x)$	$1$

The zero terms drop out, leaving only odd powers:

$\sin(x) \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots$

Step 2: Evaluate at $x = 0.1$:

$\sin(0.1) \approx 0.1 - \frac{(0.1)^3}{6} + \frac{(0.1)^5}{120} = 0.1 - 0.000\,167 + 0.000\,000\,008 \approx 0.099\,833\,4$

The exact value is also $0.099\,833\,4$ (to 6 significant figures), confirming the expansion is highly accurate with just three terms.

---

Formal Taylor Theorem with Remainder

Theorem (Taylor). Let $f$ be $n+1$ times differentiable on $(a,b)$, with $f^{(n)}$ continuous on $[a,b]$. For any $x, x_0 \in [a,b]$, there exists $\xi \in (a,b)$ such that:

$f(x) = \sum_{k=0}^{n} \frac{f^{(k)}(x_0)}{k!}(x - x_0)^k + \underbrace{\frac{f^{(n+1)}(\xi)}{(n+1)!}(x - x_0)^{n+1}}_{\text{remainder } R_n}$

The last term $R_n$ is the truncation error, the price paid for stopping at degree $n$. It is never zero for non-polynomial functions, but it can always be bounded. Here, $\xi$ is some point between $x_0$ and $x$. We don’t know exactly where, but we know it exists.If we had actually known the point, we could’ve found the exact value of error in the first place! We can at best, find the maximum possible value of $\xi$, and thus $|R_n|$. This allows us to bound the error without needing to know the exact value of $\xi$.

Bounding the Error for $\sin(x)$ at $x = 0.1$

Using $p_6(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!}$, the 7th derivative of $\sin(x)$ is $-\cos(x)$.

The remainder is:

$|f(0.1) - p_6(0.1)| = \left|\frac{f^{(7)}(\xi)}{7!}\right|(0.1)^7 = \frac{|\cos(\xi)|}{5040} \cdot (0.1)^7$

Since $|\cos(\xi)| = 1$ at $\xi = 0$ in the interval $[0, 0.1]$:

$|f(0.1) - p_6(0.1)| \le \frac{(0.1)^7}{5040} \approx 1.984 \times 10^{-11}$

In other words, if we take just the first three terms of the Taylor expansion, we can guarantee that our approximation is accurate to within $2 \times 10^{-11}$ , an incredibly tight bound!

Note

The power of the Taylor remainder is that it lets you bound the error before computing anything. If you know $|f^{(n+1)}(\xi)| \le M$ on the interval, you can guarantee the approximation stays within $\dfrac{M \cdot |x - x_0|^{n+1}}{(n+1)!}$ of the true value.