Error Analysis for Iterative Methods

We've discussed three iterative methods:

1. Bisection
2. Fixed-point iteration
3. Newton's method

For Bisection, we saw that $p_n = p + O(1/2^n)$. For, fixed-point iteration we have $p_n = p + O(K^n)$ where $K$ is the upper bound on the absolute value of the derivative. And for Newton's method $p_n = p + O(K^n)$ for all $0 < K < 1$.

We introduce some definitions to classify these methods.

Definition (Order of convergence)

Suppose $\{p_n\}_{n=0}^\infty$ is a sequence that converges to $p$ with $p_n \neq p$ for all $n$. If positive constants $\lambda$ and $\alpha$ exist such that

$$\lim_{n \to \infty} \frac{|p_{n+1}-p|}{|p_{n}-p|^\alpha} = \lambda,$$

then $\{p_n\}_{n=0}^\infty$ converges to $p$ to order $\alpha$, with asymptotic order constant $\lambda$.

An iterative method defined by $p_n = g(p_{n-1})$ is said to be of order $\alpha$ if the sequence $\{p_n\}_{n=0}^\infty$ converges to the solution of $p=g(p)$ to order $\alpha$.

Two cases are of particular interest:

1. If $\alpha = 1$ (and $\lambda < 1$) the sequence is linearly convergent.
2. If $\alpha = 2$, the sequence is quadratically convergent.

To see why quadratic convegence is faster: Assume $p_n \to p$, linearly, with constant $\lambda = .5$. Also assume $q_n \to p$, quadratically, with the same constant. Then

$$\lim_{n \to \infty} \frac{|p_{n+1} - p|}{|p_{n}-p|} = .5, \quad \lim_{n \to \infty} \frac{|q_{n+1} - p|}{|q_{n}-p|^2} = .5.$$

Let's assume that $|p_0 -p|, ~|q_0-q| < 1$. Then

$$\frac{|p_{n+1} - p|}{|p_{n}-p|} \approx .5, \quad \frac{|q_{n+1} - p|}{|q_{n}-p|^2} \approx .5$$

Then

$$|p_{n+1} - p|\approx (.5){|p_{n}-p|}, \quad |q_{n+1} - p|\approx (.5){|q_{n}-p|^2}.$$ $$|p_{n+1} - p|\approx (.5){|p_{n}-p|} \approx (.5)^2{|p_{n-1}-p|} \\\approx \cdots \approx (.5)^{n+1}{|p_{0}-p|}$$ $$|q_{n+1} - p|\approx (.5){|q_{n}-p|^2} \approx (.5)^3{|q_{n-1}-p|^4} \approx \\\cdots \approx (.5)^{2^{n}-1}{|p_{0}-p|^{2^n}}$$

n = 5;format long;
[(.5)^(n+1) (.5)^(2^n-1)]

ans =

   0.015625000000000   0.000000000465661

Theorem 1 (Fixed-point order of convergence)

Let $g \in C[a,b]$ be such that $g(x) \in [a,b]$ for all $x \in [a,b]$. Suppose, in addition, that $g'$ exists on $(a,b)$ and there exists $0 \leq K < 1$ such that

$$|g'(x)| \leq K, \quad \text{ for all } ~~ x \in [a,b].$$

If $p$ is the unique fixed point and $g'(p) \neq 0$ then for $p_0 \in [a,b]$ the sequence $p_n \to p$ linearly with asymptotic error constant

$$\lambda = |g'(p)|$$

(if $p_n \neq p$ for all $n$).

Proof

Based on the previous theorem for convergence of fixed-point iteration, we know that $p_{n} \to p$. It then follows that

$$\lim_{n \to \infty} \frac{g(p_n)- g(p)}{p_n - p} = g'(p).$$

But

$$\lim_{n \to \infty} \frac{|g(p_n)- g(p)|}{|p_n - p|} = \lim_{n \to \infty} \frac{|p_{n+1} - p|}{|p_n - p|} = |g'(p)|.$$

It is immediately clear that something special happens when $g'(p) = 0$ for a fixed point $p$. This next theorem helps describe this behavior.

Theorem

Let $p = (a+b)/2 \in (a,b)$ be a fixed point of $g \in C^2[a,b]$. Suppose that $g'(p) = 0$ and $|g'(x)| \leq K < 1$ for $x \in [a,b]$. For $p_0 \in [a,b]$, the sequence $p_n = g(p_{n-1})$ converges at least quadradically to $p$ with asymptotic rate constant $\lambda = |g''(p)|/2$.

Proof

Let $x \in [a,b]$. By the Mean-Value Theorem, for some $\xi$ between $x$ and $p$

$$g(x) = g(p) + g'(\xi)(x-p) = p + g'(\xi)(x-p).$$

Then using that $|x-p| \leq (b-a)/2$ $$g(x) = p + g'(\xi)(x-p) \leq p + |g'(\xi)||x-p| \leq (b+a)/2 + (b-a)/2 = b\\ g(x) = p + g'(\xi)(x-p) \geq p - |g'(\xi)||x-p| \geq (b+a)/2 - (b-a)/2 = a.$$

So, $g(x) \in [a,b]$ for $x \in [a,b]$. We know that $p_n \to p$, the unique fixed point.

We use Taylor's theorem to state that for some $\xi \in [a,b]$ (between $p$ and $x$ actually)

$$g(x) = g(p) + g'(p)(x-p) + \frac{g''(\xi)}{2}(x-p)^2 \\= g(p) + \frac{g''(\xi)}{2}(x-p)^2.$$

If $x = p_{n}$ we find for some $\xi_n$ between $p$ and $p_n$.

$$\frac{p_{n+1}-p}{(p_n-p)^2} = \frac{g''(\xi_n)}{2}.$$

Then $\xi_n \to p$ as $n \to \infty$ (it is closer to $p$ than $p_n$ is) and so

$$\lim_{n \to \infty} \frac{|p_{n+1}-p|}{|p_n-p|^2} = \lim_{n \to \infty} \frac{|g''(\xi_n)|}{2} = \frac{|g''(p)|}{2}.$$

A fixed-point iteration converges quadratically if you have $g(p) = p$ and $g'(p) = 0$.

This is something satisfied by Newton's method with $g(x) = x - f(x)/f'(x)$. Recall that

$$g'(x) = \frac{f(x)f''(x)}{[f'(x)]^2}.$$

Multiple roots

A function $f(x)$ as a zero of mutiplicity $m$ (or of order $m$) at $x = p$ if for $x \neq p$ we can write

$$f(x) = (x-p)^m q(x)$$

where $\lim_{x \to p} q(x) \neq 0$.

The following can be proved using Taylor's Theorem:

Theorem

A function $f \in C^m[a,b]$ has a zero of multiplicity $m$ at $p \in (a,b)$ if and only if

$$0 = f(p) = f'(p) = \cdots = f^{(m-1)}(p), \quad \text{but} \quad f^{(m)}(p) \neq 0.$$

Recall, the Newton's interation function

$$g(x) = x - f(x)/f'(x)$$

.

Assume we want to solve $f(x)= 0$, but $f'(x) = 0$. We construct a new function $\mu(x)$ out of $f(x)$ that has a simple zero. More generally, let $f(x)$ have a zero of multiplicity $m$ at $x = p$ and consider the function

$$\mu(x) = \frac{f(x)}{f'(x)} = \frac{(x-p)^m q(x)}{m(x-p)^{m-1} q(x) + (x-p)^m q'(x)} \\ = (x-p) \frac{q(x)}{m q(x) + (x-p) q'(x)}$$

It follows that $\lim_{x \to p} \mu(x)/(x-p) \neq 0$. And so, to find a zero of $f(x)$ when $f$ has a zero of higher multiplicity we can apply Newton's method to $\mu(x)$:

$$g(x) = x - \frac{\mu(x)}{\mu'(x)} = x - \frac{f(x)f'(x)}{[f'(x)]^2 - f(x)f''(x)}.$$

This can work well in some cases.

Aitken's $\Delta^2$ Method

This is a method to accelerate convergence. Given a sequence $\{p_n\}_{n=1}^\infty$, we want to construct a new sequence $\{\hat p_n\}_{n=1}^\infty$ that converges faster.

Assume

$$\frac{p_{n+1} - p}{p_n-p} \approx \frac{p_{n+2} - p}{p_{n+1}-p}.$$

Solving for $p$ gives

$$p \approx \frac{p_{n+2}p_n - p_{n+1}^2}{p_{n+2} - 2 p_{n+1} + p_n} ~\Rightarrow~ \hat p_n = \frac{p_{n+2}p_n - p_{n+1}^2}{p_{n+2} - 2 p_{n+1} + p_n}$$

$$\hat p_n = \frac{p_{n+2}p_n - p_{n+1}^2}{p_{n+2} - 2 p_{n+1} + p_n}, \quad p_n = \cos(1/n)$$

p = @(n) cos(1/n);
hatp = @(n)  (p(n+2)*p(n) - p(n+1)^2)/(p(n+2)-2*p(n+1) + p(n));
n = 200;
fprintf('n = %i \n',n)
fprintf('Original error = %0.5d \n',abs(p(n+2)-1))
fprintf('Aiken error = %0.5d \n',abs(hatp(n)-1))

n = 200 
Original error = 1.22537e-05 
Aiken error = 4.10740e-06

To see the reason this method is named as it is, define the forward difference

$$\Delta p_n = p_{n+1} - p_n.$$

Define $\Delta^kp_n = \Delta(\Delta^{k-1} p_n)$.

Then, in going back to the defining equation for $\hat p_n$:

$$\frac{p_{n+1} - \hat p_n}{p_n-\hat p_n} = \frac{p_{n+2} - \hat p_n}{p_{n+1}-\hat p_n}.$$

Now, for notational purposes define $q_n = p_n - \hat p_n$ and we write everything in terms of $q_n$:

$$p_{n+1} - \hat p_n = p_{n+1} - p_n + p_n - \hat p_n \\= \Delta p_n + q_n\\ p_{n+2} - \hat p_n = p_{n+2} - p_n + p_n - \hat p_n \\ = p_{n+2} - p_{n+1} + p_{n+1}- p_n + p_n - \hat p_n \\ = \Delta p_{n+1} + \Delta p_n + q_n$$

And then we have

$$\frac{\Delta p_n + q_n}{q_n} = \frac{\Delta p_{n+1} + \Delta p_n + q_n}{\Delta p_n + q_n} \\ \frac{\Delta p_n}{q_n} + 1= \frac{\Delta p_{n+1}}{\Delta p_n + q_n} + 1 \\ \frac{\Delta p_n}{q_n}= \frac{\Delta p_{n+1}}{\Delta p_n + q_n}\\ \Delta p_n (\Delta p_n + q_n) = q_n \Delta p_{n+1} \\ q_n (\Delta p_{n+1} - \Delta p_n ) = (\Delta p_n)^2\\ q_n \Delta^2 p_n = (\Delta p_n)^2\\ \hat p_n = p_n - \frac{(\Delta p_n)^2}{\Delta^2 p_n}$$

It can be shown that if $p_n$ is linearly convergent then $\hat p_n$ converges faster in the sense that

$$\lim_{n\to \infty} \frac{|\hat p_n - p|}{|p_n - p|} = 0.$$