Definition

An $n \times n$ matrix $A$ is said to be diagonally dominant if

$$|a_{ii}| > \sum_{j=1,~~i\neq j}^n |a_{ij}|, ~~\text{for all}~~ 1 \leq i \leq n.$$

Theorem

Jacobi's method and the Gauss-Seidel method converge if $A$ is diagonally dominant.

Proof

We know that for an iterative method with $x^{(k)} = g(x^{(k-1)}) = T x^{(k-1)} + c$, it follows that

$$\|x^{(k)}-x\| \leq \|T\|^k \|x^{(0)} - x\|,$$

in any norm. We choose the $l_\infty$ norm. We prove this only for the Gauss-Seidel method (the Jacobi method is easier). We show that

$$\|(D-L)^{-1}U \|_\infty < 1.$$

Let $\|x\|_\infty =1$ and let $y$ solve

$$(D-L)y = Ux, \quad y = (D-L)^{-1}Ux.$$

Then we need to show that $\|y\|_\infty < 1$ (which implies that $\|(D-L)^{-1}U \|_\infty < 1$). The components of $x$ and $y$ solve

$$a_{ii} y_i + \sum_{j = 1}^{i-1} a_{ij} y_j = -\sum_{j=i+1}^n a_{ij} x_j, \quad i =1,2,\ldots,n.$$

There exists (at least) one component of $y_p$ of $y$ such that $|y_p| = \|y\|_\infty \neq 0$. We consider this equation

$$a_{pp} y_p + \sum_{j = 1}^{p-1} a_{pj} y_j = -\sum_{j=p+1}^n a_{pj} x_j.$$

We use the (reverse) triangle inequality

$$|a_{pp} y_p| - \left| \sum_{j = 1}^{p-1} a_{pj} y_j \right| \leq \left| \sum_{j=p+1}^n a_{pj} x_j \right|.$$ $$|a_{pp}|\|y\|_\infty - \sum_{j = 1}^{p-1} |a_{pj}| \|y\|_\infty \leq |a_{pp} y_p| - \left| \sum_{j = 1}^{p-1} a_{pj} y_j \right| \leq \sum_{j=p+1}^n |a_{pj}|.$$ $$\|y\|_\infty \leq \frac{\sum_{j=p+1}^n |a_{pj}|}{|a_{pp}| - \sum_{j = 1}^{p-1} |a_{pj}|}.$$

Because of diagonal dominance

$$|a_{pp} | = \sum_{j=p+1}^n |a_{pj}| + \sum_{j = 1}^{p-1} |a_{pj}| + c_p, c_p > 0,$$ $$|a_{pp} |- \sum_{j = 1}^{p-1} |a_{pj}| = \sum_{j=p+1}^n |a_{pj}| + c_p,$$

This implies $$\|y\|_\infty \leq \frac{\sum_{j=p+1}^n |a_{pj}|}{|a_{pp}| - \sum_{j = 1}^{p-1} |a_{pj}|} = \frac{\sum_{j=p+1}^n |a_{pj}|}{\sum_{j=p+1}^n |a_{pj}| + c_p} < 1,$$

showing that $\|(D-L)^{-1}U\|_\infty < 1$ and convergence follows.

Corollary

For an $n \times n$ linear system $Ax=b$ if

$$c = \min_{1 \leq i \leq n} \left( 1 - \sum_{j=1}^n \left|\frac{a_{ij}}{a_{ii}} \right| \right) > 0,$$

then the iterates $\{x^{(k)}\}$ of either Jacobi's method or the Gauss-Seidel method satisfy

$$\|x^{(k)}-x\|_\infty \leq (1+c)^{-k} \|x^{(0)} -x \|_\infty.$$

Proof

From the proof of the theorem above, (choosing $p$ in the same way) it follows that

$$1 = \sum_{j=p+1}^n \left| \frac{a_{pj}}{a_{pp}} \right| + \sum_{j = 1}^{p-1} \left| \frac{a_{pj}}{a_{pp}} \right| + c_p, c_p \geq c > 0,$$

Use the notation

$$S_p^+ = \sum_{j=p+1}^n \left| \frac{a_{pj}}{a_{pp}} \right|, \quad S_p^- = \sum_{j = 1}^{p-1} \left| \frac{a_{pj}}{a_{pp}} \right|.$$

Then

$$\|(D-L)^{-1}U\|_\infty \leq \frac{S_p^+}{S_p^+ + c_p} \leq \frac{S_p^+}{S_p^+ + c}$$

.

The for any $c > 0$, it follows that the function $x/(x+c)$ is monotone increasing for $x > 0$. And, because $S_p^+ <1$ we have

$$\|(D-L)^{-1}U\|_\infty < \frac{1}{1+c}.$$

The same calculations can be performed for Jacobi's method to show

$$\|(D-L)^{-1}U\|_\infty \leq \frac{S_p^+ + S_p^-}{S_p^+ + S_p^- + c_p} \leq \frac{S_p^+ + S_p^-}{S_p^+ + S_p^- + c} \leq \frac{1}{1+c}.$$

We note that

$$\frac{S_p^+}{S_p^+ + c} \leq \frac{S_p^+ + S_p^-}{S_p^+ + S_p^- + c},$$

which is demonstrates that for diagonally dominant matrices the Gauss-Seidel method converges more rapidly than does Jacobi's method.

Relaxation Techniques

Definition

Suppose that $\hat x$ is an approximation to a the solution of the linear system $Ax = b$. The corresponding residual vector is $\hat r = b - A \hat x$.

The goal of any interative method that generates a sequence $x^{(0)}, x^{(1)},\ldots, x^{(k)}, \ldots$ is to make the residual vectors

$$r^{(k)} = b - A x^{(k)} \to 0$$

rapidly as $k \to \infty$. The relaxation methods we now describe introduce a parameter $\omega$ that gives flexibility to increase this convergence rate.

Recall that the iteration for the Gauss-Seidel method is given by

$$g_{\mathrm{GS}}(x) = (D - L)^{-1}U x + (D - L)^{-1}b.$$

We can choose our iteration to be an average of the current iterate and the next iterate. Let $\omega > 0$, and consider

$$x^{(k)} = \omega g_{\mathrm{GS}} (x^{(k-1)}) + (1-\omega) x^{(k-1)}.$$

Pulling from the Gauss-Seidel iteration equation it follows that

$$x_i^{(k)} = (1-\omega) x_i^{(k-1)} + \frac{\omega}{a_{ii}} \left(b_i - \sum_{j < i} a_{ij} x_j^{(k)} - \sum_{j > i} a_{ij} x_j^{(k-1)} \right), \quad i = 1,2,\ldots,n.$$

But to understand the convergence of the method we need it to be in the form $x^{(k)} = T x^{(k-1)} + c$. So, we rearrange this

$$x_i^{(k)} + \frac{\omega}{a_{ii}} \sum_{j < i} a_{ij} x_j^{(k)}= (1-\omega) x_i^{(k-1)} + \frac{\omega}{a_{ii}} \left(b_i - \sum_{j > i} a_{ij} x_j^{(k-1)} \right), \quad i = 1,2,\ldots,n.$$ $$a_{ii} x_i^{(k)} + {\omega}\sum_{j < i} a_{ij} x_j^{(k)}= a_{ii}(1-\omega) x_i^{(k-1)} + {\omega}- \sum_{j > i} a_{ij} x_j^{(k-1)} + \omega b_i, \quad i = 1,2,\ldots,n.$$ $$(D - \omega L) x^{(k)} = [(1-\omega) D + \omega U] x^{(k-1)} + \omega b.$$

Therefore, the the iteration is given by

$$x^{(k)} = (D - \omega L)^{-1}[(1-\omega) D + \omega U] x^{(k-1)} + \omega (D - \omega L)^{-1}b.$$

If $\omega = 0$, $x^{(k)} = x^{(k-1)}$ and the iteration does nothing. If $\omega = 1$, the iteration is just the Gauss-Seidel method. For $0 < \omega < 1$, is a more conservative approach to a Gauss-Seidel-type iteration. This is called an under-relaxation method. If $\omega > 1$ then, in some sense, the method is a more ambitious Gauss-Seidel-type method, and the method is called an over-relaxation method. When $1 < \omega < 2$ we call the resulting method Successive Over-Relaxation or SOR.

Choosing $\omega$ is a difficult task. A good value is often found by experimentation. Some theorems about choosing $\omega$ can be derived in very special cases:

Theorem

If $A$ is a symmetric ($A^T = A$), tridiagonal ($a_{ij} = 0$ if $|i-j| > 1$) with positive eigenvalues then one should choose

$$\omega = \frac{2}{1 + \sqrt{1 - r^2}},$$

where $r = \rho(D^{-1}(L+U))$ is the spectral radius of the Jacobi's method matrix.