Example

How do you perform an LU factorization of the matrix

$$A = \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}?$$

If we perform the row operation $R_1 \leftrightarrow R_2$ first

$$A_1 = \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

$$A_1 = \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$ $$R_3 + R_1 \to R_3$$ $$= \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ -1 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 2 & 0 \end{bmatrix}$$

Let's not do any more row operations:

$$A_1 =\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ -1 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 2 & 0 \end{bmatrix}$$ $$R_3 - 2 R_2 \to R_3$$ $$A_1 =\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ -1 & 2 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 0 & -2 \end{bmatrix}$$

But, if we want to factorize $A$, how is $A_1$ related to $A$? It follows that all row operations are linear operations:

If you do a row operation on a vector $v = \alpha v_1 + \beta v_2$ to get a vector $\hat v$ it is the same as doing the row operation on $v_1$ and $v_2$ to get $\hat v_1$ and $\hat v_2$, respectively, and then $\hat v = \alpha \hat v_1 + \beta \hat v_2$.

This is important because it implies that all row operations have a matrix representation. We've already found the matrix $L$ for $R_m + \alpha R_p \to R_m$ for $m > p$:

$$L = (l_{ij}), \quad l_{ij} = \begin{cases} 1, & i =j\\ \alpha,& j = m, ~~ i = p,\\ 0, & \text{otherwise}.\end{cases}, ~~\quad L^{-1} = (l'_{ij}), \quad l'_{ij} = \begin{cases} 1, & i =j\\ -\alpha,& j = m, ~~ i = p,\\ 0, & \text{otherwise}.\end{cases}$$

The matrix for $R_1 \leftrightarrow R_2$

To find the matrix representation for a (linear) transformation, you apply the transformation to each of the standard (basis) unit vectors:

$$\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} \to \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$$ $$\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} \to \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$$ $$\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \to \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$$

We assemble the resulting columns into a matrix. The $R_1 \leftrightarrow R_2$ matrix is

$$P = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

This is called a simple permutation matrix because it corresponds to ONE row interchange. Another way of constructing the matrix associated with a row operation is:

To construct the matrix associated with a row operation, apply that row operation to the identity matrix.

A general simple permutation matrix corresponding to $R_m \leftrightarrow R_k$ is given by

$$P = (p_{ij}), \quad p_{ij} = \begin{cases}1, & i = j \text{ and } i \neq m,p \text{ and } j \neq m,p\\ 1, & i = m, j = p \text{ or } i = p, j = m\\ 0, & \text{otherwise}.\end{cases}$$

It follows immediately, that $p_{ij} = p_{ji}$ so that $P^T = P$.

Returning to our example,

$$A = \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}.$$ $$PA = A_1 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ -1 & 2 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 0 & -2 \end{bmatrix} = LU$$ $$A = P^{-1} LU$$

It follows that $P^{-1} = P= P^T$.

Consider a general permutation matrix, which we define to be the product of a number of simple permulation

$$P = P_k P_{k-1} \cdots P_{2} P_{1}.$$

Notice that $P^T = P_1^T P_2^T \cdots P_{k-1}^T P_k^T = P_1^{-1} P_2^{-1} \cdots P_{k-1}^{-1} P_k^{-1} = P^{-1}$.

The permuted LU factorization

Given an invertible matrix $A$, we want to find a permutation matrix $P$ such that

$$PA = LU$$ $$A = P^T LU$$

where $L$ is a lower-triangular matrix with ones on the diagonal and $U$ is upper trianguler. Note that such a factorization is NOT unique. Different pivoting strategies give different permutation matrices.

Example

Compute the permuted $LU$ factorization for $A$ using partial pivoting

$$A = \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

You should check the following. If $P_1A$ swaps rows $i$ and $j$ of the matrix $A$ then $AP_1$ swaps columns $i$ and $j$ of $A$.

To compute this factorization, we first write

$$A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

These two identity matrices will be used to capture both the $P$ and $L$ in the matrix factorization

$$A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

We first must swap the rows one and two (following partial pivoting). Let $P_1$ be the matrix that corresponds to this. Because $P_1 = P_1^{-1}$ we can write:

$$A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_1 P_1 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_1 P_1 \begin{bmatrix} 0 & 1 & 1 \\ 1 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

$$A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_1P_1 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_1 \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

The matrix $P_1 I P_1 = I$. As we progress, this will not always happen. Recall that multiplication on the right by $P_1$ corresponds to column swaps:

$$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ -1 & 1 & -1 \end{bmatrix}$$

Now we perform the row operation $R_3 + R_1 \to R_3$. This modifies the matrix that will become $L$ in the $A = P^T LU$ factorization

$$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ -1 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 2 & 0 \end{bmatrix}$$

Following the rules of partial pivoting, we must swap rows 2 and 3. Let $P_2$ be the matrix that does this:

$$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_2 P_2 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ -1 & 0 & 1 \end{bmatrix} P_2P_2 \begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 2 & 0 \end{bmatrix}$$

$$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_2 P_2 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ -1 & 0 & 1 \end{bmatrix} P_2 \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 1 & 1 \end{bmatrix}$$ $$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_2 P_2 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ -1 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 1 & 1 \end{bmatrix}$$

$$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} P_2 \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 1 & 1 \end{bmatrix}$$ $$A = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 1 & 1 \end{bmatrix}$$

We have one final row operation: $R_3 - 1/2 R_2 \to R_3$

$$A = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & 1/2 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 0 & 1 \end{bmatrix}$$

Two factorizations:

Naive Gaussian elimination: $$A = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0\\ -1 & 2 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 & 1 \\ 0 & 1 & 1\\ 0 & 0 & -2 \end{bmatrix} = P^T LU$$

Partial pivoting: $$A = \begin{bmatrix} 0 & 0 & 1 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 \\ -1 & 1 & 0 \\ 0 & 1/2 & 1 \end{bmatrix} \begin{bmatrix} 1 & 1 & 1 \\ 0 & 2 & 0\\ 0 & 0 & 1 \end{bmatrix} = P^TLU$$

The entries of $L$ for partial pivoting will always be less than one!