Chapter 4
Introduction to Matrices
Unfortunately, no one can be told what the matrix is.
You have to see it for yourself.
— Morpheus in The Matrix (1999)
Matrices are of fundamental importance in 3D math, where they are primarily used to describe the
relationship between two coordinate spaces. They do this by defining a computation to transform
vectors from one coordinate space to another.
This chapter introduces the theory and application of matrices. Our discussion will follow the
pattern set in Chapter 2 when we introduced vectors: mathematical definitions followed
by geometric interpretations.

Section 4.1 discusses some of the
basic properties and operations of matrices strictly
from a mathematical perspective. (More matrix operations
are discussed in Chapter 6.)

Section 4.2 explains how to
interpret these properties and operations geometrically.

Section 4.3 puts the use of matrices in this
book in context within the larger field of linear algebra.
4.1Mathematical Definition of Matrix
In linear algebra, a matrix is a rectangular grid of numbers arranged into rows and
columns. Recalling our earlier definition of vector as a onedimensional array of
numbers, a matrix may likewise be defined as a twodimensional array of numbers. (The
“two” in “twodimensional array” comes from the fact that there are rows and columns, and
should not be confused with 2D vectors or matrices.) So a vector is an array of scalars, and a
matrix is an array of vectors.
This section presents matrices from a purely mathematical
perspective. It is divided into eight subsections.
4.1.1Matrix Dimensions and Notation
Just as we defined the dimension of a vector by counting how many numbers it contained, we will
define the size of a matrix by counting how many rows and columns it contains. We say that a
matrix with
$r$
rows and
$c$
columns is an
$r\times c$
(read “
$r$
by
$c$
”) matrix. For
example, a
$4\times 3$
matrix has 4 rows and 3 columns:
A
$\mathbf{4}\mathbf{\times}\mathbf{3}$
matrix
$$\left[\begin{array}{ccc}4& 0& 12\\ 5& \sqrt{4}& 3\\ 12& 4/3& 1\\ 1/2& 18& 0\end{array}\right].$$
This
$4\times 3$
matrix illustrates the standard notation for writing matrices: the numbers are
arranged in a grid enclosed by square brackets. Note that some authors may enclose the grid of
numbers with parentheses rather than brackets, and other authors use straight vertical lines. We
reserve this last notation for an entirely separate concept related to matrices, the
determinant of a matrix. (We discuss determinants in
Section 6.1.)
As we mentioned in Section 2.1, in this book we represent a matrix
variable with uppercase letters in boldface, for example,
$\mathbf{M}$
,
$\mathbf{A}$
,
$\mathbf{R}$
.
When we wish to refer to the individual elements within a matrix, we use subscript notation,
usually with the corresponding lowercase letter in italics. This is shown below for a
$3\times 3$
matrix:
Subscript notation for matrix elements
$$\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]$$
The notation
${m}_{ij}$
denotes the element in
$\mathbf{M}$
at row
$i$
and column
$j$
. Matrices use
1based indices, so the first row and column are numbered 1. For example,
${m}_{12}$
(read “
$m$
one
two,” not “
$m$
twelve”) is the element in the first row, second column. Notice that this is
different from programming languages such as C++ and Java, which use 0based array indices. A
matrix does not have a column 0 or row 0. This difference in indexing can cause some confusion if
matrices are stored using an actual array data type. For this reason, it's common for classes that
store small, fixed size matrices of the type used for geometric purposes to give each element its
own named member variable, such as float m11, instead of using the language's native
array support with something like float elem[3][3].
4.1.2Square Matrices
Matrices with the same number of rows and columns are called square matrices and are of
particular importance. In this book, we are interested in
$2\times 2$
,
$3\times 3$
, and
$4\times 4$
matrices.
The diagonal elements of a square matrix are those elements for which the row and column
indices are the same. For example, the diagonal elements of the
$3\times 3$
matrix
$\mathbf{M}$
are
${m}_{11}$
,
${m}_{22}$
, and
${m}_{33}$
. The other elements are nondiagonal elements. The
diagonal elements form the diagonal of the matrix:
The diagonal of a
$\mathbf{3}\mathbf{\times}\mathbf{3}$
matrix
If all nondiagonal elements in a matrix are zero, then the matrix is a diagonal matrix. The
following
$4\times 4$
matrix is diagonal:
A diagonal
$\mathbf{4}\mathbf{\times}\mathbf{4}$
matrix
$$\left[\begin{array}{cccc}3& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 5& 0\\ 0& 0& 0& 2\end{array}\right].$$
A special diagonal matrix is the identity matrix. The identity matrix of dimension
$n$
,
denoted
${\mathbf{I}}_{n}$
, is the
$n\times n$
matrix with 1s on the diagonal and 0s elsewhere. For
example, the
$3\times 3$
identity matrix is
The 3D identity matrix
$${\mathbf{I}}_{3}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right].$$
Often, the context will make clear the dimension of the identity matrix used in a particular
situation. In these cases we omit the subscript and refer to the identity matrix simply as
$\mathbf{I}$
.
The identity matrix is special because it is the multiplicative identity element for matrices. (We
discuss matrix multiplication in Section 4.1.6.) The basic idea is that if you
multiply a matrix by the identity matrix, you get the original matrix. In this respect, the
identity matrix is for matrices what the number 1 is for scalars.
4.1.3Vectors as Matrices
Matrices may have any positive number of rows and columns, including one. We have already
encountered matrices with one row or one column: vectors! A vector of dimension
$n$
can be viewed
either as a
$1\times n$
matrix, or as an
$n\times 1$
matrix. A
$1\times n$
matrix is known
as a row vector, and an
$n\times 1$
matrix is known as a column vector. Row
vectors are written horizontally, and column vectors are written vertically:
Row and column vectors
$$\left[\begin{array}{ccc}1& 2& 3\end{array}\right]\phantom{\rule{.5in}{0ex}}\left[\begin{array}{c}4\\ 5\\ 6\end{array}\right]$$
Until now, we have used row and column notations interchangeably. Indeed, geometrically they are
identical, and in many cases the distinction is not important. However, for reasons that will soon
become apparent, when we use vectors with matrices, we must be very clear if our vector is a row or
column vector.
4.1.4Matrix Transposition
Given an
$r\times c$
matrix
$\mathbf{M}$
, the transpose of
$\mathbf{M}$
, denoted
${\mathbf{M}}^{\mathrm{T}}$
, is the
$c\times r$
matrix where the columns are formed from the rows of
$\mathbf{M}$
. In other words,
${{\mathbf{M}}^{\mathrm{T}}}_{ij}={\mathbf{M}}_{ji}$
. This “flips” the matrix
diagonally. Equations (4.1) and
(4.2) show two examples of transposing matrices:
$$\begin{array}{}\text{(4.1)}& {\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\\ 7& 8& 9\\ 10& 11& 12\end{array}\right]}^{T}& =\left[\begin{array}{cccc}1& 4& 7& 10\\ 2& 5& 8& 11\\ 3& 6& 9& 12\end{array}\right],\text{(4.2)}& {\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]}^{T}& =\left[\begin{array}{ccc}a& d& g\\ b& e& h\\ c& f& i\end{array}\right].\end{array}$$
Transposing matricesFor vectors, transposition turns row vectors into column vectors and vice versa:
Transposing converts between row and column vectors
$$\begin{array}{rlrl}{\left[\begin{array}{ccc}x& y& z\end{array}\right]}^{T}& =\left[\begin{array}{c}x\\ y\\ z\end{array}\right]& {\left[\begin{array}{c}x\\ y\\ z\end{array}\right]}^{T}& =\left[\begin{array}{ccc}x& y& z\end{array}\right]\end{array}$$
Transposition notation is often used to write column vectors inline in a paragraph, such as
$[1,2,3{]}^{\mathrm{T}}$
.
Let's make two fairly obvious, but significant, observations regarding matrix transposition.

$({\mathbf{M}}^{\mathrm{T}}{)}^{\mathrm{T}}=\mathbf{M}$
, for a matrix
$\mathbf{M}$
of any dimension. In other words, if
we transpose a matrix, and then transpose it again,
we get the original matrix. This rule also applies
to vectors.

Any diagonal matrix
$\mathbf{D}$
is equal to its transpose:
${\mathbf{D}}^{\mathrm{T}}=\mathbf{D}$
. This includes the
identity matrix
$\mathbf{I}$
.
4.1.5Multiplying a Matrix with a Scalar
A matrix
$\mathbf{M}$
may be multiplied with a scalar
$k$
, resulting in a matrix of the same
dimension as
$\mathbf{M}$
. We denote matrix multiplication with a scalar by placing the scalar
and the matrix sidebyside, usually with the scalar on the left. No multiplication symbol is
necessary. The multiplication takes place in a straightforward fashion: each element in the
resulting matrix
$k\mathbf{M}$
is the product of
$k$
and the corresponding element in
$\mathbf{M}$
. For example,
Multiplying a
$\mathbf{4}\mathbf{\times}\mathbf{3}$
matrix by a scalar
$$k\mathbf{M}=k\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\\ {m}_{41}& {m}_{42}& {m}_{43}\end{array}\right]=\left[\begin{array}{ccc}k{m}_{11}& k{m}_{12}& k{m}_{13}\\ k{m}_{21}& k{m}_{22}& k{m}_{23}\\ k{m}_{31}& k{m}_{32}& k{m}_{33}\\ k{m}_{41}& k{m}_{42}& k{m}_{43}\end{array}\right].$$
4.1.6Multiplying Two Matrices
In certain situations, we can take the product of two matrices. The rules that govern when matrix
multiplication is allowed and how the result is computed may at first seem bizarre.
An
$r\times n$
matrix
$\mathbf{A}$
may be multiplied by an
$n\times c$
matrix
$\mathbf{B}$
. The
result, denoted
$\mathbf{A}\mathbf{B}$
, is an
$r\times c$
matrix. For example, assume that
$\mathbf{A}$
is a
$4\times 2$
matrix, and
$\mathbf{B}$
is a
$2\times 5$
matrix. Then
$\mathbf{A}\mathbf{B}$
is a
$4\times 5$
matrix:
If the number of columns in
$\mathbf{A}$
does not match the number of rows in
$\mathbf{B}$
, then
the multiplication
$\mathbf{A}\mathbf{B}$
is not defined (although
$\mathbf{B}\mathbf{A}$
may be
possible).
Matrix multiplication is computed as follows: let the matrix
$\mathbf{C}$
be the
$r\times c$
product
$\mathbf{A}\mathbf{B}$
of the
$r\times n$
matrix
$\mathbf{A}$
with the
$n\times c$
matrix
$\mathbf{B}$
. Then each element
${c}_{ij}$
is equal to the vector dot product of row
$i$
of
$\mathbf{A}$
with column
$j$
of
$\mathbf{B}$
:
$${c}_{ij}=\sum _{k=1}^{n}{a}_{ik}{b}_{kj}.$$
This looks complicated, but there is a simple pattern. For each element
${c}_{ij}$
in the result,
locate row
$i$
in
$\mathbf{A}$
and column
$j$
in
$\mathbf{B}$
. Multiply the corresponding elements
of the row and column, and sum the products.
${c}_{ij}$
is equal to this sum, which is equivalent to
the dot product of row
$i$
in
$\mathbf{A}$
with column
$j$
in
$\mathbf{B}$
.
Let's look at an example of how to compute
${c}_{24}$
. The element in row 2 and column 4 of
$\mathbf{C}$
is equal to the dot product of row 2 of
$\mathbf{A}$
with the column 4 of
$\mathbf{B}$
:
Another way to help remember the pattern for matrix multiplication is to write
$\mathbf{B}$
above
$\mathbf{C}$
. This aligns the proper row from
$\mathbf{A}$
with a column from
$\mathbf{B}$
for
each element in the result
$\mathbf{C}$
:
For geometric applications, we are particularly interested in multiplying square matrices—the
$2\times 2$
and
$3\times 3$
cases being especially important.
Equation (4.3) gives the complete equation for
$2\times 2$
matrix
multiplication:
$\mathbf{2}\mathbf{\times}\mathbf{2}$
matrix multiplication
$$\begin{array}{}\text{(4.3)}& \begin{array}{rl}\mathbf{A}\mathbf{B}& =\left[\begin{array}{cc}{a}_{11}& {a}_{12}\\ {a}_{21}& {a}_{22}\end{array}\right]\left[\begin{array}{cc}{b}_{11}& {b}_{12}\\ {b}_{21}& {b}_{22}\end{array}\right]\\ & =\left[\begin{array}{cc}{a}_{11}{b}_{11}+{a}_{12}{b}_{21}& {a}_{11}{b}_{12}+{a}_{12}{b}_{22}\\ {a}_{21}{b}_{11}+{a}_{22}{b}_{21}& {a}_{21}{b}_{12}+{a}_{22}{b}_{22}\end{array}\right].\end{array}\end{array}$$
Let's look at a
$2\times 2$
example with some real numbers:
$$\begin{array}{rl}\mathbf{A}& =\left[\begin{array}{cc}3& 0\\ 5& 1/2\end{array}\right],{\textstyle \phantom{\rule{1em}{0ex}}}\mathbf{B}=\left[\begin{array}{cc}7& 2\\ 4& 6\end{array}\right],\\ \mathbf{A}\mathbf{B}& =\left[\begin{array}{cc}3& 0\\ 5& 1/2\end{array}\right]\left[\begin{array}{cc}7& 2\\ 4& 6\end{array}\right]\\ & =\left[\begin{array}{cc}(3)(7)+(0)(4)& (3)(2)+(0)(6)\\ (5)(7)+(1/2)(4)& (5)(2)+(1/2)(6)\end{array}\right]=\left[\begin{array}{cc}21& 6\\ 33& 13\end{array}\right].\end{array}$$
Applying the general matrix multiplication formula to the
$3\times 3$
case produces
$\mathbf{3}\mathbf{\times}\mathbf{3}$
matrix multiplication
$$\begin{array}{rl}\mathbf{A}\mathbf{B}& =\left[\begin{array}{ccc}{a}_{11}& {a}_{12}& {a}_{13}\\ {a}_{21}& {a}_{22}& {a}_{23}\\ {a}_{31}& {a}_{32}& {a}_{33}\end{array}\right]\left[\begin{array}{ccc}{b}_{11}& {b}_{12}& {b}_{13}\\ {b}_{21}& {b}_{22}& {b}_{23}\\ {b}_{31}& {b}_{32}& {b}_{33}\end{array}\right]\\ & =\left[\begin{array}{ccc}{a}_{11}{b}_{11}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{12}{b}_{21}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{13}{b}_{31}& {a}_{11}{b}_{12}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{12}{b}_{22}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{13}{b}_{32}& {a}_{11}{b}_{13}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{12}{b}_{23}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{13}{b}_{33}\\ {a}_{21}{b}_{11}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{22}{b}_{21}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{23}{b}_{31}& {a}_{21}{b}_{12}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{22}{b}_{22}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{23}{b}_{32}& {a}_{21}{b}_{13}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{22}{b}_{23}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{23}{b}_{33}\\ {a}_{31}{b}_{11}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{32}{b}_{21}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{33}{b}_{31}& {a}_{31}{b}_{12}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{32}{b}_{22}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{33}{b}_{32}& {a}_{31}{b}_{13}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{32}{b}_{23}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}{a}_{33}{b}_{33}\end{array}\right].\end{array}$$
Here is a
$3\times 3$
example with some real numbers:
$$\begin{array}{rl}\mathbf{A}& =\left[\begin{array}{ccc}1& 5& 3\\ 0& 2& 6\\ 7& 2& 4\end{array}\right],{\textstyle \phantom{\rule{1em}{0ex}}}\mathbf{B}=\left[\begin{array}{ccc}8& 6& 1\\ 7& 0& 3\\ 2& 4& 5\end{array}\right];\\ \mathbf{A}\mathbf{B}& =\left[\begin{array}{ccc}1& 5& 3\\ 0& 2& 6\\ 7& 2& 4\end{array}\right]\left[\begin{array}{ccc}8& 6& 1\\ 7& 0& 3\\ 2& 4& 5\end{array}\right]\\ & =\left[\begin{array}{ccc}1{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(8)+(5){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}7{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}3{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}2& 1{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}6+(5){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}0{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}3{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}4& 1{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}1{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}(5){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(3){\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}3{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}5\\ 0{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(8)+(2){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}7{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}6{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}2& 0{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}6+(2){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}0{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}6{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}4& 0{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}1{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}(2){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(3){\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}6{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}5\\ 7{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(8)+2{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}7{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}(4){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}2& 7{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}6+2{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}0{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}(4){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}4& 7{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}1{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}2{\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}(3){\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}(4){\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}\cdot {\textstyle \phantom{\rule{negativethinmathspace}{0ex}}}5\end{array}\right]\\ & =\left[\begin{array}{ccc}37& 18& 31\\ 2& 24& 36\\ 50& 26& 19\end{array}\right].\end{array}$$
Beginning in Section 6.4 we also use
$4\times 4$
matrices.
A few notes concerning matrix multiplication are of interest:

Multiplying any matrix
$\mathbf{M}$
by a square
matrix
$\mathbf{S}$
on either side results in a matrix
of the same size as
$\mathbf{M}$
, provided that the
sizes of the matrices are such that the multiplication
is allowed. If
$\mathbf{S}$
is the identity matrix
$\mathbf{I}$
,
then the result is the original matrix
$\mathbf{M}$
:
$$\mathbf{M}\mathbf{I}=\mathbf{I}\mathbf{M}=\mathbf{M}.$$
(That's the reason it's called the identity matrix!)

Matrix multiplication is not commutative. In general,
$$\mathbf{A}\mathbf{B}\ne \mathbf{B}\mathbf{A}.$$

Matrix multiplication is associative:
$$(\mathbf{A}\mathbf{B})\mathbf{C}=\mathbf{A}(\mathbf{B}\mathbf{C}).$$
(Of course, this assumes that the sizes of
$\mathbf{A}$
,
$\mathbf{B}$
, and
$\mathbf{C}$
are such that multiplication is allowed. Note that if
$(\mathbf{A}\mathbf{B})\mathbf{C}$
is defined, then
$\mathbf{A}(\mathbf{B}\mathbf{C})$
is always defined as well.) The associativity of matrix multiplication extends to
multiple matrices. For example,
$$\begin{array}{r}\begin{array}{rl}\mathbf{A}\mathbf{B}\mathbf{C}\mathbf{D}\mathbf{E}\mathbf{F}& =((((\mathbf{A}\mathbf{B})\mathbf{C})\mathbf{D})\mathbf{E})\mathbf{F}\\ & =\mathbf{A}((((\mathbf{B}\mathbf{C})\mathbf{D})\mathbf{E})\mathbf{F})\\ & =(\mathbf{A}\mathbf{B})(\mathbf{C}\mathbf{D})(\mathbf{E}\mathbf{F}).\end{array}\end{array}$$
It is interesting to note that although all parenthesizations compute the correct
result, some groupings require fewer scalar multiplications than others.

Matrix multiplication also associates with multiplication by a scalar or a vector:
$$\begin{array}{rl}(k\mathbf{A})\mathbf{B}& =k(\mathbf{A}\mathbf{B})=\mathbf{A}(k\mathbf{B}),\\ (\mathbf{v}\mathbf{A})\mathbf{B}& =\mathbf{v}(\mathbf{A}\mathbf{B}).\end{array}$$

Transposing the product of two matrices is the
same as taking the product of their transposes in reverse order:
$$(\mathbf{A}\mathbf{B}{)}^{T}={\mathbf{B}}^{\mathrm{T}}{\mathbf{A}}^{\mathrm{T}}.$$
This can be extended to more than two matrices:
$$({\mathbf{M}}_{1}{\mathbf{M}}_{2}\cdots {\mathbf{M}}_{n1}{\mathbf{M}}_{n}{)}^{T}={{\mathbf{M}}_{n}}^{\mathrm{T}}{{\mathbf{M}}_{n1}}^{\mathrm{T}}\cdots {{\mathbf{M}}_{2}}^{\mathrm{T}}{{\mathbf{M}}_{1}}^{\mathrm{T}}.$$
4.1.7Multiplying a Vector and a Matrix
Since a vector can be considered a matrix with one row or one column, we can multiply a vector
and a matrix by applying the rules discussed in the previous section. Now it becomes very
important whether we are using row or column vectors.
Equations (4.4)–(4.7)
show how 3D row and column vectors may be pre or postmultiplied by a
$3\times 3$
matrix:
Multiplying 3D row and column vectors with a
$\mathbf{3}\mathbf{\times}\mathbf{3}$
matrix
$$\begin{array}{r}\text{(4.4)}& \left[\begin{array}{ccc}x& y& z\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]& ={\displaystyle \phantom{\rule{1.6in}{0ex}}\left[\begin{array}{ccc}x{m}_{11}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}y{m}_{21}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}z{m}_{31}& x{m}_{12}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}y{m}_{22}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}z{m}_{32}& x{m}_{13}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}y{m}_{23}{\textstyle \phantom{\rule{thinmathspace}{0ex}}}+{\textstyle \phantom{\rule{thinmathspace}{0ex}}}z{m}_{33}\end{array}\right];}\\ {\textstyle}\\ \text{(4.5)}& \left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\end{array}\right]& =\left[\begin{array}{c}x{m}_{11}+y{m}_{12}+z{m}_{13}\\ x{m}_{21}+y{m}_{22}+z{m}_{23}\\ x{m}_{31}+y{m}_{32}+z{m}_{33}\end{array}\right];\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]\left[\begin{array}{ccc}x& y& z\end{array}\right]& =\text{(undefined)};\\ \text{(4.7)}& \left[\begin{array}{c}x\\ y\\ z\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]& =\text{(undefined)}.\end{array}$$
As you can see, when we multiply a row vector on the left by a matrix on the right, as in
Equation (4.4), the result is a row vector. When we multiply a matrix
on the left by a column vector on the right, as in Equation (4.5),
the result is a column vector. (Please observe that this result is a column vector, even
though it looks like a matrix.) The other two combinations are not allowed: you cannot multiply a
matrix on the left by a row vector on the right, nor can you multiply a column vector on the left
by a matrix on the right.
Let's make a few interesting observations regarding vectortimesmatrix multiplication. First, each
element in the resulting vector is the dot product of the original vector with a single row or
column from the matrix.
Second, each element in the matrix determines how much “weight” a particular element in the input
vector contributes to an element in the output vector. For example, in
Equation (4.4) when row vectors are used,
${m}_{12}$
controls how much of
the input
$x$
value goes towards the output
$y$
value.
Third, vectortimesmatrix multiplication distributes over vector addition, that is, for vectors
$\mathbf{v}$
,
$\mathbf{w}$
, and matrix
$\mathbf{M}$
,
$$(\mathbf{v}+\mathbf{w})\mathbf{M}=\mathbf{v}\mathbf{M}+\mathbf{w}\mathbf{M}.$$
Finally, and perhaps most important of all, the result of the multiplication is a linear
combination of the rows or columns of the matrix. For example, in
Equation (4.5), when column vectors are used, the resulting column
vector can be interpreted as a linear combination of the columns of the matrix, where the
coefficients come from the vector operand. This is a key fact, not just for our purposes but also
for linear algebra in general, so bear it in mind. We will return to it shortly.
4.1.8Row versus Column Vectors
This section explains why the distinction between row and column vectors is significant and gives
our rationale for preferring row vectors. In Equation (4.4), when we
multiply a row vector on the left with matrix on the right, we get the row vector
$$\left[\begin{array}{ccc}x{m}_{11}+y{m}_{21}+z{m}_{31}& x{m}_{12}+y{m}_{22}+z{m}_{32}& x{m}_{13}+y{m}_{23}+z{m}_{33}\end{array}\right].$$
Compare that with the result from
Equation (4.5), when a column vector
on the right is multiplied by a matrix on the left:
$$\left[\begin{array}{c}x{m}_{11}+y{m}_{12}+z{m}_{13}\\ x{m}_{21}+y{m}_{22}+z{m}_{23}\\ x{m}_{31}+y{m}_{32}+z{m}_{33}\end{array}\right].$$
Disregarding the fact that one is a row vector and the other is a
column vector, the values for the components of the vector are not
the same! Thisis why the distinction between row and column
vectors is soimportant.
Although some matrices in video game programming do represent arbitrary systems of equations, a
much larger majority are transformation matrices of the type we have been describing, which express
relationships between coordinate spaces. For this purpose, we find row vectors to be preferable
for the “eminently sensible” reason [1] that the order of transformations
reads like a sentence from left to right. This is especially important when more than one
transformation takes place. For example, if we wish to transform a vector
$\mathbf{v}$
by the
matrices
$\mathbf{A}$
,
$\mathbf{B}$
, and
$\mathbf{C}$
, in that order, we write
$\mathbf{v}\mathbf{A}\mathbf{B}\mathbf{C}$
, with the matrices listed in order of transformation
from left to right. If column vectors are used, then the vector is on the right, and so the
transformations will occur in order from right to left. In this case, we would write
$\mathbf{C}\mathbf{B}\mathbf{A}\mathbf{v}$
. We discuss concatenation of multiple transformation
matrices in detail in Section 5.6.
Unfortunately, row vectors lead to very “wide” equations; using column vectors on the right
certainly makes things look better, especially as the dimension increases. (Compare the
ungainliness of Equation (4.4) with the sleekness of
Equation (4.5).) Perhaps this is why column vectors are the near
universal standard in practically every other discipline. For most video game programming, however,
readable computer code is more important than readable equations. For this reason, in this book
we use row vectors in almost all cases where the distinction is relevant. Our limited use of column
vectors is for aesthetic purposes, when either there are no matrices involved or those matrices are
not transformation matrices and the lefttoright reading order is not helpful.
As expected, different authors use different conventions. Many graphics books and application
programming interfaces (APIs), such as DirectX, use row vectors. But other APIs, such as OpenGL
and the customized ports of OpenGL onto various consoles, use column vectors. And, as we have
said, nearly every other science that uses linear algebra prefers column vectors. So be very
careful when using someone else's equation or source code that you know whether it assumes row or
column vectors.
If a book uses column vectors, its equations for matrices will be transposed compared to the
equations we present in this book. Also, when column vectors are used, vectors are premultiplied
by a matrix, as opposed to the convention chosen in this book, to multiply row vectors by a matrix
on the right. This causes the order of multiplication to be reversed between the two styles when
multiple matrices and vectors are multiplied together. For example, the multiplication
$\mathbf{v}\mathbf{A}\mathbf{B}\mathbf{C}$
is valid only with row vectors. The corresponding
multiplication would be written
$\mathbf{C}\mathbf{B}\mathbf{A}\mathbf{v}$
if column vectors were
used. (Again, note that in this case
$\mathbf{A}$
,
$\mathbf{B}$
, and
$\mathbf{C}$
would be
transposed as compared to these matrices in the row vector case.)
Mistakes like this involving transposition can be a common source of frustration when programming
3D math. Luckily, with properly designed C++ classes, direct access to the individual matrix
elements is seldom needed, and these types of errors can be minimized.
4.2Geometric Interpretation of Matrix
In general, a square matrix can describe any linear transformation. In
Section 5.7.1, we provide a complete definition of
linear transformation, but for now, it suffices to say that a linear transformation preserves
straight and parallel lines, and that there is no translation—that is, the origin does not
move. However, other properties of the geometry, however, such as lengths, angles, areas, and
volumes, are possibly altered by the transformation. In a nontechnical sense, a linear
transformation may “stretch” the coordinate space, but it doesn't “curve” or “warp” it.
This is a very useful set of transformations, including

rotation

scale

orthographic projection

reflection

shearing
Chapter 5 derives matrices that perform all of these operations. For now,
though, let's just attempt a general understanding of the relationship between a matrix and the
transformation it represents.
The quotation at the beginning of this chapter is not only a line from a great movie, it's true for
linear algebra matrices as well. Until you develop an ability to visualize a matrix, it will just
be nine numbers in a box. We have stated that a matrix represents a coordinate space
transformation. So when we visualize the matrix, we are visualizing the transformation, the new
coordinate system. But what does this transformation look like? What is the relationship between a
particular 3D transformation (i.e. rotation, shearing, etc.) and those nine numbers inside a
$3\times 3$
matrix? How can we construct a matrix to perform a given transform (other than just
copying the equations blindly out of a book)?
To begin to answer this question, let's watch what happens when the standard basis vectors
$\mathbf{i}=[1,0,0]$
,
$\mathbf{j}=[0,1,0]$
, and
$\mathbf{k}=[0,0,1]$
are multiplied by an arbitrary
matrix
$\mathbf{M}$
:
$$\begin{array}{rl}\mathbf{i}\mathbf{M}=\left[\begin{array}{ccc}1& 0& 0\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]& =\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\end{array}\right];\\ \mathbf{j}\mathbf{M}=\left[\begin{array}{ccc}0& 1& 0\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]& =\left[\begin{array}{ccc}{m}_{21}& {m}_{22}& {m}_{23}\end{array}\right];\\ \mathbf{k}\mathbf{M}=\left[\begin{array}{ccc}0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\\ {m}_{21}& {m}_{22}& {m}_{23}\\ {m}_{31}& {m}_{32}& {m}_{33}\end{array}\right]& =\left[\begin{array}{ccc}{m}_{31}& {m}_{32}& {m}_{33}\end{array}\right].\end{array}$$
In other words, the first row of
$\mathbf{M}$
contains the result of performing the transformation
on
$\mathbf{i}$
, the second row is the result of transforming
$\mathbf{j}$
, and the last row is the
result of transforming
$\mathbf{k}$
.
Once we know what happens to those basis vectors, we know everything about the transformation!
This is because any vector can be written as a linear combination of the standard basis, as
$$\mathbf{v}={v}_{x}\mathbf{i}+{v}_{y}\mathbf{j}+{v}_{z}\mathbf{k}.$$
Multiplying this expression by our matrix on the right, we have
$$\begin{array}{rl}\mathbf{v}\mathbf{M}& =({v}_{x}\mathbf{i}+{v}_{y}\mathbf{j}+{v}_{z}\mathbf{k})\mathbf{M}\\ \text{(4.8)}& & =({v}_{x}\mathbf{i})\mathbf{M}+({v}_{y}\mathbf{j})\mathbf{M}+({v}_{z}\mathbf{k})\mathbf{M}& ={v}_{x}(\mathbf{i}\mathbf{M})+{v}_{y}(\mathbf{j}\mathbf{M})+{v}_{z}(\mathbf{k}\mathbf{M})\\ & ={v}_{x}\left[\begin{array}{ccc}{m}_{11}& {m}_{12}& {m}_{13}\end{array}\right]+{v}_{y}\left[\begin{array}{ccc}{m}_{21}& {m}_{22}& {m}_{23}\end{array}\right]+{v}_{z}\left[\begin{array}{ccc}{m}_{31}& {m}_{32}& {m}_{33}\end{array}\right].\end{array}$$
Here we have confirmed an observation made in Section 4.1.7: the result of a
vector
$\times $
matrix multiplication is a linear combination of the rows of the matrix. The key
is to interpret those row vectors as basis vectors. In this interpretation, matrix
multiplication is simply a compact way to encode the operations for coordinate space
transformations developed in Section 3.3.3. A small change of
notation will make this connection more explicit. Remember that we introduced the convention to
use the symbols
$\mathbf{p}$
,
$\mathbf{q}$
, and
$\mathbf{r}$
to refer to a set of basis vectors.
Putting these vectors as rows in our matrix
$\mathbf{M}$
, we can rewrite the last line of
Equation (4.8) as
$$\mathbf{v}\mathbf{M}=\left[\begin{array}{ccc}{v}_{x}& {v}_{y}& {v}_{z}\end{array}\right]\left[\begin{array}{c}\mathbf{p}\\ \mathbf{q}\\ \mathbf{r}\end{array}\right]={v}_{x}\mathbf{p}+{v}_{y}\mathbf{q}+{v}_{z}\mathbf{r}.$$
Let's summarize what we have said.
By understanding how the matrix transforms the standard basis vectors, we know
everything there is to know about the transformation. Since the results of
transforming the standard basis are simply the rows
of the matrix, we interpret those rows as the basis vectors of a
coordinate space.
We now have a simple way to take an arbitrary matrix and visualize what sort of transformation
the matrix represents. Let's look at a couple of examples—first, a 2D example to get ourselves
warmed up, and then a fullfledged 3D example. Examine the following
$2\times 2$
matrix:
$$\mathbf{M}=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right].$$
What sort of transformation does this matrix represent? First, let's extract the basis vectors
$\mathbf{p}$
and
$\mathbf{q}$
from the rows of the matrix:
$$\begin{array}{rl}\mathbf{p}& =\left[\begin{array}{cc}2& 1\end{array}\right];\\ \mathbf{q}& =\left[\begin{array}{cc}1& 2\end{array}\right].\end{array}$$
Figure 4.1 shows these vectors in the Cartesian plane, along with the “original”
basis vectors (the
$x$
axis and
$y$
axis), for reference.
As Figure 4.1 illustrates, the
$+x$
basis vector is transformed into the vector
labeled
$\mathbf{p}$
above, and the
$y$
basis vector is transformed into the vector labeled
$\mathbf{q}$
. So one way to visualize a matrix in 2D is to visualize the L shape formed by the
row vectors. In this example, we can easily see that part of the transformation represented by
matrix
$\mathbf{M}$
is a counterclockwise rotation of approximately
$26.5{}^{\mathrm{o}}$
.
Of course, all vectors are affected by a linear transformation, not just the basis
vectors. We can get a very good idea what this transformation looks like from the L, and we can
gain further insight on the effect the transformation has on the rest of the vectors by
completing the 2D parallelogram formed by the basis vectors, as shown in
Figure 4.2.
This parallelogram is also known as a “skew box.” Drawing an object inside the box can also
help, as illustrated in Figure 4.3.
Now it is clear that our example matrix
$\mathbf{M}$
not only rotates the coordinate space, it
also scales it.
We can extend the techniques we used to visualize 2D transformations into 3D. In 2D, we had two
basis vectors that formed an L—in 3D, we have three basis vectors, and they form a “tripod.”
First, let's show an object before transformation. Figure 4.4 shows a
teapot, a unit cube, and the basis vectors in the “identity” position.
(To avoid cluttering up the diagram, we have not labeled the
$+z$
basis vector
$[0,0,1]$
, which
is partially obscured by the teapot and cube.)
Now consider the 3D transformation matrix
$$\left[\begin{array}{ccc}0.707& 0.707& 0\\ 1.250& 1.250& 0\\ 0& 0& 1\end{array}\right].$$
Extracting the basis vectors from the rows of the matrix, we can visualize the transformation
represented by this matrix. The transformed basis vectors, cube, and teapot are shown in
Figure 4.5.
As we can see, the transformation consists of a clockwise rotation of
$45{}^{\mathrm{o}}$
about the
$z$
axis as well as a nonuniform scale that makes the teapot “taller” than it was originally.
Notice that the
$+z$
basis vector is unaltered by the transformation, because the third row of
the matrix is
$[0,0,1]$
.
By interpreting the rows of a matrix as basis vectors, we have a tool for deconstructing a
matrix. But we also have a tool for constructing one! Given a desired transformation
(i.e., rotation, scale, and so on), we can derive a matrix that represents that transformation.
All we have to do is figure out what the transformation does to the basis vectors, and then place
those transformed basis vectors into the rows of a matrix. We use this tool repeatedly in
Chapter 5 to derive the matrices to perform basic linear transformations such
as rotation, scale, shear, and reflection.
The bottom line about transformation matrices is this: there's nothing especially magical about
matrices. Once we understand that coordinates are best understood as coefficients in a linear
combination of the basis vectors (see Section 3.3.3), we really
know all the math we need to know to do transformations. So from one perspective, matrices are
just a compact way to write things down. A slightly less obvious but much more compelling reason
to cast transformations in matrix notation is to take advantage of the large generalpurpose
toolset from linear algebra. For example, we can take simple transformations and derive more
complicated transformations through matrix concatenation; more on this in
Section 5.6.
Before we move on, let's review the key concepts of Section 4.2.

The rows of a square matrix can be interpreted as the
basis vectors of a coordinate space.

To transform a vector from the original coordinate space to
the new coordinate space, we multiply the vector by the matrix.

The transformation from the original coordinate space to the
coordinate space defined by these basis vectors is a linear
transformation. A linear transformation preserves straight
lines, and parallel lines remain parallel. However, angles,
lengths, areas, and volumes may be altered after transformation.

Multiplying the zero vector by any square matrix results in
the zero vector. Therefore, the linear transformation
represented by a square matrix has the same origin as the
original coordinate space—the transformation does not contain
translation.

We can visualize a matrix by visualizing the basis vectors
of the coordinate space after transformation. These basis
vectors form an `L' in 2D, and a tripod in 3D. Using a box
or auxiliary object also helps in visualization.
4.3The Bigger Picture of Linear Algebra
At the start of Chapter 2, we warned you that in this book we are focusing on just one
small corner of the field of linear algebra—the geometric applications of vectors and matrices.
Now that we've introduced the nuts and bolts, we'd like to say something about the bigger picture
and how our part relates to it.
Linear algebra was invented to manipulate and solve systems of linear equations. For example, a typical introductory problem in a
traditional course on linear algebra is to solve a system of equations such as
$$\begin{array}{rl}5{x}_{1}+{x}_{2}+{x}_{3}& =10,\\ 2{x}_{1}+2{x}_{2}+4{x}_{3}& =12,\\ {x}_{1}3{x}_{3}& =9,\end{array}$$
which has the solution
$$\begin{array}{rl}{x}_{1}& =3,\\ {x}_{2}& =7,\\ {x}_{3}& =2.\end{array}$$
Matrix notation was invented to avoid the tedium involved in duplicating every
$x$
and
$=$
. For example, the system above can be more quickly written as
$$\left[\begin{array}{ccc}5& 1& 1\\ 2& 2& 4\\ 1& 0& 3\end{array}\right]\left[\begin{array}{c}{x}_{1}\\ {x}_{2}\\ {x}_{3}\end{array}\right]=\left[\begin{array}{c}10\\ 12\\ 9\end{array}\right].$$
Perhaps the most direct and obvious place in a video game where a large system of equations must
be solved is in the physics engine. The constraints to enforce nonpenetration and satisfy
userrequested joints become a system of equations relating the velocities of the dynamic bodies.
This large system
is then solved each and every simulation frame. Another common place for traditional linear
algebra methods to appear is in least squares approximation and other datafitting applications.
Systems of equations can appear where you don't expect them. Indeed, linear algebra has exploded
in importance with the vast increase in computing power in the last half century because many
difficult problems that were previously neither discrete nor linear are being approximated
through methods that are both, such as the finite element method. The challenge begins with
knowing how to transform the original problem into a matrix problem in the first place, but the
resulting systems are often very large and can be difficult to solve quickly and accurately.
Numeric stability becomes a factor in the choice of algorithms. The matrices that arise in
practice are not boxes full of random numbers; rather, they express organized relationships and
have a great deal of structure. Exploiting this structure artfully is the key to achieving speed
and accuracy. The diversity of the types of structure that appear in applications explains why
there is so very much to know about linear algebra, especially numerical linear algebra.
This book is intended to fill a gap by providing the geometric intuition that is the bread and
butter of video game programming but is left out of most linear algebra textbooks. However, we
certainly know there is a larger world out there for you. Although traditional linear algebra
and systems of equations do not play a prominent role for basic video game programming, they are
essential for many advanced areas. Consider some of the technologies that are generating buzz
today: fluid, cloth, and hair simulations (and rendering); more robust procedural animation of
characters; realtime global illumination; machine vision; gesture recognition; and many more.
What these seemingly diverse technologies all have in common is that they involve difficult
linear algebra problems.
One excellent resource for learning the bigger picture of linear algebra and scientific computing
is Professor Gilbert Strang's series of lectures, which can be downloaded free from MIT
OpenCourseWare at ocw.mit.edu. He offers a basic undergraduate linear algebra course as
well as graduate courses on computational science and engineering. The companion textbooks he
writes for his classes [3][2]
are enjoyable books aimed at engineers (rather than math sticklers) and are recommended, but be
warned that his writing style is a sort of shorthand that you might have trouble understanding
without the lectures.
Exercises
Use the following matrices for questions 1–3:
$$\begin{array}{ccc}\mathbf{A}=\left[\begin{array}{ccc}13& 4& 8\\ 12& 0& 6\\ 3& 1& 5\\ 10& 2& 5\end{array}\right]& \mathbf{B}=\left[\begin{array}{ccc}{k}_{x}& 0& 0\\ 0& {k}_{y}& 0\\ 0& 0& {k}_{z}\end{array}\right]& \mathbf{C}=\left[\begin{array}{cc}15& 8\\ 7& 3\end{array}\right]\\ \mathbf{D}=\left[\begin{array}{cc}a& g\\ b& h\\ c& i\\ d& j\\ f& k\end{array}\right]& \mathbf{E}=\left[\begin{array}{ccc}0& 1& 3\end{array}\right]& \mathbf{F}=\left[\begin{array}{c}x\\ y\\ z\\ w\end{array}\right]\\ \mathbf{G}=\left[\begin{array}{cccc}10& 20& 30& 1\end{array}\right]& \mathbf{H}=\left[\begin{array}{c}\alpha \\ \beta \\ \gamma \end{array}\right]\end{array}$$

For each matrix, give the dimensions of the matrix and
identify whether it is square and/or diagonal.

Transpose each matrix.

Find all the possible pairs of matrices that can be
legally multiplied, and give the dimensions of the resulting
product. Include “pairs” in which a matrix is multiplied by
itself. (Hint: there are 14 pairs.)

Compute the following matrix products. If the product is not possible,
just say so.

(a)
$\left[\begin{array}{cc}1& 2\\ 5& 0\end{array}\right]\left[\begin{array}{cc}3& 7\\ 4& 1/3\end{array}\right]$

(b)
$\left[\begin{array}{cc}6& 7\\ 4& 5\end{array}\right]\left[\begin{array}{cc}3& 3\end{array}\right]$

(c)
$\left[\begin{array}{ccc}3& 1& 4\end{array}\right]\left[\begin{array}{ccc}2& 0& 3\\ 5& 7& 6\\ 1& 4& 2\end{array}\right]$

(d)
$\left[\begin{array}{cccc}x& y& z& w\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$

(e)
$\left[\begin{array}{cccc}7& 2& 7& 3\end{array}\right]\left[\begin{array}{c}5\\ 1\end{array}\right]$

(f)
$\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{m}_{11}& {m}_{12}\\ {m}_{21}& {m}_{22}\end{array}\right]$

(g)
$\left[\begin{array}{cc}3& 3\end{array}\right]\left[\begin{array}{cc}6& 7\\ 4& 5\end{array}\right]$

(h)
$\left[\begin{array}{ccc}{a}_{11}& {a}_{12}& {a}_{13}\\ {a}_{21}& {a}_{22}& {a}_{23}\\ {a}_{31}& {a}_{32}& {a}_{33}\end{array}\right]\left[\begin{array}{ccc}{b}_{11}& {b}_{12}& {b}_{13}\\ {b}_{21}& {b}_{22}& {b}_{23}\end{array}\right]$

For each of the following matrices, multiply on the left by the row vector
$[5,1,2]$
.
Then consider whether multiplication on the right by the column
vector
$[5,1,2{]}^{\mathrm{T}}$
will give the same or a different result. Finally,
perform this multiplication to confirm or correct your expectation.

(a)
$\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$

(b)
$\left[\begin{array}{ccc}2& 5& 3\\ 1& 7& 1\\ 2& 1& 4\end{array}\right]$

(c)
$\left[\begin{array}{ccc}1& 7& 2\\ 7& 0& 3\\ 2& 3& 1\end{array}\right]$
This is an example of a symmetric matrix.
A square matrix is symmetric if
${\mathbf{A}}^{\mathrm{T}}=\mathbf{A}$
.

(d)
$\left[\begin{array}{ccc}0& 4& 3\\ 4& 0& 1\\ 3& 1& 0\end{array}\right]$
This is an example of a skew symmetric or antisymmetric matrix.
A square matrix is skew symmetric if
${\mathbf{A}}^{\mathrm{T}}=\mathbf{A}.$
This implies that the diagonal elements of a skew symmetric matrix must be 0.

Manipulate the following matrix expressions to remove the parentheses.

(a)
${\left({\left({\mathbf{A}}^{\mathrm{T}}\right)}^{\mathrm{T}}\right)}^{\mathrm{T}}$

(b)
${\left({\mathbf{B}\mathbf{A}}^{\mathrm{T}}\right)}^{\mathrm{T}}\left({\mathbf{C}\mathbf{D}}^{\mathrm{T}}\right)$

(c)
${\left(\left({\mathbf{D}}^{\mathrm{T}}{\mathbf{C}}^{\mathrm{T}}\right){\left(\mathbf{A}\mathbf{B}\right)}^{\mathrm{T}}\right)}^{\mathrm{T}}$

(d)
${\left({\left(\mathbf{A}\mathbf{B}\right)}^{\mathrm{T}}{\left(\mathbf{C}\mathbf{D}\mathbf{E}\right)}^{\mathrm{T}}\right)}^{\mathrm{T}}$

Describe the transformation
$\mathbf{a}\mathbf{M}=\mathbf{b}$
represented by each of the following matrices.

(a)
$\mathbf{M}=\left[\begin{array}{cc}0& 1\\ 1& 0\end{array}\right]$

(b)
$\mathbf{M}=\left[\begin{array}{cc}\frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}\end{array}\right]$

(c)
$\mathbf{M}=\left[\begin{array}{cc}2& 0\\ 0& 2\end{array}\right]$

(d)
$\mathbf{M}=\left[\begin{array}{cc}4& 0\\ 0& 7\end{array}\right]$

(e)
$\mathbf{M}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

(f)
$\mathbf{M}=\left[\begin{array}{cc}0& 2\\ 2& 0\end{array}\right]$

For 3D row vectors
$\mathbf{a}$
and
$\mathbf{b}$
, construct a
$3\times 3$
matrix
$\mathbf{M}$
such that
$\mathbf{a}\times \mathbf{b}=\mathbf{a}\mathbf{M}$
.
That is, show that the cross product of
$\mathbf{a}$
and
$\mathbf{b}$
can be represented as
the matrix product
$\mathbf{a}\mathbf{M}$
, for some matrix
$\mathbf{M}$
. (Hint: the
matrix will be skewsymmetric.)

Match each of the following figures (1–4) with their corresponding transformations.

$\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

$\left[\begin{array}{cc}2.5& 0\\ 0& 2.5\end{array}\right]$

$\left[\begin{array}{cc}\frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}\end{array}\right]$

$\left[\begin{array}{cc}1.5& 0\\ 0& 2.0\end{array}\right]$

Given the
$10\times 1$
column vector
$\mathbf{v}$
, create a matrix
$\mathbf{M}$
that,
when multiplied by
$\mathbf{v}$
, produces a
$10\times 1$
column vector
$\mathbf{w}$
such that
$${w}_{i}=\{\begin{array}{ll}{v}_{1}& \text{if~}i=1,\\ {v}_{i}{v}_{i1}& \text{if~}i>1.\end{array}$$
Matrices of this form arise when some continuous function is discretized. Multiplication
by this first difference matrix is the discrete equivalent of continuous
differentiation. (We'll learn about differentiation in Chapter 11
if you haven't already had calculus.)

Given the
$10\times 1$
column vector
$\mathbf{v}$
, create a matrix
$\mathbf{N}$
that,
when multiplied by
$\mathbf{v}$
, produces a
$10\times 1$
column vector
$\mathbf{w}$
such that
$${w}_{i}=\sum _{j=1}^{i}{v}_{j}.$$
In other words, each element becomes the sum of that element and all previous elements.
This matrix performs the discrete equivalent of integration, which as you
might already know (but you certainly will know after reading Chapter 11)
is the inverse operation of differentiation.

Consider
$\mathbf{M}$
and
$\mathbf{N}$
, the matrices from Exercises 10
and 11.

Discuss your expectations of the product
$\mathbf{M}\mathbf{N}$
.

Discuss your expectations of the product
$\mathbf{N}\mathbf{M}$
.

Calculate both
$\mathbf{M}\mathbf{N}$
and
$\mathbf{N}\mathbf{M}$
.
Were your expectations correct?
To be civilized is to be potentially master of all possible ideas,
and that means that one has got beyond being shocked,
although one preserves one's own moral aesthetic preferences.
— Oliver Wendell Holmes (1809–1894)
References
[1]
Tomas AkenineMöller, Eric Haines, and Natty Hoffman.
RealTime Rendering, Third edition.
Natick, MA: A K Peters, Ltd., 2008.
http://www.realtimerendering.com/.
[2]
Gilbert Strang.
Computational Science and Engineering.
Cambridge, UK: WellesleyCambridge, 2007.
[3]
Gilbert Strang.
Introduction to Linear Algebra, Fourth edition.
Cambridge, UK: WellesleyCambridge, 2009.