Linear Algebra I: Vector Spaces and Linear Maps

Part I of a six-part lecture series converging on the singular value decomposition and geometric algebra.

0. Why this series exists

A chemistry student is asked to balance the combustion of methane. She writes down the skeleton

and then juggles the coefficients until the carbon, hydrogen and oxygen counts agree, arriving at . She is told this is bookkeeping. It is linear algebra. Element conservation is three linear equations in four unknowns; the balanced reaction is a vector in the null space of a matrix; the balance is unique up to overall scale because that null space is one-dimensional. Every fact she was asked to accept as a knack is a theorem, and the theorem is proved in this lecture.

The same student, two chapters later, draws an ICE table for the Haber process and solves for the extent of reaction. The table records the amounts

which is a straight line in the space of species amounts, and the equilibrium condition that cuts that line is a quartic. Both objects live in the same little table, in the same variable , and they could not be more different in kind. Nobody says so. The words that would say so have been worn smooth.

That is the thesis of this series. The subject is taught as matrix manipulation, and the manipulation hides the linear maps. Row-reduce, invert, multiply, take a determinant: a curriculum of procedures on arrays of numbers, in which the objects the arrays represent are never named. The cost is paid later, in every application area the reader will meet. A chemical engineer calls an ICE table linear. A deep-learning framework calls an array a tensor. A textbook calls a linear layer. A paper calls attention scores an inner product. Each of these is false, each has a true statement standing right behind it, and the true statement is more useful.

The arc. Part I builds vector spaces, bases, dimension, linear maps, rank-nullity, matrices as coordinate representations, the four fundamental subspaces, and the solution set of , applied to stoichiometry. Part II builds eigenstructure, the Jordan form and the matrix exponential, applied to linear systems of ordinary differential equations. Part III builds inner products, orthogonality and the spectral theorem, applied to the attention mechanism. Part IV builds tensors and the calculus of matrix functions, applied to backpropagation. Part V proves the singular value decomposition and applies it to principal component analysis and low-rank adaptation. Part VI builds geometric algebra, and the six parts converge on one theorem. A linear map moves vectors, and it also moves the areas and volumes they span. Part VI extends so that it acts on all of these at once. If the singular values of are , that extension stretches a -dimensional piece by a product of of the , and it stretches -dimensional volume by , which is . Rotation, stretch, area, volume and determinant turn out to be one object. That is where the series ends, and every part is aimed at it.

What is assumed. Arithmetic, and the concept of a function as a rule assigning one output to each input. Nothing else. Matrices are constructed here rather than presumed, and no result is used before it is proved.

Relation to the manifolds series. Analysis on Manifolds is self-contained: it builds the linear algebra it needs, exactly as much of it as the derivative and exterior algebra require, and it builds it from scratch. This series is the fuller account of the linear side, running far past what an analysis course has room for, and the two are complementary. A reader may take either first.

0.1. Terminology

Before any mathematics, four words. The series uses them in one sense each, and never in any other.

These definitions pay for themselves in every application the series touches. In each one, the loose word hides a fact worth knowing.

Each row is a promissory note, settled in the part that owns it. The chemistry row is settled in §7 of this lecture.

1. Vector spaces

This section fixes the arena. Everything the series proves is a statement about vector spaces and the linear maps between them, so the axioms come first and the examples come immediately after, to show how little the axioms demand.

1.1. Fields

Vectors are scaled by numbers, and the numbers must themselves be well behaved. A field is exactly the amount of good behaviour required.

Throughout, denotes a field, and the reader who prefers may read everywhere without losing anything until Part II, where the complex numbers become unavoidable.

1.2. The vector space axioms

1.3. Subspaces

2. Span, independence, basis

This section fixes what it means to give coordinates to a space, and proves the one fact that makes coordinates trustworthy: every basis of a given space has the same number of elements. That fact is the Steinitz exchange lemma, and it is the first real theorem of the subject.

2.1. Span

2.2. Linear independence

2.3. Basis and the Steinitz exchange

The next theorem is the one everything rests on. It says that an independent list can never be longer than a spanning list, and everything about dimension follows.

3. Linear maps

The objects are in place. This section introduces the linear maps between them, and proves the theorem that governs every one of them.

3.1. Definition, kernel, image

3.2. Rank-nullity

4. Matrices as coordinates

Now the central point of the lecture. A matrix has not appeared yet, and it appears now for one reason: a linear map, once bases are chosen at both ends, is determined by finitely many scalars. Those scalars are the matrix. They belong to the pair of bases as much as to the map, and changing the bases changes every one of them while changing nothing about the map.

4.1. The matrix of a linear map

4.2. Change of basis and similarity

Three panels, three separate things. Panel one draws the linear map with no frame present at all: the image of the unit sphere under , with the directions leaves invariant. Panel two is a basis you may rotate. Panel three is the array of components. Turn the frame and watch: the nine numbers churn under [eq:eq-similarity], and the object in panel one does not stir, because the map never heard of your basis. The trace and the determinant sit still while the entries move.

One control is offered here, and a second is held back until Part IV. Rotations are gentle: an orthogonal change of basis is the special case in which several inequivalent transformation laws happen to agree, which is exactly why an array can be mistaken for a tensor indefinitely. Part IV unlocks the shear, the agreement collapses, and the array is caught.

5. The four fundamental subspaces

Given a matrix, four subspaces come for free, two in the source and two in the target. This section defines them and proves the one theorem tying them together, which is that a matrix has the same rank read by rows as read by columns.

6. Solving

Everything proved so far collapses into one clean statement about the oldest question in the subject. This section answers when a linear system has a solution, when it has only one, and what the set of all of them looks like.

The figure runs [eq:eq-rank-nullity] as a conservation law. Slide the rank of a linear map and the two dimensions trade against each other: as the image collapses from a solid to a plane to a line, the kernel grows from a point to a line to a plane, and the sum stays at three. A system whose image has collapsed is one whose may now fail to be reachable, and whose solution set, where it exists, has become a large affine set rather than a point. Existence and uniqueness are the two ends of the same slider.

7. Stoichiometry: an affine set cut by a nonlinear variety

Here is the sharpest statement in this lecture, and the reason chemistry was chosen as the application. Take the Haber process, , and write down its ICE table. The amounts are

which is degree one in : an affine set, a straight line in the space of species amounts, direction , base point . The equilibrium condition that picks a point on that line is

which, cleared of denominators, is degree four in . The quotient on the left is written in mole amounts rather than activities, and the that appears in the worked problem of §7.3 is a number chosen to make that problem come out, not a measured constant; the remark in §7.3 says exactly what is being assumed, and nothing in the degree count depends on it. Same variable, same table, two utterly different kinds of object: a line and a quartic hypersurface. The line is linear algebra and this lecture solves it completely; the quartic is not and never will be. A course that calls the whole table "linear" has thrown away the only distinction that explains why one half is easy.

7.1. Balancing is a null-space search

For methane combustion, with species ordered and elements ordered , the atom counts are read straight off the formulae:

Three elements, four species, so has rank at most three and by [rank-transpose]: there is always at least one balancing vector, before any arithmetic is done. The rank is in fact three, so the kernel is exactly one-dimensional, and the balance is unique up to scale. Scaling to the primitive integer vector, and fixing the sign so that methane is consumed, gives : methane and oxygen negative, carbon dioxide and water positive, magnitudes . That is the balanced equation

the answer the student got by juggling. Two facts she was never told are now theorems. Her answer is unique up to scale because . And whenever a reaction network has , no amount of juggling gives a unique answer, because there is no unique answer to be had: the number of independent reactions among a set of species is , which is the statement of [rank-nullity] in a laboratory.

from sympy import Matrix, Rational, ilcm, igcd

# Species: CH4, O2, CO2, H2O.  Elements: C, H, O.
# E holds ATOM COUNTS only: every entry non-negative, no reaction in sight.
E = Matrix([[1, 0, 1, 0],
          [4, 0, 0, 2],
          [0, 2, 2, 1]])

print("E =", E.tolist())
print("rank E =", E.rank())

ns = E.nullspace()
print("dim ker E =", len(ns), "(unique up to scale iff this is 1)")

# Clear the rational null-space vector to a primitive integer vector.
v = ns[0]
den = 1
for e in v:
  den = ilcm(den, Rational(e).q)
ints = [int(Rational(e) * den) for e in v]
g = 0
for k in ints:
  g = igcd(g, k)
ints = [k // g for k in ints]

# Fix the sign so that methane (species 0) is CONSUMED: nu_0 < 0.
if ints[0] > 0:
  ints = [-k for k in ints]

print("signed coefficients (CH4, O2, CO2, H2O) =", ints)
print("E . nu =", (E * Matrix(ints)).T.tolist(), "-> every element conserved")
print("magnitudes match the textbook answer [1, 2, 1, 2]:",
    [abs(k) for k in ints] == [1, 2, 1, 2])

7.2. Conservation laws are a left null space

A network of several reactions gets a second matrix, and the two must be told apart.

7.3. The ICE table: degree one meets degree four

Return to the Haber process with the feed in the species order . By [eq:eq-ice-affine],

each of them degree one in . The whole ICE table is one point moving along one line, and the extent of reaction is a coordinate on that line, in the sense of [coordinates-unique] applied to the one-dimensional direction space . The feed is in stoichiometric ratio, so both reactants are exhausted together at , and the physically admissible segment is .

Substituting the affine amounts into [eq:eq-equilibrium] and clearing denominators,

Three species amounts on three axes, one point in the space of states. The straight line is the affine set [eq:eq-ice-affine], along which the extent of reaction is the coordinate. The flat sheet is a conservation plane, one row of , and the line lies inside it exactly because that row is in the left null space of ([conservation-left-null]). The curved sheet is the equilibrium variety [eq:eq-equilibrium], which is where the linearity stops. Drag and watch the reaction quotient climb to at the one point where the line pierces the curve. The straight object and the curved object are drawn in the same picture, because the point of the picture is that they are two objects.

8. The determinant we have not defined

A debt, stated plainly. The determinant has been leaned on twice in this lecture without ever being defined: [p-invertible] asserts a change-of-basis matrix is invertible, and every reader knows the test for invertibility is a nonvanishing determinant, and the remark after [similarity] claimed the determinant survives a change of basis. Neither claim was proved, because the determinant cannot be properly defined at this stage. The formula with the sum over permutations computes something not yet named.

The debt is settled three times, each settlement stronger than the last.

• Part II axiomatises it: is the unique function of the columns of a matrix that is multilinear, alternating, and equal to one on the identity. Uniqueness makes the permutation formula a theorem rather than a definition, invariance under [eq:eq-similarity] falls out of multiplicativity, and the meaning of is signed volume. Part II needs it because the characteristic polynomial needs it.
• Part IV explains it: a linear map on an -dimensional space induces a linear map on the one-dimensional space , and a linear map on a one-dimensional space is multiplication by a scalar. That scalar is . The alternating multilinear axioms of Part II are then simply the definition of , and the reason the determinant is basis-independent is that was constructed without a basis.
• Part VI reveals it: is the factor by which stretches -dimensional volume, and it equals , the product of the singular values. This is the top-dimensional case of the theorem stated in §0, in which rotation, stretch, area, volume and determinant become one object.

Part II begins with eigenvalues, and to have eigenvalues we need a determinant. That is where the series goes next.

References

1. S. Axler, Linear Algebra Done Right, 4th ed., Springer, 2024. DOI
2. P.R. Halmos, Finite-Dimensional Vector Spaces, 2nd ed., Springer, 1974. DOI
3. R.A. Horn and C.R. Johnson, Matrix Analysis, Cambridge University Press, 1985. DOI
4. G. Strang, Linear Algebra and Its Applications, 4th ed., Cengage, 2006.
5. R. Aris, Elementary Chemical Reactor Analysis, Prentice-Hall, 1969.