Linear Algebra

We've reviewed vector spaces, but the operations available to us are a little too basic to be useful yet. In this article, we'll introduce the concept of linear algebra.

In linear algebra, we'll be using the vectors introduced in the last article and adding some extra pieces called linear operators. Linear operators will be represented by capital letters like $A$, $B$, $C$ etc. Linear operators will act on vectors to produce new vectors. We'll notate it like this:

In English this says: When the linear operator $A$ acts on the vector $\mathbf{v}$, it produces the vector $\mathbf{w}$. Linear operators can also act across different vector spaces. i.e. in the example above, it might be the case that $\mathbf{v}$ and $\mathbf{w}$ are from different vector spaces.

We define the vector spaces a linear operator acts on as:

This says: the linear operator $A$ takes in vectors from the vector space $V$ and produces vectors in the vector space $W$.

It's worth at this point mentioning a word that might be on the mind of some readers: Matrices. If you've studied matrices before, you will notice that linear operators appear to be very similar. They both involve operations on vectors. Matrices are one possible representation of linear operators. But, as we've said before, we're more interested in studying abstract objects and their properties rather than specific representations. So, we'll be focusing on the abstract properties of linear operators rather than concrete representations.

Linear operators only follow 2 very simple rules:

Additivity

Homogeneity

From these simple rules, we can derive a very important property of linear operators. Remember from the previous article we inroduced the idea of a basis. One of the properties we had for a basis is that it must be spanning. i.e. any vector in the vector space can be written as a linear combination of the basis vectors.

So, say we have a vector $\mathbf{v}$ in a vector space $V$. If we have a basis$\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n$ in this vector space, then we can write $\mathbf{v}$ as$\mathbf{v} = c_1\mathbf{e}_1 + c_2\mathbf{e}_2 + \ldots + c_n\mathbf{e}_n$ for some set of scalars $c_1, c_2, \ldots, c_n$.

Now look what happens if we apply a linear operator $A: V \rightarrow W$ to $\mathbf{v}$:

This shows us something very interesting: To calculate how $A$ acts on any vector $\mathbf{v}$ we only need to know how it acts on the basis vectors of the vector space. This is a very powerful property as it means, for example in a 2D vector space, we only need to know how $A$ acts on two basis vectors instead of every possible vector in the space.

Let's look for example at the identity operator $I$ acting on a 2D vector space. The identity operator is essentially the "do nothing" operator as it takes every vector to itself. We define $I$ on our 2 basis vectors as:

And can therefore verify that $I$ behaves as we expect:

So it works!

Inner-products

The next concept we'll need to introduce is an inner-product. An inner product is an operation that takes two vectors and produces a scalar. We'll denote it with a dot ($\cdot$) like this:

Note that we've reused the $\cdot$ symbol that we use to multiply scalars together: $a \cdot b$. This is just convention and usually won't be confusing in context as we denote scalars and vectors differently.

Here are the rules that an inner-product must follow:

Linearity in the second argument

This shows us that the inner product distributes over addition and scalar multiplication

Conjugate symmetry

Where $^*$ denotes the complex conjugate operator. Of course this is only relevant for vector spaces over the field of complex numbers. If we were working with real numbers we'd have: $\mathbf{v} \cdot \mathbf{w} = \mathbf{w} \cdot \mathbf{v}$

We specify this complex conjugate explicitly here since it will be useful later in the course.

Positive definiteness

This means that the inner-product of a vector with itself is always greater than or equal to $0$ and is only equal to $0$ if the vector is $\mathbf{0}$.

Remember that $0$ is the additive identity of the field and $\mathbf{0}$ is the additive identity of the vector space.

Some of you might be aware of something called the dot product, which is a specific inner-product in euclidean vector spaces but, as we've said before, we're more interested in the abstract properties of inner-products rather than any specific representations here.

With this inner-product, we will now define two new concepts: orthogonality and normality.

We say two vectors $\mathbf{v}$ and $\mathbf{w}$ are orthogonal if$\mathbf{v} \cdot \mathbf{w} = 0$.

We say a single vector $\mathbf{v}$ is normal if $\mathbf{v} \cdot \mathbf{v} = 1$. This is sometimes called a unit vector.

If some vectors are orthogonal and normal we will describe them as orthonormal.

Something we will be interested in are orthonormal bases. That is, a set of basis vectors that are not only linearly independent and spanning, but also orthonormal:

Why is this useful? Well, consider taking the inner product of two vectors $\mathbf{v}$ and $\mathbf{w}$. We can represent these vectors in terms of the orthonormal basis vectors: $\mathbf{v} = v_1\mathbf{e}_1 + v_2\mathbf{e}_2 + \ldots + v_n\mathbf{e}_n$and $\mathbf{w} = w_1\mathbf{e}_1 + w_2\mathbf{e}_2 + \ldots + w_n\mathbf{e}_n$. Then the inner product of these 2 vectors is:

But from our definition earlier, we know that the inner product of any 2 basis vectors is 0 unless $i = j$. So we can simplify it:

And since our basis vectors are normal, when we take the inner product of them with themselves we get 1:

So by representating our vectors in an orthonormal basis we managed to find a nice way to calculate it. This also shows us that no matter what orthonormal basis we chose, we would always get the same result which tells us that the inner product is independent of the basis we chose.

Exercises

Exercise 1

Exercise 2