Theory

Scalar product, orthogonality and the spectral theorem

The standard scalar product, norm and angles, orthonormal bases, orthogonal projections with the Gram-Schmidt process, orthogonal matrices and the orthogonal diagonalization of symmetric matrices.

On this page

Standard scalar product and norm
Cauchy-Schwarz inequality and angle between vectors

Standard scalar product and norm

1. Standard scalar product

Given $u = (u_{1}, \dots, u_{n})$ and $v = (v_{1}, \dots, v_{n})$ in $R^{n}$ , the standard scalar product of $u$ and $v$ is the real number

⟨ u, v ⟩ = u_{1} v_{1} + u_{2} v_{2} + \dots + u_{n} v_{n},

also denoted by $u \cdot v$ . Writing the vectors as columns one gets $⟨ u, v ⟩ = u^{T} v$ , that is, the row-by-column product of the row $u^{T}$ by the column $v$ ; this is the form used in computations with matrices. For $u = (1, 2, - 1)$ and $v = (3, 0, 1)$ one obtains $⟨ u, v ⟩ = 3 + 0 - 1 = 2$ . In $R^{2}$ and in $R^{3}$ the definition coincides with the scalar product of geometric vectors, for which we refer to the topic on vectors in the plane and in space.

2. Properties of the scalar product

For every $u, v, w \in R^{n}$ and every $α, β \in R$ the following hold:

symmetry: $⟨ u, v ⟩ = ⟨ v, u ⟩$ ;
bilinearity: $⟨ α u + β w, v ⟩ = α ⟨ u, v ⟩ + β ⟨ w, v ⟩$ , and likewise in the second argument;
positive definiteness: $⟨ v, v ⟩ = v_{1}^{2} + \dots + v_{n}^{2} \geq 0$ , with equality only if $v = 0$ .

The three properties follow directly from the definition, because the scalar product is a sum of products of components. Bilinearity makes it possible to expand products as in algebra: for example $⟨ u + v, u + v ⟩ = ⟨ u, u ⟩ + 2 ⟨ u, v ⟩ + ⟨ v, v ⟩$ .

Scalar products on a vector space

On any real vector space $V$ , a scalar product is any function $⟨ \cdot, \cdot ⟩ : V \times V \to R$ that is symmetric, bilinear and positive definite. An example on $M_{n} (R)$ is $⟨ A, B ⟩ = tr (A^{T} B)$ , which is the sum of the products of the entries in the same position. The definitions and theorems of this topic use only the three properties, therefore they hold for every scalar product; in exam exercises the scalar product is almost always the standard one on $R^{n}$ .

3. Norm and distance

The norm of $v \in R^{n}$ is the number

∥ v ∥ = ⟨ v, v ⟩ = v_{1}^{2} + \dots + v_{n}^{2},

and the distance between $u$ and $v$ is $d (u, v) = ∥ u - v ∥$ . By positive definiteness the norm is zero only for the zero vector; from bilinearity it follows that $∥ α v ∥ = ∣ α ∣ ∥ v ∥$ . A vector of norm $1$ is called a unit vector: if $v \neq = 0$ , the vector $\frac{v}{∥ v ∥}$ is a unit vector with the same direction and the same orientation as $v$ , and passing from $v$ to its unit vector is called normalization. For $v = (1, 2, - 1)$ one has $∥ v ∥ = 6$ , therefore the unit vector is $\frac{1}{6} (1, 2, - 1)$ .

Cauchy-Schwarz inequality and angle between vectors

1. Cauchy-Schwarz inequality

For every $u, v \in R^{n}$

∣ ⟨ u, v ⟩ ∣ \leq ∥ u ∥ ∥ v ∥,

and equality holds if and only if $u$ and $v$ are linearly dependent. The inequality guarantees that, for nonzero $u$ and $v$ , the quotient $\frac{⟨ u , v ⟩}{∥ u ∥ ∥ v ∥}$ lies in $[- 1, 1]$ , and the definition of angle in $R^{n}$ is based on this fact.

Proof

If $v = 0$ both sides are zero and the vectors are dependent. Let $v \neq = 0$ . For every $t \in R$ , positive definiteness and bilinearity give

0 \leq ∥ u - t v ∥^{2} = ∥ v ∥^{2} t^{2} - 2 ⟨ u, v ⟩ t + ∥ u ∥^{2} .

The right-hand side is a polynomial of degree two in $t$ with positive leading coefficient that takes no negative values, therefore its discriminant is less than or equal to zero:

4 ⟨ u, v ⟩^{2} - 4∥ u ∥^{2} ∥ v ∥^{2} \leq 0,

that is, $⟨ u, v ⟩^{2} \leq ∥ u ∥^{2} ∥ v ∥^{2}$ , from which the statement follows by taking the square root. The discriminant is zero if and only if the polynomial has a root $t_{0}$ , that is, if and only if $∥ u - t_{0} v ∥ = 0$ ; by positive definiteness this is equivalent to $u = t_{0} v$ , that is, to the linear dependence of $u$ and $v$ .

2. Triangle inequality

For every $u, v \in R^{n}$ one has $∥ u + v ∥ \leq ∥ u ∥ + ∥ v ∥$ . In terms of distances, $d (u, w) \leq d (u, v) + d (v, w)$ for every $u, v, w$ : in a triangle each side is less than or equal to the sum of the other two.

Proof

Expanding with bilinearity and applying the Cauchy-Schwarz inequality one obtains

∥ u + v ∥^{2} = ∥ u ∥^{2} + 2 ⟨ u, v ⟩ + ∥ v ∥^{2} \leq ∥ u ∥^{2} + 2∥ u ∥ ∥ v ∥ + ∥ v ∥^{2} = (∥ u ∥ + ∥ v ∥)^{2},

and the statement follows by taking the square root, since both sides are nonnegative.

3. Angle between two vectors

Let $u, v \in R^{n}$ be nonzero. The angle between $u$ and $v$ is the unique $θ \in [0, π]$ such that

cos θ = \frac{⟨ u , v ⟩}{∥ u ∥ ∥ v ∥} .

The definition makes sense for every $n$ thanks to the Cauchy-Schwarz inequality, and in $R^{2}$ and $R^{3}$ it gives back the geometric angle. The sign of the scalar product gives the type of angle: $θ$ is acute if $⟨ u, v ⟩ > 0$ , right if $⟨ u, v ⟩ = 0$ , obtuse if $⟨ u, v ⟩ < 0$ . For $u = (3, 1)$ and $v = (1, 2)$ one gets $⟨ u, v ⟩ = 5$ , $∥ u ∥ = 10$ , $∥ v ∥ = 5$ , therefore $cos θ = \frac{5}{50} = \frac{1}{2}$ and $θ = \frac{π}{4}$ .

The vectors

u = (3, 1)

and

v = (1, 2)

form the angle

θ = \frac{π}{4}

, since

cos θ = \frac{1}{2}

The theory continues with 7 more sections (orthogonality and orthonormal bases, orthogonal projection, the gram schmidt process…), plus all worked examples, the solutions and the PDF.

Unlock now