Thispageintentionallyleftblank
This page intentionally left blank
1Linear algebra1.1IntroductionMultivariate analysis deals with issues related to the observations of many,usually correlated, variables on units of a selected random sample.Theseunits can be of any nature such as persons, cars, cities, etc. The observa-tions are gathered as vectors; for each selected unit corresponds a vectorof observed variables. An understanding of vectors,matrices,and,moregenerally, linear algebra is thus fundamental to the study of multivariateanalysis. Chapter 1 represents our selection of several important resultson linear algebra. They will facilitate a great many of the concepts inmultivariate analysis. A useful reference for linear algebra is Strang (1980).1.2Vectors and matricesTo express the dependence of the x ERn on its coordinates, wemay writeanyofx=(ri, i=1,...,n)=()=In this manner,x is envisaged as a "column"vector.The transpose of x isthe“"row"vector x'e Rnx=()=(r1,...,an)
1 Linear algebra 1.1 Introduction Multivariate analysis deals with issues related to the observations of many, usually correlated, variables on units of a selected random sample. These units can be of any nature such as persons, cars, cities, etc. The observations are gathered as vectors; for each selected unit corresponds a vector of observed variables. An understanding of vectors, matrices, and, more generally, linear algebra is thus fundamental to the study of multivariate analysis. Chapter 1 represents our selection of several important results on linear algebra. They will facilitate a great many of the concepts in multivariate analysis. A useful reference for linear algebra is Strang (1980). 1.2 Vectors and matrices To express the dependence of the x ∈ Rn on its coordinates, we may write any of x = (xi, i = 1,.,n)=(xi) = x1 . . . xn . In this manner, x is envisaged as a “column” vector. The transpose of x is the “row” vector x ∈ Rn x = (xi) = (x1,.,xn)
21.LinearalgebraAn m x n matrix A E Rm may also be denoted in various ways:(a11a1nA=(aj,i=1,..,m, j=1,...,n)=(a)=::(amlamnThe transpose of A is the n x m matrix A' e Rm:a11aml...A'= (at)= (aj) =:..ainamn..Asquare matrix S e Rn satisfying S =S'is termed symmetric.Theproduct of the m × n matrix A by the n × p matrix B is the m × p matrixC=ABfor whichnaikbkicii=k=1The trace of A e Rn is tr A = r- aui and one verifies that for Ae Rmand BERm,tr AB=trBAIn particular, row vectors and column vectors are themselves matrices,so that for x, y e Rn, we have the scalar resultxy-rui=y'x.This provides the standard inner product,(x.y)=x'y,in Rn with theassociated"euclidian norm" (length or modulus)1/2[x| = (x, x)1/2The Cauchy-Schwarz inequality is now proved.Proposition 1.1 (x,y)l≤x| lyl, Vx,y E R",with equality if and onlyif(iff)x=\yforsome>ER.Proof.If x =y.forsome)ER.theequalityclearlyholds.If not.0 < [x - Ay/2 = x/2 - 2)(x, y) + >2|y|2, V> e R; thus, the discriminant of口the quadratic polynomial must satisfy 4(x,y)?- 4/x/2/y/2<0.Thecosineof theangle betweenthevectorsx0and y0 is just(x,y)cos(0) [x/ /ylOrthogonality is another associated concept.Two vectors x and y in IRnwill be said to be orthogonal iff (x,y) = O.In contrast, the outer (ortensor)product of x and y is an n × n matrixxy'=(riyi)
2 1. Linear algebra An m × n matrix A ∈ Rm n may also be denoted in various ways: A = (aij , i = 1, . . . , m, j = 1,.,n)=(aij ) = a11 ··· a1n . . . . . . . am1 ··· amn . The transpose of A is the n × m matrix A ∈ Rn m: A = (aij ) = (aji) = a11 ··· am1 . . . . . . . a1n ··· amn . A square matrix S ∈ Rn n satisfying S = S is termed symmetric. The product of the m × n matrix A by the n × p matrix B is the m × p matrix C = AB for which cij = n k=1 aikbkj . The trace of A ∈ Rn n is tr A = n i=1 aii and one verifies that for A ∈ Rm n and B ∈ Rn m, tr AB = tr BA. In particular, row vectors and column vectors are themselves matrices, so that for x, y ∈ Rn, we have the scalar result x y = n i=1 xiyi = y x. This provides the standard inner product, x, y = x y, in Rn with the associated “euclidian norm” (length or modulus) |x| = x, x 1/2 = n i=1 x2 i 1/2 . The Cauchy-Schwarz inequality is now proved. Proposition 1.1 |x, y| ≤ |x| |y|, ∀x, y ∈ Rn, with equality if and only if (iff ) x = λy for some λ ∈ R. Proof. If x = λy, for some λ ∈ R, the equality clearly holds. If not, 0 < |x − λy| 2 = |x| 2 − 2λx, y + λ2|y| 2, ∀λ ∈ R; thus, the discriminant of the quadratic polynomial must satisfy 4x, y2 − 4|x| 2|y| 2 < 0. ✷ The cosine of the angle θ between the vectors x = 0 and y = 0 is just cos(θ) = x, y |x| |y| . Orthogonality is another associated concept. Two vectors x and y in Rn will be said to be orthogonal iff x, y = 0. In contrast, the outer (or tensor) product of x and y is an n × n matrix xy = (xiyj )
31.3.Image spaceand kerneland thisproduct is not commutative.The concept of orthonormal basis plays amajorrole in linear algebra.Asetfvi) of vectors inRn is orthonormalif『o,ijv'V, = ou =l,i=j.The symbol Si is referred to as the Kronecker delta. The Gram-Schmidtorthogonalizationmethodgivesaconstruction ofan orthonormal basisfroman arbitrary basis.Proposition 1.2 Let [vi,...,vn) be a basis of R".Defineui=Vi/lvil,ui=wi/lwil,where wi = vi - Ei-i(v,u,)uj, i = 2,...,n. Then, [ui,...,un) is anorthonormal basis1.3Image space and kernelNow, a matrix may equally well be recognized as a function either of itscolumn vectors or its row vectors:(gi)A= (ai,...,an) -+(gm）fora,ERm, j=l,...,n or g, eRn, i=l,...,m.If wethenwriteB= (bi,...,bp)with bje R", j=1,...,P,we find thatAB= (Ab1,...,Abp)=(g/b)In particular, for x Rn,we have expressly thatTi)nT(1.1)Ax= (ai,...,an)Tiai=1orgigixAx=(1.2)......x(gm(gmx)The orthogonal complement of a subspaceVCRn is,by definition,thesubspaceVI=(yeR":ylx, VxeV)
1.3. Image space and kernel 3 and this product is not commutative. The concept of orthonormal basis plays a major role in linear algebra. A set {vi} of vectors in Rn is orthonormal if v ivj = δij = 0, i = j 1, i = j. The symbol δij is referred to as the Kronecker delta. The Gram-Schmidt orthogonalization method gives a construction of an orthonormal basis from an arbitrary basis. Proposition 1.2 Let {v1,., vn} be a basis of Rn. Define u1 = v1/|v1|, ui = wi/|wi|, where wi = vi − i−1 j=1(v iuj )uj , i = 2,.,n. Then, {u1,., un} is an orthonormal basis. 1.3 Image space and kernel Now, a matrix may equally well be recognized as a function either of its column vectors or its row vectors: A = (a1,., an) = g 1 . . . g m for aj ∈ Rm, j = 1,.,n or gi ∈ Rn, i = 1,.,m. If we then write B = (b1,., bp) with bj ∈ Rn, j = 1,.,p, we find that AB = (Ab1,., Abp)=(g ibj ). In particular, for x ∈ Rn, we have expressly that Ax = (a1,., an) x1 . . . xn = n i=1 xiai (1.1) or Ax = g 1 . . . g m x = g 1x . . . g mx . (1.2) The orthogonal complement of a subspace V ⊂ Rn is, by definition, the subspace V⊥ = {y ∈ Rn : y ⊥ x, ∀x ∈ V}
41. Linear algebraExpression (1.1) identifies the image space of A, Im A = (Ax : x E R"],with the linear span of its column vectors and the expression (1.2) revealsthe kernel, ker A = (x E Rn : Ax = 0], to be the orthogonal complementof the row space, equivalently ker A = (Im A'). The dimension of thesubspace Im A is called the rank of A and satisfies rank A = rank A'whereas the dimension of ker A is called the nullity of A. They are relatedthrough the following simple relation:Proposition 1.3 For any AERm, n=nullity A+rank A.Proof. Let (vi,..., vv) be a basis of ker A and extend it to a basis[Vi,..., Vv, V+1,...,Vn]of Rn. One can easily check [Avv+1,..., Avn] is a basis of Im A. Thus,口n = nullity A+rank A.1.4 Nonsingular matrices and determinantsWe recall some basic facts about nonsingular (one-to-one)linear transfor-mations and determinantsBy writing A Rn in terms of its column vectors A = (ai,...,an) withaj E Rn, j=l,...,n, it is clear thatA is one-to-oneai,...,an is a basis ker A= o]and also from the simplerelation n=nullityA+rank A,Ais one-to-one→Aisone-to-oneand onto.These are all equivalent ways of saying A has an inverse or that A is non-singular.Denote by o(1),...,o(n) a permutation of 1,...,n and by n(o)its parity.Let Sn be the group of all the n! permutations. The determinantis, by definition, the unique function det : Rn -→ R, denoted [A/= det(A),that is,(i) multilinear: linear in each of ai,...,an separately(ii) alternating: [(ag(1),...,ag(n)|= (-1)n(o) [(ai,...,an)l(iii) normed: |1|= 1.This produces the formula[A| = (-1)n()a1o(1). ana(n)aESnby which one verifiesAB|=|A/|B| and |A'|=|A/:
4 1. Linear algebra Expression (1.1) identifies the image space of A, Im A = {Ax : x ∈ Rn}, with the linear span of its column vectors and the expression (1.2) reveals the kernel, ker A = {x ∈ Rn : Ax = 0}, to be the orthogonal complement of the row space, equivalently ker A = (Im A )⊥. The dimension of the subspace Im A is called the rank of A and satisfies rank A = rank A , whereas the dimension of ker A is called the nullity of A. They are related through the following simple relation: Proposition 1.3 For any A ∈ Rm n , n = nullity A + rank A. Proof. Let {v1,., vν} be a basis of ker A and extend it to a basis {v1,., vν, vν+1,., vn} of Rn. One can easily check {Avν+1,., Avn} is a basis of Im A. Thus, n = nullity A + rank A. ✷ 1.4 Nonsingular matrices and determinants We recall some basic facts about nonsingular (one-to-one) linear transformations and determinants. By writing A ∈ Rn n in terms of its column vectors A = (a1,., an) with aj ∈ Rn, j = 1,.,n, it is clear that A is one-to-one ⇐⇒ a1,., an is a basis ⇐⇒ ker A = {0} and also from the simple relation n = nullity A + rank A, A is one-to-one ⇐⇒ A is one-to-one and onto. These are all equivalent ways of saying A has an inverse or that A is nonsingular. Denote by σ(1),.,σ(n) a permutation of 1,.,n and by n(σ) its parity. Let Sn be the group of all the n! permutations. The determinant is, by definition, the unique function det : Rn n → R, denoted |A| = det(A), that is, (i) multilinear: linear in each of a1,., an separately (ii) alternating: aσ(1),., aσ(n) = (−1)n(σ) |(a1,., an)| (iii) normed: |I| = 1. This produces the formula |A| = σ∈Sn (−1)n(σ) a1σ(1) ··· anσ(n) by which one verifies |AB| = |A| |B| and |A | = |A|