//------------------------------// // Day 3 - Inner and Outer Products // Story: Theoretical Physics is Magic // by Smeapancol //------------------------------// [Best viewed with 'small' size text and 1.5 size spacing. If the equations seem out of order, reload the page and they might be fixed by then. For a pdf version of this chapter with better formatting, please click here.] Unit 1: Space Lesson 1- Algebra Day 3- Inner and Outer Products The next morning at dawn Twilight dutifully appeared in the Canterlot Castle courtyard. During peacetime the gates were open and there were a few ponies around sight-seeing On business. The sun was not quite up and she could see a tiny dark speck at the top of a tower, keeping watch over the night. She was tired from not being able to sleep in after her all-night studying session the other day, but she didn’t want to do anything that might make Luna cause more trouble. It was not a terribly great distance to the Canterlot but it was not the easiest commute to make every day. However, there was little one could do when a princess demands her presence. Presently Celestia alighted next to her sister on the tower to cast the spell to make the sun rise. Soon after, Luna gracefully floated down into the courtyard. “Good morning, disciple Twilight Sparkle. Are you ready?” “Yes, I mean such a lovely morning... who wouldn’t be ready for mathematics at this time?” said Twilight with a hint of sarcasm. “Lovely morning? Well I admit it is one of my sister’s better dawns but to tell you the truth I do not think she put as much effort into it as she could have!” “Er, it was a lovely night too I mean! I could tell that you worked hard at it.” Luna smiled. “Why thank you! That is right, I did! Now, today we'll take just a little step closer to geometry. Today I will begin to show you how Linear Algebra can take on a geometrical interpretation. Now the other day you suggested that a vector might be a thing with a magnitude and direction, which was, of course, completely wrong. It's not, of course, but let us think about things which do have a magnitude and direction and think about how our theory of vectors can apply to them.” “So, like velocities and accelerations.” “Yes, like those. So what is missing from our theory of vectors?” “I don’t know.” “Everything! Our vectors have neither a magnitude nor direction! That is another reason you should never define a vector that way.” “How can you say they don’t have a magnitude and direction? It seems like they do.” “What would a magnitude be? At the very least, it would be a function V→ℝ that tells how to come up with a length for each vector. But there is nothing like that in the definition! And as for a direction, just consider this example in .” Luna summoned a block of chalk, kicked off a piece with her forehoof, and wrote directly upon the inner wall of the Canterlot courtyard. You can see that v and w may look like they have nearly the same direction, but now The result of M applied to them is that they now look like they are in almost opposite directions. Remember how all M induces an isomorphism from to itself, so the relationship of the vectors before applying it is no more real than after. This just shows that a vector space on its own has no concept of angels.” “Wait a minute... what’s that?” “What?” “That square of numbers you called M!” “Oh dear... did I forget to tell you about matrices?” “Well actually I do know what a matrix is, but I want to hear your explanation.” “Very well!” said Luna, who smiled slightly at the flattery. “We know we can define a linear transformation Z:V→W by what it does to each basis vector in V, right? This is possible because, since linear combinations on a basis define vectors uniquely, a map on a basis uniquely defines a map on V without ever trying to send the same vector to different places. Say that I define a map Z:V→W by . In other words, each is sent by M to the corresponding . The together form a basis in V, but the might not form a basis in W. I can now write Z as (3.1) where I have written little arrow on top of the vectors because otherwise this will get confusing. If V is n-dimensional and W is m-dimensional, the linear map must define n vectors, each with m components. That means there are m n components in total that are required to specify the linear transformation, you see?” “Yes.” Definition 3.1 : Matrix “So just as you can denote a vector as a list of length n, you can denote a linear transformation by an m⨯n box, which is called a matrix. Normally this is done with the components of each written vertically like so, you see?” where means the jth column of the ith row of Z.” “It seems a little bit confusing,” said Twilight, “that the indices on the components of Z are reversed relative to those on the .” “It is confusing, but that is the convention. So watch out Twilight Sparkle! Now find the multiplication rule for Z in terms of the components of the vectors in W. You may use to represent a basis in W.” Twilight broke off her own piece of chalk. “I can start by writing each in terms of its components using the basis . (3.2) then I replace the ωs with zs.” (3.3) Proposition 3.1 “So,” said Twilight, each row of the matrix defines something like a linear combination on the components of . However, the and are numbers, not vectors, so it’s not quite the same.” “Correct. Actually, it is not bad at all that I forgot to tell you about matrices yesterday because this is just sort of operation we need to give vectors magnitudes and directions. I now define the dot product. (3.4) you just sum the product of the components of two vectors. Now we can redefine matrix multiplication like so: (3.5) where now, each row in the matrix is a kind of vector rather than each column.” “That’s much less confusing.” “And it gives us a notion of length too.” “Yes. If this were a vector in Euclidean space, then the length of a line segment is (3.6) by the Pythagorean theorem.” “The dot product rule. This rule also give us a concept of an angle. Think of two unit vectors and . This diagram shows that their dot product will be related to the angle between them. their dot product would be given by so you see that only the part of which is parallel to contributes to their dot product. and if were orthogonal to , then the dot product would be zero. That will become our definition of orthogonality soon. Twilight said, “So the angle would be given by (3.7) but does this diagram really give the right idea? You’ve actually drawn it so that one of the vectors is along an axis to make the conclusion obvious, but is that true in general?” “That is a good question. If neither vector was aligned to an axis, then we could just draw rotated axes to align with either of the vectors and the conclusion would be the same. Of course, that depends on knowing that the dot product is rotation-independent. In the future, we will construct spaces which we know are rotation independent, and then we will find that some algebraic expression has an especially simple form when rotated in alignment with some axis. Since the space is rotation-independent, we will know that it does not actually matter how the expression is aligned, and will make general conclusions from it. We must now ponder upon how this idea of a dot product can be generalized. We have an idea of the kind of thing we want, but we do not have the general idea yet. The dot product is not a real thing because it depends on the particular basis-two vectors will have a different dot product with one another if you just change the basis of the space. So we need to search for something more objective.” Twilight nodded. “But wait a minute-is that really necessary? The dot product works, right? Why don’t we just say that one basis is correct and use it? This is all too esoteric. That’s exactly what I had trouble with in physics-there are so many concepts that are not, strictly speaking, necessary at all!” Definition 3.2 : Operator Suppose I define a linear map H:V→V-a linear map from a space to itself is called an operator, by the way-which performs a coordinate transform on V. In other words, it is defined as a map from one basis of V to another. It is therefore invertible, correct?” “Now, as I was saying, I defined the operator H. What do you get if you want to change coordinates yet preserve the value of the dot product?” “I would have to multiply both vectors by H. That would be a change in coordinates. But then I would have to also multiply each by , since otherwise the dot product would not be preserved.” “Right. You can think of the and as being the new vectors and the two s as somehow modifying the dot product itself. In order to write this expression in a more understandable way, let us think about something simpler. Suppose we have an invertable operator and we define a new operator M by “Wait a minute. Are you sure there must actually be an operator M which can fit in there?” “Yes. behaves just like a change of basis on , so there must be some change of basis on that produces the same result in the end. Your job is to find the relationship between M and .” Twilight was perplexed. “I’m really not sure how I’d go about this.” “Let me give you another way of thinking about it. You know how to multiply a vector by an operator on the left. Imagine now that you can multiply a vector on the right. instead, and you want to figure out how to do it.” Twilight stared at the wall for a moment, and then she smiled. “I have it! Right multiplication by a matrix can be defined by the associativity rule. In other words, “Go on!” said Luna. “I just have to expand that expression out and see if I can find an equivalent (3.8) “Oh dear, this is going to get messy...” Twilight stopped. She could feel her four ankles quiver. “Somehow I’ll have to factor out , and then what’s leftover will be .” Luna whispered in her ear. “Stay calm and do this problem for me. I promise I will show you something wonderful when you are done!” Twilight blushed and wondered what Luna meant. However, her words had brought some of her confidence back. “Alright...” whimpered Twilight. “Try writing it as an summand,” Luna suggested. With trepidation, Twilight wrote on the wall again. (3.9) “There, you see!” said Luna. “You factored that whole thing out and it was as easy as moving a summand sign to the left. Now remember, by the rules of the dot product, unless i=j.” Twilight closed her eyes and tried to think about what Luna said. This would mean that she could set j=i in the summand and remove the sum over j! She wrote (3.10) “That is it!” whispered Luna. “But now what?” “Are you sure you want me to tell you?” Twilight’s heart was pounding. (3.11) “Bravo!” said Luna. “you have the right multiplication rule!” “It’s just like left multiplication... except that the sum is over the columns of the matrix rather than the rows.” “Exactly. Good disciple!” Twilight smiled in spite of herself. “And what was the wonderful thing you were going to show me?” Luna nodded mysteriously. “I will show you. Definition 3.3 : Abstract index notation You had been writing vectors something for your proof. This notation is very redundant. You know that a vector is a sum of components over a basis. All that matters is the components. So, get rid of the and get rid of the . Just write a vector as . The i is now a free index. I have used free indices earlier today, but now they have a special meaning. It means spatial vector. We have also been writing something like to denote the product of M with . But we know how to matrix multiply, so there is no need to write it out so explicitly. Instead, write and assume that the repeated index j must be summed over. The expression has one free index, which makes it a spatial vector. Then we just write a linear operator as , and the two free indices indicate that it is a linear operator.” “Isn’t it technically a matrix rather than a linear operator?” asked Twilight. “In writing each object with an index aren’t we implicitly denoting the object as an array with individual parts? I thought we always wanted to be careful not do anything that is basis dependent.” “Historically, you are correct. But the notation does not need to be interpreted in this way. It is true that we do not wish to do anything that depends on a particular basis and in fact we do not even know if every vector space has a basis. However, we can always think of the indices as simply denoting the kind of object and the kind of multiplication rather than an actual index over an array.” “I see.” “I want you to try to prove the same thing you just did using the abstract index notation. Try to show how right-multiplication of linear operators works.” Luna then wrote (3.12) Twilight squinted at the expression for a moment. “Well now there’s nothing to prove! The way that the matrix multiplication is written makes it obvious! and M are just transposes of one another.” “What have you learned, Twilight Sparkle?” “I went through all that horrible algebra with summands when I could have just done... nothing!” “Can you interpret a more generally applicable lesson to this?” Twilight hung her head. “I won’t discount the power of abstraction. I’ll try to learn how to use the best mental tools.” “That is right, Twilight Sparkle!” Luna turned back to the wall and with a wave of her hoof said, “Now to return to the generalization of the dot product. As we decided, I have written this in abstract index notation now. Changing the basis results in two new vectors, and , as well as a new operator , which looks like the product of an operator with its own transpose. So we can think of the more general form of the dot product, which we shall call the inner product. It will work something like that. Let’s define (3.13) This is a linear map which is a square of two linear maps. You will now prove some properties of . You may use whichever notation you find most convenient. Proposition 3.2 The first property is symmetry. You must show that .” Twilight wrote (3.14) and said, “For one of the steps, I had to rename some of the repeated indices-of course that’s fine because the letter on the indices is meaningless.” Proposition 3.3 “The second property,” said Luna, “is positive definiteness. That means , and it’s only zero when . Twilight said, “That one is easy. You can clearly see that is a dot product of a vector with itself. (3.15) The dot product of a vector with itself is a sum of squares, which we know will always positive, at least for real numbers.” Definition 3.4 : Inner-product space “Yes, and we will only bother to think about real vector spaces for now, so that is the most general result we need. However, eventually we shall want to generalize the concept. A real inner-product space is a vector space V over ℝ which comes with a symmetric, positive definite linear map M:V⨯V→ℝ. If then define (3.16) Definition 3.5 : Quadratic form and the operator g defines the linear map. The positive-definiteness of g implies that it is invertable. This is what I shall mean when I use the dot notation from now on. The operator g which defines the inner product is called a positive definite quadratic form. Definition 3.6 : Length, unit vector And finally we have an objective way to talk about lengths and angles. The length of a vector is is written and given by . A vector whose length is 1 is a unit vector. Definition 3.7 : Parallel, orthogonal Two unit vectors and are orthogonal if and parallel if . Definition 3.8 : Orthonormal basis Finally, these concepts allow us to construct a basis for our space that more closely resembles the familiar one from geometry. In geometry, you have the x, y, and z axes, and they are normally perpendicular to one another. And advancing along the x-coordinate by one unit is equivalent to a distance of one unit. Now we know how to make something like that because we know how to require that each vector in a basis is perpendicular to all the others and have a length of one.” Twilight began to fume with frustration when she heard that. “That is so obvious it doesn’t make sense to even consider other ways of doing it! Why do we have to go through so much math just to end up constructing the most obvious thing ever??” wailed Twilight. “That is just a cultural bias. A few thousand years ago, when mathematics was getting started, nobody had even heard of orthonormal bases!” Twilight groaned and let her head droop to the floor. “Now now,” Luna said sweetly, “What was the lesson you just told me you had learned a few minutes ago?” Twilight sighed. “I will learn the tools because they will make things clearer in the end.” “That is right! Do not forget so quickly, Twilight Sparkle. Proposition 3.4 Now I want you to prove that an orthonormal set of vectors is linearly independent. This should be an easy one!” Twilight thought for a moment and tried to find the perfect way to state the proof. “If two vectors, and , are orthogonal, then Since the inner product is linear, it distributes over addition. The terms and are zero because the vectors are orthogonal. That leaves . Because the inner product is positive definite, the only way that this expression could be zero is if both and are zero. In other words, two orthogonal vectors are linearly independent. If any two orthogonal vectors are linearly independent, it follows that an orthogonal set of vectors must be linearly independent.” “Quite correct, Twilight Sparkle. Proposition 3.5 “The last thing we shall prove this pleasant but somewhat garish morning is that an orthonormal basis always exists. To do that we shall have to think a little bit about projection operators. By a projection of onto , I mean that one finds the component of in the direction of . I will say that is a unit vector. This will make the problem easier. What do you think the formula for that is?” “From your diagram, it seems as if ought to be the length of the result we want, and is in the right direction. So I think the answer is .” Definition 3.9 : Projection “That is right, and that is how a projection shall be defined. Now, is the projection a linear operation on ?” “It is. It is easy to see that it would distribute over addition and commute with scalars.” “Let us call the operator . Can you write a formula for this operator acting on ?” “I think I can using the abstract index notation. (3.17) That’s an inner product-a scalar-multiplied by a vector. The two operations together work out like a linear map.” Definition 3.10 : Outer product “Yes. Now let me define another kind of product. The outer product is a way of multiplying two vectors in an inner-product space to form a linear map. Here is the definition. I put the g matrix in there because now, when you multiply by a vector, multiplication by w works like a proper inner product. Otherwise the result would not be coordinate-independent. And now we can write as (3.18) Make sense?” Twilight nodded. “Yes, but if you can multiply two vectors to produce a linear map, shouldn’t you we able to multiply more of them together to produce a new kind of object?” “Very good question! You can indeed do that. However, we shall save the more general theory for another day. My next question is, suppose you have a unit vector and you wish to project a vector so that it is perpendicular to rather than parallel? Show me the operator for that.” “Well I want an operator which gives if and are perpendicular and gives 0 if they’re parallel. That should be enough to define the operator.” She then wrote it out on the stone floor. “That should be enough to define how the operator works,” she said. Now if I just wrote something like (3.19) Then this has the desired properties. It just subtracts the parallel part from .” “Indeed. Suppose that you had several unit vectors , , and so on which are all orthogonal, and you wish to project a vector so that it is perpendicular to all of them?” “That’s not a difficult generalization at all. That would just be (3.20) “Can you use that in the proof that every inner product space has an orthonormal basis?” “That seems quite feasible now. Given a basis , I just define (3.21) and so on. The vectors will form an orthonormal basis. And of course I’ve written normalize to indicate that the vector must be normalized by dividing it by its own norm.” “Exactly! You have discovered what is called the Graham-Schmidt process. There are some caveats to this proof. Because the proof is recursive, it only proves that a vector space with a countable basis has an orthonormal basis. However, that is good enough for our purposes.” “So,” said twilight, “we have finally reached the point where we can do ninth-grade geometry.” “That is right!” said Luna, not appearing to have caught Twilight’s sarcasm, “and to think all those poor young ponies think they’re doing math every day when they really don’t know what they’re doing at all!” Twilight sighed to herself. “Well the books I tried to read did say a lot about operators, and scalars, and matrices. So I suppose I must be learning something.” “Of course you are.” Luna said with a flick of her mane. “But now I must now be off. Farewell, Twilight Sparkle.” Luna then bolted aloft. “Wait!” yelled Twilight. She at least wanted to thank Luna for being such a dedicated teacher and putting up with her complaints. But it was too late. Luna was merely a speck in the sky and she soon disappeared behind the mountain. On her way back home, Twilight mused. Luna could be a frustrating pony, but Twilight had enjoyed the morning in spite of herself. She had hoped to ask Luna more about how the material they had studied today actually related to physics, but Luna had left so abruptly that she hadn’t had the chance. It seemed like Luna was trying to be friendly with her, but was not comfortable speaking about topics other than math. She wondered if she might be able to do more to bring Luna out of her shell next time.