Math Help - Orthogonal Projection and Minimization

1. Orthogonal Projection and Minimization

This is a relatively quick question.

Suppose we have a vector space $V$ with $v\in{V}$ and subspace U with $u\in{U}$ and we want to find some vector u such that $||v-u||$ is at small as possible which I know implies that u is a good approximation of v.

It is shown that
$||v-P_U(v)||\leq||v-u||$

where $P_U(v)$ is the orthogonal projection for $V=U\oplus{U^{\perp}$
and I understand how to prove this inequality. Supposedly finding $P_U(v)$ gives me a good approximation of v (which evidently it actually does), but I don't quite understand why.

I have some speculations on why however. I believe the inequality implies that unless $P_U(v)=u$, then $P_U(v)$ will always be a better approximation than u.

However, I don't really have a good understanding on why the orthogonal projection is truly a good approximation of v.

Thank you.

2. Think of a simple example: $V$ is the xy plane, and $U$ is the subspace consisting of all points on the line $y=x.$ You can show that it is a subspace, because it contains the origin, and is closed under scalar multiplication and vector addition.

Now then. If your vector $v\in U,$ then obviously the best approximation is $v$ itself, because then $0\le\|v-u\|=0.$ If, on the other hand, $v\not\in U,$ then what is the vector in $U$ closest to $v?$ Well, it would be the point on the line that is closest to $v.$ And that point is the point you would get by dropping the perpendicular from $v$ to the line $y=x.$ That also turns out to be the orthogonal projection onto the subspace $U.$

The essential idea here is that the straight line (in this case, the perpendicular), is the shortest path between two points, at least in the Euclidean norm. And that "perpendicular" is equivalent to the orthogonal projection onto the subspace.

Make sense? Does this help?

3. Originally Posted by Ackbeet
Think of a simple example: $V$ is the xy plane, and $U$ is the subspace consisting of all points on the line $y=x.$ You can show that it is a subspace, because it contains the origin, and is closed under scalar multiplication and vector addition.

Now then. If your vector $v\in U,$ then obviously the best approximation is $v$ itself, because then $0\le\|v-u\|=0.$ If, on the other hand, $v\not\in U,$ then what is the vector in $U$ closest to $v?$ Well, it would be the point on the line that is closest to $v.$ And that point is the point you would get by dropping the perpendicular from $v$ to the line $y=x.$ That also turns out to be the orthogonal projection onto the subspace $U.$

The essential idea here is that the straight line (in this case, the perpendicular), is the shortest path between two points, at least in the Euclidean norm. And that "perpendicular" is equivalent to the orthogonal projection onto the subspace.

Make sense? Does this help?
Thanks for the reply. I see how the projection gives a good approximation in your example with the point and line. I was also wondering how the inequality with the norms show that the orthogonal projection is a good approximation. I understand the intuition behind it, but what does the inequality show?

Thank you

4. Are you talking about in the case when $v\in U,$ and I got the expression

$0\le\|u-v\|=0?$

In that case, the inequality shows that this is the best approximation possible, because the expression $\|u-v\|$ is a measure of how good the approximation is, and it is always non-negative, by the axioms of a norm.

Make sense?

5. Originally Posted by Ackbeet
Are you talking about in the case when $v\in U,$ and I got the expression

$0\le\|u-v\|=0?$

In that case, the inequality shows that this is the best approximation possible, because the expression $\|u-v\|$ is a measure of how good the approximation is, and it is always non-negative, by the axioms of a norm.

Make sense?
.

Sorry, I meant this inequality:
$||v-P_U(v)||\leq||v-u||$

The expression $0\le\|u-v\|=0$ does make sense though.

6. Ah. That inequality shows that the orthogonal projection is the best approximation, because the distance between the vector $v$ and the vector $P_{U}(v)$ is smaller than the distance $\|v-u\|$ for any other vector $u\in U.$ (Note that $P_{U}(v)\in U.$)

A normed vector space is also a metric space, right? You have the induced metric

$d(u,v)=\|u-v\|,$

which obeys all the requirements for a metric. So the expression $\|u-v\|$ is the distance between the two vectors $u$ and $v.$

Does that make sense?

7. Thanks!

Yes, this makes perfect sense now. Thank you.

8. You're welcome! Have a good one.