# Thread: Least square approximation of a function and being in subspace W

1. ## Least square approximation of a function and being in subspace W

Least Squares Approximation Theorem:

Let $\displaystyle f$ be continuous on $\displaystyle [a,b]$, and let $\displaystyle W$ be a finite-dimensional subspace of $\displaystyle C[a,b]$. The least square approximating function of $\displaystyle f$ with respect to $\displaystyle W$ is given by

$\displaystyle g = \langle f,\mathbf{w_1} \rangle \mathbf{w_1} + \langle f,\mathbf{w_2} \rangle \mathbf{w_2} + ... + \langle f,\mathbf{w_n} \rangle \mathbf{w_n},$

$\displaystyle \text{where } B = \{\mathbf{w_1}, \mathbf{w_2}, ..., \mathbf{w_n}\} \text{ is an orthonormal basis for } W.$

Proof:
To show that $\displaystyle g$ is the least squares approximating function of $\displaystyle f$, prove that the inequality

$\displaystyle \parallel f-g \parallel \,\, \leq \,\, \parallel f-\mathbf{w} \parallel$
is true for any vector $\displaystyle \mathbf{w}$ in $\displaystyle W$. By writing $\displaystyle f-g$ as

$\displaystyle f - g = f - \langle f,\mathbf{w_1} \rangle \mathbf{w_1} + \langle f,\mathbf{w_2} \rangle \mathbf{w_2} + ... + \langle f,\mathbf{w_n} \rangle \mathbf{w_n}$

you can see that $\displaystyle f-g$ is orthogonal to each $\displaystyle \mathbf{w_i}$, which in turn implies that it is orthogonal to each vector in $\displaystyle W$. In particular, $\displaystyle f-g$ is orthogonal to $\displaystyle g-\mathbf{w}$

$\displaystyle \text{...the rest of the proof continues}$

-------------Proof Ends------------

My question has two parts.

First question:

Why proving

$\displaystyle \parallel f-g \parallel \,\, \leq \,\, \parallel f-\mathbf{w} \parallel$

is enough for this proof? There might be other vectors that make the least square approximation smaller. Why choose particular orthonormal basis?

Second question:

What is the reason behind the implicit statement $\displaystyle g-\mathbf{w}$ is in the subspace $\displaystyle W$ ?

What reasoning make $\displaystyle g-\mathbf{w} \in W$? Is there any theorem I can't remember now for this question?

2. ## Re: Least square approximation of a function and being in subspace W

1. Because w is left arbitrary. It's true for ALL vectors w in the subspace W.

2. Because g is in W, and w is in W. Since W is a subspace, linear combinations of vectors in the subspace remain in the subspace.

3. ## Re: Least square approximation of a function and being in subspace W

Originally Posted by Ackbeet
1. Because w is left arbitrary. It's true for ALL vectors w in the subspace W.

2. Because g is in W, and w is in W. Since W is a subspace, linear combinations of vectors in the subspace remain in the subspace.
Thank you for your answers. I get the answer no. 2. But having difficulty in answer no. 1.

I'll tell you my understanding.

$\displaystyle \mathbf{w_i}$ is not a generalized vector. $\displaystyle \mathbf{w_i}$ is a orthonormal basis that are normalized and orthogonal to each other. Most vectors in subspace $\displaystyle W$ are not normalized and orthogonal. So why a particular class of vector speaks for all generalized vectors in subspace $\displaystyle W$? I don't understand that.

Can you kindly tell me what you meant by "left arbitrary"?

4. ## Re: Least square approximation of a function and being in subspace W

Hmm. Well, I could be wrong, but it looks to me like you might be confusing the arbitrary $\displaystyle \mathbf{w}$ vector chosen at the beginning of the proof, and the basis vectors $\displaystyle \mathbf{w}_{i}$ of the subspace $\displaystyle W.$ They are not the same thing. It might have been clearer in the writing of the proof to pick an arbitrary vector $\displaystyle \mathbf{v}\in W,$ instead of $\displaystyle \mathbf{w}\in W.$ Does that clear things up a bit?

5. ## Re: Least square approximation of a function and being in subspace W

Originally Posted by Ackbeet
Hmm. Well, I could be wrong, but it looks to me like you might be confusing the arbitrary $\displaystyle \mathbf{w}$ vector chosen at the beginning of the proof, and the basis vectors $\displaystyle \mathbf{w}_{i}$ of the subspace $\displaystyle W.$ They are not the same thing. It might have been clearer in the writing of the proof to pick an arbitrary vector $\displaystyle \mathbf{v}\in W,$ instead of $\displaystyle \mathbf{w}\in W.$ Does that clear things up a bit?
Sorry Ackbeet. I'm extremely sorry. You're right. I mixed up $\displaystyle \mathbf{w}$ with $\displaystyle \mathbf{w_i}$. Now it's clear.

In the proof it's clearly written "is true for any vector $\displaystyle \mathbf{w}$". I wish I were more attentive. Sorry. Again thanks.

6. ## Re: Least square approximation of a function and being in subspace W

No apologies needed, man. The questions you asked are what this forum is for.

Have a good one!