Thread: Least square approximation of a function and being in subspace W

1. Least square approximation of a function and being in subspace W

Least Squares Approximation Theorem:

Let $f$ be continuous on $[a,b]$, and let $W$ be a finite-dimensional subspace of $C[a,b]$. The least square approximating function of $f$ with respect to $W$ is given by

$g = \langle f,\mathbf{w_1} \rangle \mathbf{w_1} + \langle f,\mathbf{w_2} \rangle \mathbf{w_2} + ... + \langle f,\mathbf{w_n} \rangle \mathbf{w_n},$

$\text{where } B = \{\mathbf{w_1}, \mathbf{w_2}, ..., \mathbf{w_n}\} \text{ is an orthonormal basis for } W.$

Proof:
To show that $g$ is the least squares approximating function of $f$, prove that the inequality

$\parallel f-g \parallel \,\, \leq \,\, \parallel f-\mathbf{w} \parallel$
is true for any vector $\mathbf{w}$ in $W$. By writing $f-g$ as

$f - g = f - \langle f,\mathbf{w_1} \rangle \mathbf{w_1} + \langle f,\mathbf{w_2} \rangle \mathbf{w_2} + ... + \langle f,\mathbf{w_n} \rangle \mathbf{w_n}$

you can see that $f-g$ is orthogonal to each $\mathbf{w_i}$, which in turn implies that it is orthogonal to each vector in $W$. In particular, $f-g$ is orthogonal to $g-\mathbf{w}$

$\text{...the rest of the proof continues}$

-------------Proof Ends------------

My question has two parts.

First question:

Why proving

$\parallel f-g \parallel \,\, \leq \,\, \parallel f-\mathbf{w} \parallel$

is enough for this proof? There might be other vectors that make the least square approximation smaller. Why choose particular orthonormal basis?

Second question:

What is the reason behind the implicit statement $g-\mathbf{w}$ is in the subspace $W$ ?

What reasoning make $g-\mathbf{w} \in W$? Is there any theorem I can't remember now for this question?

2. Re: Least square approximation of a function and being in subspace W

1. Because w is left arbitrary. It's true for ALL vectors w in the subspace W.

2. Because g is in W, and w is in W. Since W is a subspace, linear combinations of vectors in the subspace remain in the subspace.

3. Re: Least square approximation of a function and being in subspace W

Originally Posted by Ackbeet
1. Because w is left arbitrary. It's true for ALL vectors w in the subspace W.

2. Because g is in W, and w is in W. Since W is a subspace, linear combinations of vectors in the subspace remain in the subspace.

I'll tell you my understanding.

$\mathbf{w_i}$ is not a generalized vector. $\mathbf{w_i}$ is a orthonormal basis that are normalized and orthogonal to each other. Most vectors in subspace $W$ are not normalized and orthogonal. So why a particular class of vector speaks for all generalized vectors in subspace $W$? I don't understand that.

Can you kindly tell me what you meant by "left arbitrary"?

4. Re: Least square approximation of a function and being in subspace W

Hmm. Well, I could be wrong, but it looks to me like you might be confusing the arbitrary $\mathbf{w}$ vector chosen at the beginning of the proof, and the basis vectors $\mathbf{w}_{i}$ of the subspace $W.$ They are not the same thing. It might have been clearer in the writing of the proof to pick an arbitrary vector $\mathbf{v}\in W,$ instead of $\mathbf{w}\in W.$ Does that clear things up a bit?

5. Re: Least square approximation of a function and being in subspace W

Originally Posted by Ackbeet
Hmm. Well, I could be wrong, but it looks to me like you might be confusing the arbitrary $\mathbf{w}$ vector chosen at the beginning of the proof, and the basis vectors $\mathbf{w}_{i}$ of the subspace $W.$ They are not the same thing. It might have been clearer in the writing of the proof to pick an arbitrary vector $\mathbf{v}\in W,$ instead of $\mathbf{w}\in W.$ Does that clear things up a bit?
Sorry Ackbeet. I'm extremely sorry. You're right. I mixed up $\mathbf{w}$ with $\mathbf{w_i}$. Now it's clear.

In the proof it's clearly written "is true for any vector $\mathbf{w}$". I wish I were more attentive. Sorry. Again thanks.

6. Re: Least square approximation of a function and being in subspace W

No apologies needed, man. The questions you asked are what this forum is for.

Have a good one!