Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

Hi, I wonder if anyone can give me a handle on this problem. I have sampled 100,000 point from an isometric gaussian in 98 dimensional space (each point translates into a particular face in a "face space" model). I have then defined a uniform distribution over the 100, 000 points. so far so good.

I have then used a kind of decision procedure (making sibjective judgements about which face is most similar to a target face) to update the distribution, such than some points have more mass than others.

What I want to do now is to re-sample from another gaussian, based on this updated distribution. It seems to me that the mean of the new gaussian should be the expectation of the points (i.e. the sum of all the points weighted by their respective probabilities).

Does that seem right to people?

More problematically, for me at least, how can I use the probabilities to estimate a new covariance matrix?

Many thanks for any insights. MD

Re: Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

Hey Mathsdog.

Are you familiar with bayesian statistics, posterior distributions and conjugate priors?

Re: Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

Hi Chiro, yes I am to some extent (and trying to learn more and more). Indeed, that would be how I would like to treat this problem ideally. The Gaussian is self conjugate, so that makes sense here right? But I cannot see exactly how I can obtain a posterior, using the (now-non-uniform) discrete distribution over my points (sampled from my prior, right). You usually see to the heart of the matter. What would you suggest?

Thanks, MD

Re: Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

In the bayesian setting you can keep updating your likelihood and posterior with estimates from the data.

You don't necessarily need a conjugate prior, but analytically it makes things a lot easier. If you use a general prior then you may need to use a numerical integration scheme depending on what kind of functional form you end up getting.

Basically you are estimating parameters so if your likelihood is a multivariate normal, then you are estimating the mean vector and covariance matrix (unless you are assuming them to be known and constant).

Remember that all the values of the mean and covariance are continuous (variances are always > 0 though but not necessarily covariances) so your prior should be continuous (unless for some reason the means and/or covariances are not continuous).

In any case you select a prior, use your likelihood (multivariate Guassian) and calculate your posterior. Once you do that you re-use the results from your posterior to generate a new prior and you keep repeating the process.

If however you have discrete distributions and you are mixing them with non-discrete distributions, then I would recommend that you either take the discrete distribution and turn into a continuous "step-wise" distribution or take the continuous distribution and integrate over the various bins and that will become your discrete distribution.

Re: Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

Hi Chiro,

I'm not sure if I am being clear. I am trying to locate a region in high dimensional space. This is within a "face space", a high dimensional representation that can approximate pretty much any face. My strategy is as follows:

1. Sample from a prior with a mean at the origin and a diagonal covariance with large values on the diagonal.

2. I define a discrete distribution over these sampled points. This is as much as to say: all of these randomly sampled points in high dimensional space are equally likely to be near the target region. I could compute the density of each point from the gaussian from which i sampled and then renormalise to obtain a valid distribution over the sampled points, and I may do that. But for now I just use a uniform.

So now I have two things. First, I have a matrix M of m rows, each row a point, and n columns, each column corresponding to a dimension in continuous space. Secondly, I have a vector V of m probabilities. Each element is a probability for a row in the matrix M.

3. Now I use subjective judgements to update the probabilities in V. I show the subject two faces, corresponding to two rows in M. If he/she thinks face A resembles the target more than face B I update the probabilities in V to reflect this.

4. Having repeated step 3 several times I now have the same matrix M as I originally had, but an updated vector V of probabilities.

5. I would like to now re-sample from a new gaussian, to obtain a new matrix M', using the probabilities in V and the points in M to guide my choice.

And this is where I am stuck. How can I combine M and V to obtain M' in a probabilistically principled way?

I hope that is a bit clearer than before.

Thanks again for your attention. MD

Re: Fitting a Gaussian to high dimensional data with a discrete pdf defined over them

This sounds a bit like a self-organizing map (SOM). Are you familiar with these? If not I recommend you read up on them because it sounds exactly like the problem you are describing.

Self-organizing maps update probabilities and structures in a similar way based on what is realized in the data.