You could look at the distance between the ponts and the elipse, every point that is further away than a certain threshold (maybe a certain amount of the elipse's circumference) is excluded, then you try to fit the elipse again. While the set of "false" points has changed from the previous time the elipse was fit to the points (which will be the empty set if the elipse hasn't been fit before), repeat and fit the elipse again.