Do you see that h is discontinuous at rational numbers?
For irrational x, consider (without loss of generality) a rational ε = 1 / n. What can you say about the cardinality of the set {x' ∈ (x - 1, x + 1) | h(x') > 1 / n}?
Given ε, our task is to find a δ such that h(x) < ε for all x ∈ (x - δ, x + δ). I suggested looking at x's that violate this property. Fortunately, there is only a finite number of them within distance 1 from x. If this is so, can you find (at least indirectly) an appropriate δ?
(Strictly speaking, if we are looking x's violating h(x) < ε, the definition of the set in post #2 should have h(x') ≥ 1 / n, not h(x') > 1 / n.)
The function h is called Thomae's function.
intuitively, the idea is this:
if ε > 1, any δ will do, since |h(x) - h(x_{0})| = |h(x)| ≤ 1, at any irrational x_{0}
but of course, we need to be able to choose ANY ε > 0.
so what about 1 > ε > 1/2? well, there's just 3 points we need to avoid: 0,1/2 and 1. so in this case, we can take:
δ = min{1-x_{0},(1/2) - x_{0},x_{0}}
how about 1/2 > ε > 1/3? now we need to avoid 2 more points: 1/3 and 2/3. so again let δ be the minimum of the distances of x_{0} from the 5 points we want to stay away from.
ok, how about 1/3 > ε > 1/4? just two more points to add to our finite list: 1/4 and 3/4 (now we have 7).
indeed, at every stage, say the k-th, we at MOST add k-1 points we need to avoid (it WILL be k-1 if k is prime, and less than k-1 if k is composite, because then we've already counted some).
even if we add up all these finite numbers, we still get another finite number.
now for ANY ε > 0, there is SOME positive integer N with ε < 1/N (N might be very large, of course). and since we know that all we have to do is find the minimim distance of x_{0} from some finite (but possibly large) set of points, we can (in theory, at least) find a δ that works. and that's all we need for continuity, is to show the EXISTENCE of such a δ. of course, explicitly describing such a δ is preferable, but not necessary.
(it is important that the set of points we have to "avoid" is finite. otherwise, we might wind up with an inf (greatest lower bound), instead of a true minimum, and it could turn out that such an inf is 0, which would not be good at all).
what happens is, the sheer number of "large denominator rationals" tend to cluster around the irrationals (so they have enough rationals nearby to keep the function h(x) nearby), but at each rational point, if you zoom in close enough, it's all by itself (that is, for some small ε, namely ε < 1/q, we can't find a δ that will make h continuous at p/q).