since n(n-1) / 2 is a perfect square, so n(n-1) > 2
solving that we get (n-2)(n+1) > 0
so n < -1 or n > 2
The problem is to find all n for which n choose 2 is a perfect square. In other words, when m is an integer.
I'm not really at all sure how to approach this, any help is appreciated.
Also, here are the 10 smallest solutions
n = 2 m = 1
n = 9 m = 6
n = 50 m = 35
n = 289 m = 204
n = 1682 m = 1189
n = 9801 m = 6930
n = 57122 m = 40391
n = 332929 m = 235416
n = 1940450 m = 1372105
n = 11309769 m = 7997214
This is a quadratic Diophantine equation in two variables. In general the theory for these can be rather involved, and I haven't learned all of it. Dario Alpern has made a very useful online solver that can show steps, although reading the steps is usually not enough to understand the process. He also wrote an explanation of his methods here. Your equation can be re-expressed as: (putting x=n and y=m)
The solver gives these solutions:
Using (0,0) as a seed will generate negative solutions (which you may discard), and using (1,0) will generate positive ones.
Also, the sequence of n-values is on OEIS as A055997. Among other things, the page gives a recursive formula similar to a Lucas sequence.
It's an interesting topic, and if I were more familiar with it I would be able to give you a sketch of a solution method, but if you want to learn more this should give you plenty to work with.