Neither do I and possibly nobody else if you don't tell us what's in your mind. Of course, if there's a valid reason to deduce that the limit matrix is in $\displaystyle SO(2)$ then we're almost done since all such matrices have the form of the already known limit matrix...
Tonio
Well, now that I'm less tired I'm not even sure if what I was thinking makes sense, but I've at least answered my question.
So, let me get as far as I can and see if either A) it works or B) I run into problems and give up.
Let $\displaystyle M=\displaystyle \begin{bmatrix}1 & \frac{x}{n} \\ \frac{-x}{n} & 1\end{bmatrix}$ and let $\displaystyle A=\lim_{n\to\infty}M^n$.
Since, $\displaystyle \text{Trans}:\text{GL}(2,\mathbb{R})\to\text{GL}(2 ,\mathbb{R}):B\mapsto B^T$ is continuous we may say that $\displaystyle A^T=\lim_{n\to\infty}\left(M^n\right)^T$ and so $\displaystyle AA^T=\lim_{n\to\infty}M^n\cdot\lim_{n\to\infty}\le ft(M^n\right)^T=\lim_{n\to\infty}M^n\left(M^T\righ t)^n$. But, it's pretty easy to show that $\displaystyle MM^T=I$ and so $\displaystyle M^n\left(M^T\right)^n=M\cdots M\cdot M^T\cdots M^T=I$ and so $\displaystyle AA^T=\lim_{n\to\infty}I=I$ and so $\displaystyle A\in\text{O}(2)$ and also since $\displaystyle \det:\text{GL}(2,\mathbb{R})\to\mathbb{R}$ is continuous we see that $\displaystyle \det A=\det\left(\lim_{n\to\infty}M^n\right)=\lim_{n\to \infty}\det M^n=\lim_{n\to\infty}\left(\det M\right)^n$, but it is fairly easy to see that $\displaystyle \det M=\frac{x^2}{n^2}+1$ and so $\displaystyle \det A=\lim_{n\to\infty}\left(\frac{x^2}{n^2}+1\right)^ n=1$. Thus, $\displaystyle A\in\text{SO}(2)$.
Ta-da?
Oops! I knew I would make a stupid error!
$\displaystyle MM^T=\left(1+\frac{x^2}{n^2}\right)I$ and so
$\displaystyle M^n\left(M^T\right)^n=M\cdots MM^T\cdots M^T=\left(1+\frac{x^2}{n^2}\right)^nI^n=\left(1+\f rac{x^2}{n^2}\right)^nI\to I$ and the result still holds true that $\displaystyle A$ is orthogonal!
I made no claim that it was a proof. I merely gave incentive to believe what the answer should be.
That said, I don't think (I haven't tried) it would be very hard to show that it converges.
EDIT: Embarrassing as it is to say, it isn't even obvious to me what "converge" would mean here. What metric are we talking about with $\displaystyle \text{GL}\left(n,\mathbb{R}\right)$ or I guess more properly $\displaystyle \mathfrak{M}^{2\times 2}(\mathbb{R})$? The only thing that immediately comes to mind is the one induced by $\displaystyle \|\cdot\|_{\text{op}}$
for a sequence of $\displaystyle m \times m$ matrices $\displaystyle M_n=[x_{ij}^{(n)}]$ we say that $\displaystyle \lim_{n\to\infty}M_n=M=[a_{ij}]$ if $\displaystyle \lim_{n\to\infty}x_{ij}^{(n)}=a_{ij},$ for all $\displaystyle 1 \leq i,j \leq m.$ in our problem i really don't see any way to prove that the limit exists
without actually finding the limit! haha
I'd like to point out that the limit can be written $\displaystyle \lim_{n\rightarrow \infty}\left(I+\frac{A}{n}\right)^n$, which is just $\displaystyle e^A$, which converges for any square matrix $\displaystyle A$.
What's more, in this case the matrix $\displaystyle A$ is skew-symmetric, and it follows from the theory of Lie groups that $\displaystyle e^A$ is orthogonal.
No; the Lie algebra of the orthogonal group is the vector space of skew-symmetric matrices, and the exponential map takes an element of the Lie algebra to its corresponding Lie group.
For example, the Lie algebra of the unit circle is the imaginary line (it's the tangent space to the identity), and the exponential takes the imaginary line to the unit circle.
Interesting thread, four proofs so far. Here's mine (which is essentially Tonio's but without much use of linear algebra):
Simply note that the matrix defines a $\displaystyle \mathbb{C}$-linear transformation $\displaystyle A_n: \mathbb{C} \rightarrow \mathbb{C}$, namely multiplication by $\displaystyle \left( 1+i\frac{x}{n} \right) ^n$ and this converges to $\displaystyle e^{ix}$ trivially.