Is there a degree of abstraction beyond which mathematicians can not cross? Some point where we're limited to working within some set of rules and can't go any further for any branch of mathematics?
Methinks you are making a hidden assumption here: the assumption that something "more abstract" is something more powerful, and more useful. Even admitting that abstraction can sometimes be extremely powerful and useful, it is, as I understand the concept, not at all the case that the "highest level of abstraction" is particularly powerful or useful. In fact, it leads to a perfectly uninteresting theory: a theory about objects that are all the same (i.e. undistinguishable), because you have thrown out the making of any distinctions between them.
Maybe it would be a useful exercise for you to inquire a little more deeply into how some of the currently more abstract theories, like the theory of groups, rings, and fields, vector spaces, set theoretical topology, and category theory and the like have come about. This would give you an idea of how the process of finding higher, but still useful (not sterile) abstractions over time in practice works.
For example, over time, people notice that a great many theorems of several existing theories could be proven for all those theories at once, if one happened to introduce a suitably chosen common abstraction of all of them. So that new abstraction would have to catch enough details to still allow substantial proofs to be formulated.
There are books about these developments, I just haven't got any references handy. So this would be under the rubric: history of mathematics (but not beginning with Babylonian mathematics, perhaps).
In connection with this, Simon Singh's Fermat's Enigma goes into how it was proven that modular forms are equivalent to elliptical equations which was an astonishing result (proving Fermat's Last Theorem incidentally) and appears to be the closest part of Langland's program to unify mathematics to date.
I am in no way able to judge "Langland's program to unify mathematics". Maybe you could explain to ignoramuses like myself what "unifying mathematics" would amount to in this case?
The reason I ask is this: the introduction of a higher level abstraction (like for example topological spaces) does not at all necessarily make the lower level abstractions it subsumes (like metric spaces) completely superfluous. Also, we have groups, rings, and fields, but we still want to be able to talk about natural or real numbers, and their particular properties, do we not?
Nor did I till I googled it just now.
Higher levels of abstraction are developed all the time. As far as I know, the most abstract level we've reached so far is the discipline of Category Theory. Before that it was Abstract Algebra and Topology. Mathematical Logic gets pretty rarefied.
I admit to having little patience with those who insist on keeping an eye on practicalities. Go as far out into the wilds of pure thought as you like, wonderboy.
I regard myself as a math explorer and I do like to experiment. My opinion is if you stick very close to practicalities, you may miss out on making discoveries.
We have many people who like the practical and many more who like the abstract/theoretical in math and I applaud them both.
I hope the following isn't too technical (linking up number theory with the representation theory of certain groups):
Langlands program - Wikipedia, the free encyclopedia