Thanks Ackbeet, I'll try to show you my calculations. It's been a while since I've done any maths (and then only in Swedish), so please forgive me it's not represented the way you're used to. For the same reason, it's entirely possible I messed up some of the calculations - thanks for checking them for me!
The hexagon is 8830 units from side to side, 10050 from point to point. (According to the tent manufacturer - this is a tent seen from above. The measurements could be wrong for all I know.)
By using the following (from Wikipedia) I calculated the length of the sides of the gray triangles in the picture:
In a right triangle with acute angles measuring 30 and 60 degrees, the hypotenuse is twice the length of the shorter side, and the longer side is equal to the length of the shorter side times √3.The long side of the gray triangle is half of the side-to-side measurement: 8830/2 = 4415.
To get the short side, I used 4415 / √3 = 2549.
The hypotenuse is twice the short side, 5098.
Since every coordinate is exactly in the middle of a side, I make a smaller triangle with the hypotenuse 2549.
Using the same formulas, I get a new short side of 1274 (2549/2) and a new long side of 2207 (1274 * √3).
The long side gives the x value, 2207.
The length from point to point, halved, minus the short side gives the negative y value: 10050/2 - 1274 = 3751
I then made the same calculations with the hexagon with a side up, as it would be when rotated 30 degrees clockwise to get the target coordinates.
I just realized that I've been using the computer's coordinate system, where (0, 0) is in the top left corner (hence negative y upwards and positive y downwards). It's generally the other way around in maths, isn't it? Could this be the source of my problems with the formula you posted? I've tried but I don't seem to get the correct results. For example, I get -3786 as the x value for the next door counter-clockwise from the blue door when I rotate the hexagon 30 degrees clockwise, when I'd expect it to be 0.
If I invert the y value before making the calculation, and then invert it back, I'm still off by 35. Any suggestions?