Really not grasping something here.

Say I have a square with the vertex(0,0) at one corner and I can resize the shape by dragging the corner opposite, in and out. (achieved by increasing or decreasing the width and height). When resizing the shape, the vertex at (0,0) does not move, but all other vertices do.

When I rotate this square, and then resize the shape it doesnt resize by keeping the vertex in the opposite corner fixed, but instead the vertex at (0,0) stays fixed as before. This means the square doesn't shrink to the correct corner when you drag it inwards, but always shrinks to the vertex at (0,0).

I can't figure out how to change this.

just so I dont say something silly and not related to your problem,

are you calculation the corners mathematically and plotting them,
are you using NSBezierPath with NSAffineTranform,
are you using OpenGL,
or something else?