Okay, so the length and width of the paper are 3, and you're subtracting 2x from them. This you seem to have figured out. Then we fold the flaps up to create the sides of the box. Since the flaps are x units long, the box will be x units high. so the formula for volume will be

Look at the pictures I drew you, do you see how the dotted lines are where the flaps fold up? And you can see that they are all x units long.

Another way of thinking about this might be that the left flap takes up x units, the base takes up 3-2x units, and the right flap takes up x units. So when you fold the flaps up, the base will be left at 3-2x, and the left height will be x, and the right height will be x. Anyway, look at the problem I drew again, and maybe try cutting some squares out of a piece of paper to see it.