Personally, I would have chosen e^(2x) as the function to integrate, but let's do it your way.

int{e^(2x)cos(3x)}dx

= (1/3)sin(3x)e^(2x) - int{2(1/3)sin(3x)e^(2x)}dx

= (1/3)sin(3x)e^(2x) - (2/3)int{sin(3x)e^(2x)}dx .........do this last integral by parts again

= (1/3)sin(3x)e^(2x) - (2/3)[(-1/3)cos(3x)e^(2x) - int{2(-1/3)cos(3x)e^(2x)}dx]

= (1/3)sin(3x)e^(2x) - (2/3)[(-1/3)cos(3x)e^(2x) + (2/3)int{cos(3x)e^(2x)}dx]

= (1/3)sin(3x)e^(2x) - [(-2/9)cos(3x)e^(2x) + (4/9)int{cos(3x)e^(2x)}dx]

= (1/3)sin(3x)e^(2x) + (2/9)cos(3x)e^(2x) - (4/9)int{cos(3x)e^(2x)}dx

So now we have:

int{e^(2x)cos(3x)}dx = (1/3)sin(3x)e^(2x) + (2/9)cos(3x)e^(2x) - (4/9)int{cos(3x)e^(2x)}dx

So we solve this equation for int{e^(2x)cos(3x)}dx

=> int{e^(2x)cos(3x)}dx + (4/9)int{cos(3x)e^(2x)}dx = (1/3)sin(3x)e^(2x) + (2/9)cos(3x)e^(2x)

=> (13/9)int{e^(2x)cos(3x)}dx = (1/3)sin(3x)e^(2x) + (2/9)cos(3x)e^(2x)

=> int{e^(2x)cos(3x)}dx = (9/13)[(1/3)sin(3x)e^(2x) + (2/9)cos(3x)e^(2x)]

=> int{e^(2x)cos(3x)}dx = (3/13)sin(3x)e^(2x) + (2/13)cos(3x)e^(2x) + C

Also, see the last note of this post: http://www.mathhelpforum.com/math-he...l-problem.html