if the joint PMF is p(x,y)=k(x²+y²)
find the PMF 3X-2Y
is this given by substituting x=3X
y=-2Y
in which case it becomes:
p(3x,-2y)=k(9x²+4y²)
i am not sure if this is correct:
Your answer is not correct. You can see that is wrong by trying to use it to answer the question "what is the probability that 3x-2y = 5". There is no way to input this information into your function to get an answer.
Starting over, define a single variable Z = 3X - 2Y
You want the PMF of Z
$\displaystyle P(Z=z) = P(3X - 2Y = z) = P(Y=\frac{z+3X}{2})$
ie, to get a particular value of Z we cna have any value of X, provided that Y = (z+3x}/2
$\displaystyle P(Z=z) = \sum_x k \left[ x^2 + \left( \frac{z+3x}{2} \right)^2 \right]$
The last step is called the method of convolutions, which hopefully you have seen before if you were set this problem.