This problem has no continous properties, thus you have to calculate the expected profit for every possible number of magazins ordered. The shop owner will choose the option that yields the highest profits. The expected payoff is the expected profit (which is bounded in every case by the number of magazins ordered. Thus e.g. the exp. profit for 2 ordered magazines will be: (0.1 * 1 + 0.2 * 2 + 0.4 * 2 + 0.3 * 2) * 2 - 1* 2. Btw: For a more general case the notation would be: (0.1 * 1 + 0.2 * min(ordered,2) + 0.4 * min(ordered,3) + 0.3 * min(ordered,4) * 2 - 1 * ordered. The shop owner will choose this number of magazines that yields the highest profit, this variable is fixed since the shop owner only knows the distribution of demand.

You cannot however introduce a variabiable Y_k like you claimed since this is not a random event anymore but something you have to find a decision for.