topsquark thanks you for adding units!

Personally I hate this problem. It always comes up and has a lot of fiddly little details. So let's start this one from the beginning to make sure we are all on the same page.

There is a "mind trap" buried in this problem, and it's possible your professor doesn't know about it. The difficulty is that the instant in which the motion starts is slightly different than the instant after the motion starts. Thus the applied force to make the system move is perhaps different from what you professor would tell you.

The instant in which the motion starts:

We have two Free-Body Diagrams, one for m1 and one for m2. I am going to define +x to be positive to the right and +y straight upward in both diagrams.

FBD for m2: We have a weight (w2) straight down, a normal force from the table (N2T) straight up, and a normal force from m1 (N21) straight downward. I will apply the force (F) to this block to the right (in the +x direction.)

Now, we apply just enough of an F to make m2 move. This means two things:

1)We are barely applying enough F to counter the static friction between the table and m2. Thus the friction force on m2 (f2T) due to the table is at it's maximum static friction value and is to the left (because it is resisting F, which is to the right) and that m2 is not accelerating.

2) The friction force on m1 due to m2 (f12) is going to be to the right because m2 is making it move to the right. Thus the friction force on m2 by m1 (f21, make sure you've got the notation here) must be to the left, because it is a Newton's 3rd Law pair to f12. This will in general NOT be equal to the maximum possible static friction force, so we have no formula for this force. In fact, we need to check to make sure that m1 is not sliding; we don't even know yet if we have static friction or not.

FBD for m1: We've got a weight (w1) acting straight down, a normal force from m2 (N12) straight upward, and a friction force (f12) acting to the right.

A few more notational points. I'm going to call the coefficient of static friction between m2 and the table u(s2T) and the coefficient of statice friction between m2 and m1 u(s21). Likewise the coefficients of kinetic friction will be u(k2T) and u(k21) respectively.

So. Newton's 2nd Law on m2:

F(net,x) = F - f2T - f21 = 0 (Remember m2 is not accelerating)

F(net,y) = N2T - w2 - N21 = 0

So F = f2T + f21 = u(s2T)*N2T + f21

and

N2T = w2 + N21 = m2*g + N21

To finish this we need Newton's 2nd on m1:

F(net,x) = f12 = m1*a

F(net,y) = N12 - w1 = 0

So N12 = w1 = m1*g

Thus going back to the N2T equation we get:

N2T = m2*g + N21 = m2*g + m1*g (N21 = N12 by Newton's 3rd Law)

Thus going back to the F equation we get:

F = u(s2T)*N2T + f21 = u(s2T)*(m2*g + m1*g) + f21

We've still got that pesky f21.

We know that m1 has to be accelerating. Why? Look at the F(net,x) equation for m1. f12 = m1*a. Now if there were no friction bewteen m1 and m2 then the acceleration would be 0 so the blocks would move together. But if the friction is 0 then there is nothing to make m1 move! So we have to assume a non-zero f12 = f21, which means that m1 is accelerating.

Now the question is, is the friction on m1 static? Well, if it was then m1 would not be sliding over the surface of m2 and thus they would have the same acceleration. (Note that m2 is moving with no acceleration, thus v = constant. But it IS moving.) But they can't: m2 has no acceleration and m1 does. Thus the friction between m1 and m2 must be kinetic, which is a good thing because that gives us a formula to use:

f12 = u(k21)*N12 = f21 (By Newton's 3rd Law.)

So:

F = u(s2T)*(m2*g + m1*g) + f21 = u(s2T)*(m2*g + m1*g) + u(k21)*N12

F = u(s2T)*(m2*g + m1*g) + u(k21)*m1*g

(Inserting the units at the end):

F = 0.5*(15.5*9.8 + 5.4*9.8) + 0.1*5.4*9.8 = 107.702 N

The instant after the motion starts:

Now m2 is sliding so the friction goes from static to kinetic. That means that we are applying too much force to just barely keep the system in motion (since u(s2T) > u(k2T): I have yet to see this contradicted, but I don't know why they couldn't be, say, equal.) So now m2 is accelerating to the right. All the features of the FBDs are the same as before, and once again we need to find out if the friction between m1 and m2 is static or kinetic.

So: Newton's 2nd Law on m2:

F(net,x) = F - f2T - f21 = m2*a2 (m1 and m2 may have different accelerations, and F is what we calculated before.)

F(net,y) = N2T - w2 - N21 = 0

Again we have N2T = w2 + N21 = m2*g + N21

and

Newton's 2nd on m1:

F(net,x) = f12 = m1*a1

F(net,y) = N12 - w1 = 0

So N12 = w1 = m1*g

Thus N2T = m2*g + N21 = m2*g + m1*g

and

a2 = (1/m2)*(F - f2T - f21) = (1/m2)*(F - u(k2T)*N2T - f21)

a2 = (1/m2)*(F - u(k2T)*(m2*g + m1*g) - f21)

Again (sigh) what's f21 = f12?

Well, what would it take to make m1 slide over m2? A force greater than u(s21)*N12 = u(s21)*m1*g, the maximum possible static friction force bewteen m1 and m2. We can calculate this: 0.1*5.4*9.8 = 5.292 N. So the question is whether f12 is greater than 5.292 N?

Well, if m1 is static with respect to m2 then a = a1 = a2. Then we can do an FBD on the whole system and get that

F - u(k2T)*(m2 + m1)*g = (m2 + m1)*a

a = (1/(m2 + m1))(F - u(k2T)*(m2 + m1)*g) =

1.23321 m/s^2

So if the friction between m2 and m1 is static, we know the acceleration of m1: a1 = 1.23321 m/s^2 and thus we know f12 = m1*a1 = 6.65931 N > 5.292 N. Therefore m1 must be sliding across the surface of m2 and we must use kinetic friction.

Back to where we were:

a2 = (1/m2)*(F - u(k2T)*(m2*g + m1*g) - f21)

f21 = f12 = u(k21)*N12 = u(k21)*m1*g, so

a2 = (1/m2)*(F - u(k2T)*(m2*g + m1*g) - u(k21)*m1*g)

a2 = 1.32142 m/s^2

To get the acceleration a1:

f12 = m1*a1

a1 = (1/m1)*f12 = (1/m1)*u(k21)*N12 = (1/m1)*u(k21)*m1*g = u(k21)*g

a1 = 0.98 m/s^2

-Dan