You are right, it is the distance over which the force is applied, not the duration of time, which is relevant in the computation of work done. However, you cannot say that if you double the force, the time will be cut in half. Displacement varies quadratically with acceleration, which is proportional to the (net) applied force. Since there is a force of kinetic friction to contend with, doubling the applied force will not double the net force.