I think the best reason for why the dot and cross products are what they are is because they are useful. There are probably an infinitude of ways to define a product between two vectors.

In Physics the first basic use of the dot product comes in the form of the work equation:

. Work in this sense is more or less the "effort" required to accomplish a task, and it makes sense (when you think about it for a while) that the work done should depend on the size of the angle between the applied force and displacement. The use of the cosine function is pretty much a convenience, since it has the properties that are desirable. Even the concept of no work being done by an applied force that is perpendicular to a given displacement makes a certain amount of sense: the force gets nothing done.

As for the cross product, the first place a Physics student sees it is the torque equation, but I think the best argument for its use is actually for angular momentum. I won't go into details (unless you really want them) but again there are sensible features to using the sine of the angle between the vectors and (if you use some imagination) a reason for the perpendicular vector nature of the result.

I will note that the dot product between two vectors is slightly more general than the cross product because there is a very easy way to generalize the dot product to higher dimensions, but as far as I know the cross product as defined can only be used in 3D. (I believe that the problem is not that it can't be generalized, but that it can't be uniquely generalized and that there is more than one useful generalization, but please don't quote me on that.)

-Dan