you know that if one of A and B is normal, then AB will be a subgroup. so the group G that you're looking for has to be non-abelian. there are many examples. here are two of them:

Example 1G finite: let then which doesn't divide so AB is not a subgroup.

Example 2G infinite: let let then now

but so AB is not a subgroup.