Extend your negative-rule to hours. It is much easier in the 24h-format:
Subtract the hours
Subtract the minutes
If the minutes are negative, add 60 to the minutes and subtract 1 from hours.
If the hours are negative, add 24 to the hours and subtract 1 form the days.
AM/PM would require to study a lot of cases, just convert the times before calculating with them.