Extend your negative-rule to hours. It is much easier in the 24h-format:

Subtract the hours

Subtract the minutes

If the minutes are negative, add 60 to the minutes and subtract 1 from hours.

If the hours are negative, add 24 to the hours and subtract 1 form the days.

AM/PM would require to study a lot of cases, just convert the times before calculating with them.