I didn't pick up a maths book until I was 21. I'm 24 now, and I've continued like this ever since. I'm often haunted by the thought of "late-starter" and the question, "If I started as early as 3, how much would I know today?" I'll never be first-class but I feel confident to say I'm above average.
When you refer to calculus, I'll make the assumption it's single-variable calculus. I don't find it too difficult. I can make it hard on myself at times. The procedure of jumping from a Riemann sum straight to a definite integral so as to compute the area under the curve annoyed me (especially because it was the fundamental theorem of calculus!) in that authors can get lazy with their proof.
Have you considered MIT open courseware on YouTube? It's really clear and concise. David Jerrison is one of my favourite modern mathematicians.