In virtually all jobs in industry or academia technical work requires programming of some sort.
Unless you plan on programming or doing research on operations systems or perhaps artificial intelligence I wouldn't go into computer science. Learning CS to learn to program is not an efficient way to go.
Perhaps not in math but certainly in engineering and physics you learn to program somehow. We had a semester on it and that was 30 years ago. I imagine it's become more important over the years.
I learned the programming I needed for work the hard way, self study and practice. There are a zillion books out there to teach yourself the language of your choice. I'm a bit out of date so I won't recommend one. I learned C and assembly and then C++, learning a bunch of others, pascal, fortran, forth, lisp, etc. along the way. I was an electrical/systems engineer and was programming or at least designing programs constantly.
I'd stick with the physics. You can pick up all the important CS on your own and if you end up wanting to go down a more advanced CS path you can probably get someone to pay you to do that.
I'd even consider making physics your major with math as your minor. In physics you'll have to pick up a ton of math anyway. Pretty advanced math these days. You could probably even double major and have a bunch of overlap. Physics is an extremely exciting field at the moment.
Just my opinion.