This is for the US and probably wrong in lots of other places. Here, it is common to change fields between a bachelors and a doctoral program. But this is because of the nature of both. The undergraduate program here is very general. Math students study not only math but science, philosophy, history and other fields, often including language. In other places the undergrad program is often much more focused.
So, if you have a UK bachelors and it is tightly focused on math, then it will be harder even here to enter a doctoral program in a different field. The "preliminary" coursework in a doctoral program is generally advanced courses, not introductions. So, I'd guess that you have some gaps to fill in switching fields. I wouldn't say "impossible", but I'd suggest it will be a challenge.
Note that as a US math undergraduate I did actually have a bit of physics lab experience.
I suggest that you talk to someone in physics near where you are to get a local answer. And you only really learn whether you can be accepted by applying.