Why are there no physical theories depending on the continuum hypothesis?

The continuum hypothesis or its negation can be added to ZFC without changing its validity. Yet, to my knowledge, there are no known real-world consequences of either choice. How can that be? How do we decide which axioms in a fundamental mathematical theory matters?

Further, to me, if there are no observable effects of an axiom, it should be taken as false. This is of course a problematic position to take, since those observable effects might show up much later. Thus, it is probably sensible for mathematicians to investigate the consequences of choosing either that or that. In that way, a path to observability might be found.

But, my position might have a more fundamental issue. What axiom should be taken as false? Should I choose A or ¬A as false? This will probably come down to applicability. If A provides many weird theorems and applications (as e.g. the axiom of choice does) then probably A should be taken as false.