Honestly, despite all the crap universities get, taking an undergraduate degree with a double major in physics and maths is an awesome way to do this. You'll meet people who are similarly passionate, be naturally competitive with them which is a motivating force not to be underestimated, and you'll meet a diverse set of teachers who each will have some awesome insights into these fields and you'll get to see first-hand how they think about solving problems.
Physics, and to a lesser extent maths[1], are topics where the top 1% of ability are actually concentrated at universities. My advice would be find a cheap university nearby and start enrolling in courses. If you're bright, motivated and take ownership of your own learning, the faculty will love interacting with you. If you're doing it to learn, don't sweat about the prestige of the place. There are people everywhere who will be much better than you at this stuff, and in some ways it's extremely motivating if you feel like with some hard work you can surpass some of your teachers, and it's extremely motivating when the best teachers recognize you as having more potential than the average student. You're never gonna feel either of these things at MIT.
[1] The problem with maths in academia is that it's massively biased toward proofs of mathematics and not use of mathematics. I've met very few PhD mathematicians who are even as close to as good at applying appropriate mathematics to problems then someone with a PhD in physics who consider themselves >50% theorist. PhD mathematicians are wonderfully knowledgeable if you say "tell me about this field of mathematics" and it's a field they know. But there is a certain extent to which they like to work by building things on a frictionless ice world, and get uncomfortable if asked to build something on the rough ground of the real world.
Physics, and to a lesser extent maths[1], are topics where the top 1% of ability are actually concentrated at universities. My advice would be find a cheap university nearby and start enrolling in courses. If you're bright, motivated and take ownership of your own learning, the faculty will love interacting with you. If you're doing it to learn, don't sweat about the prestige of the place. There are people everywhere who will be much better than you at this stuff, and in some ways it's extremely motivating if you feel like with some hard work you can surpass some of your teachers, and it's extremely motivating when the best teachers recognize you as having more potential than the average student. You're never gonna feel either of these things at MIT.
[1] The problem with maths in academia is that it's massively biased toward proofs of mathematics and not use of mathematics. I've met very few PhD mathematicians who are even as close to as good at applying appropriate mathematics to problems then someone with a PhD in physics who consider themselves >50% theorist. PhD mathematicians are wonderfully knowledgeable if you say "tell me about this field of mathematics" and it's a field they know. But there is a certain extent to which they like to work by building things on a frictionless ice world, and get uncomfortable if asked to build something on the rough ground of the real world.