I can only tell you that I do not have mental imagery and am not constantly surprised by my surroundings. Object permanence has absolutely no dependency on visualization; it is completely unsurprising to me that the stop sign near my house looks the same each time I encounter it.
I totally get that you, having lived a life where mental imagery is such an integral part of your baseline experience, assume that many of the things you rely on it for require it. However, the human brain is impressively adaptable, and it turns out many, many aspects that people assume are linked (and may well be for them as individuals) are not globally so.
This thread is full of such realizations - people assuming ability at chess, art, tetris, spatial reasoning, abstract reasoning, architectural design, conceptualizing of DB schemas, etc. must be correlated with facets of thought like strength of mental imagery, presence of an inner monologue, ability to dream, etc. In all of those cases, people have chimed in with (anecdotal, to be sure) counterexamples. It turns out brains are generally capable of doing a lot of different things in a lot of different ways.
In another comment you wrote "stop sign is a red octagon (with a white boundary) with the word "STOP" in the center. I could draw you a plausible picture of one without issue". The fact that you knew all of that without having to actually find a stop sign and look at it, that's mental imagery. It may be a skill some people are very good at (i.e. they can remember lots of small details) and others are not, but it's not that you don't have this skill at all.
What I know is that people describe not having aphantasia like
“I’m sitting in a room with a dog and a TV. Then someone says ‘visualize a stop sign.’ And, so then I’m sitting in a room with a dog, a TV and a stop sign. I’m know the stop sign isn’t real. It might be fuzzy or even black & white. But, I can see it between the dog and the TV.”
Well… I know exactly what a stop sign looks like. But, there’s no stop sign in the room with me and my dog. It’s not there at all.
No, that is more like hyperphantasia, where the visualized object is superimposed on top of one's visual reality. Seeing something in one's mind's eye is more like there is a a buffer right above my visual reality where I can imagine something inside that buffer. It is not superimposed over reality but I can focus on both the visual buffer and the imagination buffer simultaneously.
I totally get that you, having lived a life where mental imagery is such an integral part of your baseline experience, assume that many of the things you rely on it for require it. However, the human brain is impressively adaptable, and it turns out many, many aspects that people assume are linked (and may well be for them as individuals) are not globally so.
This thread is full of such realizations - people assuming ability at chess, art, tetris, spatial reasoning, abstract reasoning, architectural design, conceptualizing of DB schemas, etc. must be correlated with facets of thought like strength of mental imagery, presence of an inner monologue, ability to dream, etc. In all of those cases, people have chimed in with (anecdotal, to be sure) counterexamples. It turns out brains are generally capable of doing a lot of different things in a lot of different ways.