This is a misconception I see pop up frequently online. In terms of the color spectrum, there are plenty of things—even things that have qualities in common with color—that aren't on the color spectrum. And while there are colors outside of what humans can see, we generally use it not to refer to the entire electromagnetic spectrum, but only to the subset that makes up light visible to human eyes.

Likewise, when we talk about the "autism spectrum," we're not including every exhibition of traits associated with autism. You can have some traits associated with autism without being "on the spectrum."

Also, perhaps as importantly, "spectrum" isn't a term that generally applies only to color, or even electromagnetism.

I would argue that for the average person, therefore, 'spectrum' is an unfortunate choice of analogy, since most people believe that it encompasses every possible color. One should not need specialist knowledge to discuss an issue of this kind in common terms.

The idea that the word "spectrum" doesn't automatically imply "color" is hardly specialist knowledge, though.

"There's a spectrum between..." is not an uncommon way to describe something where there's a range of possible things.