like an exotic word that people only pull out to be semi-fancy, like "whom"
Semi-fancy? Man, that's a pretty low bar for fifty-cent words. I use it so I sound like I actually went to school and paid attention. If those with whom I speak find basic grammar fancy, that's on them.
What makes the latter sentence sound highfalutin is that you've been required to contort it away from idiomatic American English sentence structure in order to force in a "whom". The usual way of phrasing the sentence avoids "who" entirely: "If the people I speak with" or "If the people I'm talking to".
Semi-fancy? Man, that's a pretty low bar for fifty-cent words. I use it so I sound like I actually went to school and paid attention. If those with whom I speak find basic grammar fancy, that's on them.