What makes the latter sentence sound highfalutin is that you've been required to contort it away from idiomatic American English sentence structure in order to force in a "whom". The usual way of phrasing the sentence avoids "who" entirely: "If the people I speak with" or "If the people I'm talking to".