Maybe I'm being foolish, but isn't the point of e2e that Facebook wouldn't even know what you were sending (a link or otherwise), it being encrypted in flight?

The WhatsApp client knows, since that's the "end." Nothing technical stops Facebook from bundling some code in the client to pass data about the messages back to a central server.

Yes, but it's hard to trust e2e when keys go through a blackbox (Facebook's API), and clients are controlled by Facebook (closed source applications and JS).

It's better than no encryption, but not what technical people usually mean by e2e encryption

Facebook can send the URLs from client to their service, that just bypasses the e2e channel and opens up a nice side channel for Facebook to peek into messages.