Neither of those imply a right to not be annoyed. They imply a right to walk away from the annoyance. They would, similarly, have the “right” (in whatever weird sense the word is being used, here) to craft a plan that would make that difficult. And so on.

“Rights” are a bizarre/incoherent way to look at this.

They don't have any right to my attention precisely because I have the right to walk away. How is that not the moral framework involved?