I used to work one of the top 1000 visited websites, and we have massive bot issues where 60% of our traffic was bots and had to implement solutions similar to cloudflare to reduce the bots. Also, with the raise of ai, it's become even more important since a lot of ai data scraping companies do not respect robots.