Same ratio roughly. 80% Crawlers and agents, 20% human. Loads of the agents actually serve the content to humans, mostly in ChatGPT.

Wow, that's a great point... I hadn't considered that. I assumed it's all training.

From what I understand, Cloudflare is trying to create a way for agents to consume content in a more structured manner than allowed for attention to the author, and potentially payment along with it.

I don't want to be paid but I'd love to see how often context from my writing winds up in a session a human is actively using.

It is good to make a proper distinction, in the ChatGPT context, between crawlers and agents. The crawlers go for the content to build a new model, the agents serve content to users. The last one can be very useful.

They use different user-agent strings. The crawlers obfuscate themselves and use residential proxies. The agents call themselves ChatGPT-User. Of course Cloudflare wants OpenAI to pay them for not blocking ChatGPT-User by default.

It's true, crawlers used for AI training don't say they are crawlers at all.

I am on a low tier Ghost subscription, I could not rewrite some of the HTML. So I do this with Cloudflare and then cache it again.

Yes, the architecture setup is generated by ChatGPT but in itself it says what it needs.

I started blocking some of them. But for now I want to improve visibility before further blocking or optimising. The dashboard helps with this.

Thanks. It seems to be very local/incidental. The page works from the locations I can test, but I’ll check whether one edge cache or request path served a bad response.

White page from one of Germany's largest ISPs.

Built a Cloudflare Worker that classifies all traffic by visitor type: humans, AI crawlers, SEO bots, residential proxies. On most days, humans are a minority. The article explains what the patterns reveal and why edge logs show things your analytics tool can't.

A short story on the decades long search for the blue LED, the missing piece in the production of screens and energy efficient white light.

Social media platform Twitter reverted to cropping images in the timeline. Not for the web and desktop app, but they do for iPhone.

The new Visual Look Up finds words in the tiniest of corners. A story on how I searched for SOUL and found it in a little Amsterdam bookstore.

How small bits of text explain and support your message and make your story complete.