I am seeing from these comments that my proposed solution was pretty naïve. I intended this blog post to be a sort of thought experiment to challenge assumptions made about the web pre-AI rather than my thought up technical solution to be the main focus.

I might go back and re-write some of this post to gear the focus more towards my main points of the social contract between bot and human shifting (especially with copyleft/ShareAlike), the web becoming less “open”, how this is not a new idea since the DMCA already considers standards for automated access, etc.

Its a link to my blog post:

https://agamsingh9.codeberg.page/posts/ai-web-scrapers/

Is the link not working?

I was almost done writing my blog post addressing the same issue! I’ll give this a read to make sure I don’t re-say anything you did. The state site generator I use (Hugo) also looks very similar to yours lol.

I have found more websites refuse to load with Mullvad VPN than when I use Proton VPN. I use Mullvad on my phone since it’s the only one that is able to be used while I also use Tailscale VPN at the same time on Android.

Is this hosted somewhere? Maybe distributed? I would love a privacy respecting distributed LLM chatbot.