AI Crawler Mitigation via a Few Lines of Caddy

To mitigate AI crawlers, simply require a bit of Javascript:

domain.com {
    # Match all requests without a "verified" cookie"
    @unverified not header Cookie *verified*

    # Serve them a JS page that sets the cookie
    handle @unverified {
        header Content-Type text/html
        respond <<EOF
            <script>
            document.cookie = 'verified=1; Path=/;';
            window.location.reload();
            </script>
        EOF 418
    }

    # Handle all other requests normally
    reverse_proxy localhost:3001
}

This works because these crawlers have disabled Javascript.