Rules for particular files
Block crawling of a CDN domain
Block web crawlers from crawling a CDN custom domain by returning a disallow robots.txt response.
Was this page helpful?
Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Block web crawlers from crawling a CDN custom domain by returning a disallow robots.txt response.
Was this page helpful?