Meta robots and X-Robots-Tag on the Tranco top 1,000 homepages (550 read, 2026-10-06): which page-level AI directives are used, conflicts with headers...
Next.js 16 streaming metadata sends users a page before generateMetadata resolves, but blocks crawlers for a complete head. How it works, plus a cheat...