User-Agent: *
Allow: /
Disallow: /api/
Disallow: /dashboard
Disallow: /admin
Disallow: /account
Disallow: /nav-state
Disallow: /md/
Content-Signal: search=yes, ai-input=yes, ai-train=yes

Host: https://scrapercompany.com
Sitemap: https://scrapercompany.com/sitemap.xml