User-agent: Scrapy Disallow: / User-agent: * Disallow: Sitemap: https://demo.cbr-iisc.ac.in/sitemap_index.xml