Knowledge base›Technical SEO›What is robots.txt?

    Technical SEO

    What is robots.txt?

    Instructions for crawlers about what they may request.

    robots.txt lives at the root of the site. It says which paths a crawler may fetch. The file does not remove a URL from search results. That takes noindex or another indexing setting.

    Was this article clear?