robots.txt lives at the root of the site. It says which paths a crawler may fetch. The file does not remove a URL from search results. That takes noindex or another indexing setting.
Was this article clear?
Knowledge base›Technical SEO›What is robots.txt?
Technical SEO
Instructions for crawlers about what they may request.
robots.txt lives at the root of the site. It says which paths a crawler may fetch. The file does not remove a URL from search results. That takes noindex or another indexing setting.