Robots.txt file In magento
- Here are the pages we recommend barring off from search engines:
- Internal search result pages: /catalogsearch/ and /search/. While we recommend adding the robots noindex directive (either the meta tag or HTTP header) to these pages as a second line of defense, the first line of defense is the robots.txt file.
- Login pages /customer/account/login/ – login pages.
- URLs containing SID and PHPSESSID parameters.
- Reference the XML sitemap in the robots.txt file.
- Filtering criteria such as price, color, material and size.
- This boils down to the following robots.txt:
- User-agent: *
Disallow: /catalogsearch/
Disallow: /search/
Disallow: /customer/account/login/
Disallow: /*?SID=
Disallow: /*?PHPSESSID=
Disallow: /*?price=
Disallow: /*&price=
Disallow: /*?color=
Disallow: /*&color=
Disallow: /*?material=
Disallow: /*&material=
Disallow: /*?size=
Disallow: /*&size= - In order to edit your robots.txt file in Magento 2, navigate to:
- On the screen that follows, scroll down to Search Engine Robots. There you can define your robots.txt content:
- XML sitemap: limits and robots.txt
- Things get interesting down below; under Sitemap File Limits, you’ll find the fields Maximum No of URLs Per File and Maximum File Size.
- For most Magento stores, the default Maximum No of URLs Per File limit is correct as it aligns with SEO best practices.
- You can raise the maximum file size to 52428800, as this equals 50 MB.
- Once this is all set up, it’s time to create the XML sitemap and configure its (re)generation settings. To do this, go to Marketing > Site Map (under SEO & Search) and click the orange Add Sitemap button.
Your Magento 2 store has a variety of URL patterns that you don’t want search engines to touch. :
Editing your robots.txt file in Magento 2
STEP 1:
Content > Configuration (Under Design).
STEP 2:
There, edit the settings for Main Website instead of the Default Store View, because the latter won’t allow you to change the robots.txt file.
XML sitemap: Maximum No of URLs Per File
XML sitemap: Maximum File Size
XML sitemap: Enable Submission to Robots.txt
In the screen that follows, you define the XML sitemap’s name and location: :
STEP 1:
Fill in the XML Sitemap’s name in the field Filename, example: sitemap.xml.
STEP 2:
Then fill in the XML sitemap’s path in the Path field – for example /pub/ in our case because of the nature of the Magento installation in our test setup.
STEP 3:
Next, click the Save & Generate button, and you’re all done!
