Magento robots.txt file content

Posted by Damodar Bashyal on January 31, 2014

 

From today onwards I am going to apply these rules to fix issues with SEO.

Magento admin / System / configuration / General / Web / Auto-redirect to Base URL / Yes (301 Moved Permanently)

Also, add robots.txt file with below content.

User-agent: *
Disallow: /*.css$
Disallow: /*.js$
Disallow: /*.php$
Disallow: /*?SID=
Disallow: /404/
Disallow: /app/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /cgi-bin/
Disallow: /checkout/
Disallow: /control/
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /customer/
Disallow: /customize/
Disallow: /downloader/
Disallow: /error_log
Disallow: /errors/
Disallow: /includes/
Disallow: /index.php/
Disallow: /install.php
Disallow: /js/
Disallow: /lib/
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /magento/
Disallow: /newsletter/
Disallow: /pkginfo/
Disallow: /poll/
Disallow: /report/
Disallow: /review/
Disallow: /scripts/
Disallow: /sendfriend/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /STATUS.txt
Disallow: /tag/
Disallow: /var/
Disallow: /wishlist/

That should help to reduce waste of server resources and hits on the server, making servers perform better for actual customers. Do you recommend any other magento tips and tricks to improve seo?

 
not published on website


QR Code: Magento robots.txt file content