Hiding of layered nav for bots
-
-
- ladbroke
- Sr. Rocketeer
- Posts: 166
- Thanks: 4
-
Hi, I have seen a load of layered nav page results turning up in google webmaster tools. What are people's thoughts on hiding layered nav from bots? I saw this post and have managed to implement it on my RT Ionosphere template (The solution is in the 5th post)
www.magentocommerce.com/boards/viewthread/347945/
In Ionoshpere the file is /app/design/frontend/base/default/template/rokmagemodules/rokmage-categoryview/rokmage-categoryview-filter.phtml
I've 'viewed as google' in GWMT and the layered nav is now hidden.
Do people think this is beneficial, or more of a bad idea?
p.s. there is an error in the last part of the code in that link, but once corrected it works. - Last Edit: 10 years 13 hours ago by ladbroke.
-
-
-
- Matt
- Preeminent Rocketeer
- Posts: 21561
- Thanks: 3090
- messin' with stuff
-
Your better to simply mark these pages with a meta tag rel=canonical or, honestly, just don't worry about it... mostly (I would think) the layered navigation should have fairly unique content since each page is a different filter of your products... and for my money, the more pages I can get Google to index, the better...
support.google.com/webmasters/answer/66359?hl=en
//I'm still pretty new to Magento though... so curious if your saying these layered nav pages are duplicate content or not... as you can see in the post above though, rel=canonical is Google's recommendation - SEARCH the forum first! These boards are rich in knowledge and vast in topics. This includes searching just the 'Solved' forums, using Google, and using ChatGPT
-
-
-
- Ben Lee
- Elite Rocketeer
- Posts: 4193
- Thanks: 42
-
This is kind of a big topic and its not an easy one to solve. There are several suggested solutions or ways to go about "cleaning up", but none are perfect. Hiding the nav block from bots is a good bit of it though.
Here is a great video to watch that explains all aspects of this issue:
Magento SEO: How to handle problems caused by layered navigation?
And here's one of those "while you're at it..." things that you can take a look at:
Ultimate Magento Robots.txt File Examples
I have personally done the webmaster tools edits to tell Google what urls to stop tracking and I also have a custom robots.txt file. I have not gone with the hiding the nav block procedure just yet, but I may add that.
-
-
-
- ladbroke
- Sr. Rocketeer
- Posts: 166
- Thanks: 4
-
Thanks guys, it's good to kick some ideas around. In robots.txt telling Google not to crawl any URL containing a '?' would essentially do the same as hiding the layered nav, so maybe a bit overkill with hiding layered nav completely? From what I can make out Disallow: /*? would stop any layered nav URL's from being crawled.
**Edit, after watching the video in that link, rather than just reading the text, I'll stick with both the robots.txt Disallow: /*?, and hiding the layered nav! You would think that telling Google to ignore any url containing a ? would be simple enough, but obviously not! - Last Edit: 10 years 4 hours ago by ladbroke.
-
-
-
- ladbroke
- Sr. Rocketeer
- Posts: 166
- Thanks: 4
- Incidently, even with both those measures in place, BingBot (which seems to always be crawling my site, continuously) is still crawling URL's containing a ?......I'm not too bothered by Bingbot, but really want things to be correct with Google. Fingers crossed, wait and see!
-
-
-
- Matt
- Preeminent Rocketeer
- Posts: 21561
- Thanks: 3090
- messin' with stuff
-
could take bingbot a bit to refetch your robots.txt... it could be using a cached version
www.bing.com/blogs/site_blogs/b/webmaste...gbot-s-question.aspx
//good stuff all around on this thread... I'll be doing some optimization for the first time on a new (my first) Magento site so I'm glad this came up! - SEARCH the forum first! These boards are rich in knowledge and vast in topics. This includes searching just the 'Solved' forums, using Google, and using ChatGPT
-
-
-
- ladbroke
- Sr. Rocketeer
- Posts: 166
- Thanks: 4
-
I've done the same with the Sort by: code today to prevent lots of weird url's being crawled. I used the same code that I used for layered nav, but placed it in /app/design/frontend/base/default/template/catalog/product/list/toolbar.phtml
It seems to have worked when I 'view as google' in webmaster tools. I might be being really paranoid with this, not too sure!
-
-
-
- ladbroke
- Sr. Rocketeer
- Posts: 166
- Thanks: 4
-
Just an update on this if anybody is interested. I've attached a screenshot form GWT and it shows a massive drop in URL errors. I have had some sitemap issues (in a separate thread), but even putting that to one side, I've seen a steady drop since hiding layered nav and sort by: from google. I feel pretty sure the remaining 135 url errors will drop off over the next couple of days.
What the benefits of this will be long term, I don't know, hopefully increased sales! But if nothing else it allows me to focus on genuine errors with out combing through thousands of layered nav URLS first.
This image is hidden for guests.
Please log in or register to see it. -
The following users have thanked you: Matt
-
Time to create page: 0.058 seconds