0
Welcome Guest! Login
0 items Join Now

Hiding of layered nav for bots

    • ladbroke's Avatar
    • ladbroke
    • Sr. Rocketeer
    • Posts: 166
    • Thanks: 4

    Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Hi, I have seen a load of layered nav page results turning up in google webmaster tools. What are people's thoughts on hiding layered nav from bots? I saw this post and have managed to implement it on my RT Ionosphere template (The solution is in the 5th post) www.magentocommerce.com/boards/viewthread/347945/

      In Ionoshpere the file is /app/design/frontend/base/default/template/rokmagemodules/rokmage-categoryview/rokmage-categoryview-filter.phtml

      I've 'viewed as google' in GWMT and the layered nav is now hidden.

      Do people think this is beneficial, or more of a bad idea?

      p.s. there is an error in the last part of the code in that link, but once corrected it works.
    • Last Edit: 5 years 11 months ago by ladbroke.
    • Matt's Avatar
    • Matt
    • Preeminent Rocketeer
    • Posts: 20628
    • Thanks: 2926
    • messin' with stuff

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Your better to simply mark these pages with a meta tag rel=canonical or, honestly, just don't worry about it... mostly (I would think) the layered navigation should have fairly unique content since each page is a different filter of your products... and for my money, the more pages I can get Google to index, the better...

      support.google.com/webmasters/answer/66359?hl=en

      //I'm still pretty new to Magento though... so curious if your saying these layered nav pages are duplicate content or not... as you can see in the post above though, rel=canonical is Google's recommendation
    • SEARCH the forum first! These boards are rich in knowledge and vast in topics :woohoo:
    • Ben Lee's Avatar
    • Ben Lee
    • Elite Rocketeer
    • Posts: 4193
    • Thanks: 42

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • This is kind of a big topic and its not an easy one to solve. There are several suggested solutions or ways to go about "cleaning up", but none are perfect. Hiding the nav block from bots is a good bit of it though.

      Here is a great video to watch that explains all aspects of this issue:

      Magento SEO: How to handle problems caused by layered navigation?

      And here's one of those "while you're at it..." things that you can take a look at:

      Ultimate Magento Robots.txt File Examples

      I have personally done the webmaster tools edits to tell Google what urls to stop tracking and I also have a custom robots.txt file. I have not gone with the hiding the nav block procedure just yet, but I may add that.
    • The following users have thanked you: ladbroke, Matt

    • ladbroke's Avatar
    • ladbroke
    • Sr. Rocketeer
    • Posts: 166
    • Thanks: 4

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Thanks guys, it's good to kick some ideas around. In robots.txt telling Google not to crawl any URL containing a '?' would essentially do the same as hiding the layered nav, so maybe a bit overkill with hiding layered nav completely? From what I can make out Disallow: /*? would stop any layered nav URL's from being crawled.

      **Edit, after watching the video in that link, rather than just reading the text, I'll stick with both the robots.txt Disallow: /*?, and hiding the layered nav! You would think that telling Google to ignore any url containing a ? would be simple enough, but obviously not!
    • Last Edit: 5 years 11 months ago by ladbroke.
    • ladbroke's Avatar
    • ladbroke
    • Sr. Rocketeer
    • Posts: 166
    • Thanks: 4

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Incidently, even with both those measures in place, BingBot (which seems to always be crawling my site, continuously) is still crawling URL's containing a ?......I'm not too bothered by Bingbot, but really want things to be correct with Google. Fingers crossed, wait and see!
    • Matt's Avatar
    • Matt
    • Preeminent Rocketeer
    • Posts: 20628
    • Thanks: 2926
    • messin' with stuff

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • could take bingbot a bit to refetch your robots.txt... it could be using a cached version

      www.bing.com/blogs/site_blogs/b/webmaste...gbot-s-question.aspx

      //good stuff all around on this thread... I'll be doing some optimization for the first time on a new (my first) Magento site so I'm glad this came up!
    • SEARCH the forum first! These boards are rich in knowledge and vast in topics :woohoo:
    • ladbroke's Avatar
    • ladbroke
    • Sr. Rocketeer
    • Posts: 166
    • Thanks: 4

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • I've done the same with the Sort by: code today to prevent lots of weird url's being crawled. I used the same code that I used for layered nav, but placed it in /app/design/frontend/base/default/template/catalog/product/list/toolbar.phtml

      It seems to have worked when I 'view as google' in webmaster tools. I might be being really paranoid with this, not too sure!
    • ladbroke's Avatar
    • ladbroke
    • Sr. Rocketeer
    • Posts: 166
    • Thanks: 4

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Just an update on this if anybody is interested. I've attached a screenshot form GWT and it shows a massive drop in URL errors. I have had some sitemap issues (in a separate thread), but even putting that to one side, I've seen a steady drop since hiding layered nav and sort by: from google. I feel pretty sure the remaining 135 url errors will drop off over the next couple of days.

      What the benefits of this will be long term, I don't know, hopefully increased sales! But if nothing else it allows me to focus on genuine errors with out combing through thousands of layered nav URLS first.
      This image is hidden for guests.
      Please log in or register to see it.
    • The following users have thanked you: Matt

    • Ben Lee's Avatar
    • Ben Lee
    • Elite Rocketeer
    • Posts: 4193
    • Thanks: 42

    Re: Hiding of layered nav for bots

    Posted 5 years 11 months ago
    • Thanks for the update!

      I think I'm going to seriously look at doing this on my sites now!

Time to create page: 0.097 seconds