jump to last post 1-4 of 4 discussions (5 posts)

How Does the Robot.txt Blocked URL Google Site Diagnostics Work

  1. glendoncaba profile image81
    glendoncabaposted 6 years ago

    I have been checking my site diagnostics for AdSense and saw a robot.txt blocked URL message I just don't understand.  What does this mean?  The solution involves pasting something on page but i am afraid to mess up the page.  Help!
    There are three URL's for the same hub that the adsense crawler is unable to access.

    This is the message for solving the problem: 

    If you would like to grant our crawler access to your pages, you can do so without granting permission to any other bots. Simply add the following two lines of text to the top of your robots.txt file:

    User-agent: Mediapartners-Google

    This change will allow our crawler to index the content of your site and provide you with the most relevant Google ads for that content.

    EDIT:  Not 3 URL's but the same one shown as blocked on 3 days.  Does this explain the reduction in clicks for past couple of days?

  2. mel22 profile image60
    mel22posted 6 years ago

    if I'm right i beleive its within the meta tag which is there for hub staff only (admin ) to prevent what access a regular hubber has access to. I asked before if I could change the normal (Blu) text ads to another colr in the source code and I was given a resounding NO! I beleive only admin has access to certain criteria within the hub source through locked code. I may be wrong but either way I was given a resounding No! even analytics code was taken care of by admin rather than pasting in code directly ,by pasting to affiliates page. I thnk its to keep people with messing with a uniform look within hubpages overall. I beleive adsense code (outside of hubpages) on a personal website is the only way allowable to  mess with code. Hope that helps !

    1. glendoncaba profile image81
      glendoncabaposted 6 years ago in reply to this

      I am no computer genius, but thanx for trying.

  3. mel22 profile image60
    mel22posted 6 years ago

    if its only about checking stats then you can set up URL channels to monitor.. this does not involve pasting code to source, just pasting URL to specific channel you want monitored within adsense itself. Can be ( overall hub pages) or an individual URL if checking on only one individual hub. Not sure if I'm answering correctly so i'll let sopmeone else take the question if this is not what you're referring to.

  4. glendoncaba profile image81
    glendoncabaposted 6 years ago

    I would post the blocked URL but dont want to run the risk of getting banned.

    My concern is that this is now my highest traffic hub (the one on the internet hoax) and I may not be maximising earnings.

    Weird URL.  Not the simple URL of article but some long weird stuff.