Blocking Google is too much work
by Randall McCarley
January 21st, 2008
Google is too big and is using anticompetitive measures to shape the internet - and all business - to their design. Either you believe that or you believe that they really do carry out their “do no evil” motto despite the countless reports to the contrary. At this point, I don’t see how I can change your mind (though I can probably guess your political party).
Anyway, last year I posted reasons why I banned Google from this site. Which is all good except…
It didn’t work.
Once Google has content in it’s index, its up to you to go to Google and tell them to take it out by removing your indexed pages one at a time, through their webmaster tools.
Also, Google has multiple bots for finding and indexing content so even though I blocked GoogleBot from pulling any new content from this site, I didn’t block Feedfetcher or the Adsense bots - though why Adsense was poking around I’ll never know.
Oh, and Googlebot itself has been poking around despite blocking it with the robots.txt. Ha ha.
SEOmoz has two recent posts about keeping content from the search engines starting with 12 ways to keep your content hidden from the search engines and a neat post on the robots exclusion protocol.
It seems to me to get a site out of Google it either has to be a complete failure so that Google doesn’t bother with it or the work is put on the site owner to keep pushing Google out and then follow up to remove anything Google decides to index anyway. Now I know why all those newspaper companies were so upset - it really is a lot of work.
Anyway, I have personal and professional obligations and fighting Google over content I always intended free access to just isn’t in the cards. I’m removing the line in the robots.txt that “excludes” (LOL) Google from indexing this site. And I may as well make some money so whenever I get around to it, the ads will come back.
It was an interesting experiment because Google referrals did not drop during this test!
As to the above mentioned obligations… My wife has an illness that is going to require surgery in the summer. She will be ok (ie. she won’t die or anything horrible like that). This will require a lot of my personal time and money. So I’m working as much as possible for the next few months to build a reserve. If you have work, please let me know - my rates are competitive. If you are a faithful follower of this blog, expect less. It’s probably best if you subscribe via email (see top of the right bar) to catch any irregular updates from here out.
Next Article: The most important issue in the presidential election is not the war in Iraq: It is net neutrality Previous Article: SEO Refugee flies new colors; new, faster server online
January 23rd, 2008 at 11:49 am
That is a very interesting article. Look forward to seeing if the experiment works.
January 24th, 2008 at 6:44 pm
Interesting that traffic didn’t stop during the experiment.
Randall I hope everything goes well with your wife and good luck in building the reserve.
January 27th, 2008 at 2:47 pm
Well Rand, you gave it your best. I find that admirable in itself… kinda like the guy who took on city hall. But unfortunately, you are correct; there’s just no way to reverse the self-imposed value they have on the webmaster and general internet using community at large.
If I were you, I would experiment more with their lack of observing server level commands to not index or not follow. That would make for a damn good revelation to put out in the public
January 28th, 2008 at 1:32 am
@ SEO Honolulu - Google’s reaction to nofollow in my experience is pretty interesting. First, the links do count for a few days-2 weeks as regular links so nofollow commands must be implemented on a supplementary basis. Second, anchor text also counts for 2 weeks or longer.
I discovered this when someone was yanking my chain and comment spammed some blogs with my name and unfriendly remarks in the anchor text.
Suddenly 14thC was getting hits for phrases like “randall mccarley is a XXXX”!
I had to follow the trail backward to see how it happened and was impressed once I figured it out. All the links were nofollowed and I got hits for those phrases for 2 weeks to a month.
January 31st, 2008 at 7:54 pm
Randall, I find Google is hit or miss with robots.txt. Some pages/folders it stays away from; some not.
I was doing some research for an article, and came across some good info for getting Google to not list pages at Webmasterworld:
http://www.webmasterworld.com/html/3409150.htm
Apparently, if you want Google to drop pages, the way to go is:
According to pageoneresults in the post above, using content=”none” results in Google not displaying the page, whereas robots.txt may still result in a URI-only listing in Google’s index.
Good luck with business, and take care of your wife.
January 31st, 2008 at 7:57 pm
Hmmm. I forgot that tags would get stripped out of my comment…
Anyway, “Apparently, if you want Google to drop pages, the way to go is:”…
meta name=”robots” content=”none”
February 1st, 2008 at 7:10 pm
That’s good info John, thanks. Again the issue comes up that *I* have to make site-wide changes to get/keep Google out.
June 27th, 2008 at 1:52 am
Well Randall McCarley I pray that you wife will be completely ok soon and try my best to promote this page so that more visitors come and visit this page.
Seconly the only way to remove page from google index is webmaster tool because robot.txt works but still google cralw it that i found in my webstat
Finally as one of my friend said that nofollow is still important though it is nofollow but always crawled by google.
Thanks Randall for your efforts.
will visit again