I love to optimize my WordPress-based blog. The only problem is, I rarely have enough time to do it — and still, there are some small improvements, which may take less than 5 minutes of your time, and yet have a tangible impact on your overall blog optimization.
One of these things is how we can prevent Google (and other search engines) from indexing (searching) the WordPress RSS feeds.
The next few lines will be dedicated to this problem (and how we can solve it).
Where to start?
I remember that some time ago I was checking which pages of optimiced.com are indexed in Google.
I was puzzled by the fact that, beside the blog posts, I have found a lot of RSS feeds, which were also indexed.
Why you do not need Google to index/spider the RSS feeds?
First of all, the indexed (searched) content is duplicated – the last 10 posts or the last comments, available via RSS, can be read on the blog itself. Second point, RSS is meant to be used with an RSS reader, not to be read in the browser window (text and images won’t be formatted, for example). Last, but not least, who would like after a performed internet search to land on a un-formatted RSS page with comments, for example, instead of on the blog post itself, to which the comments are related? And this happened to me, and more than once…
(Example: you can use this link to subscribe to the RSS feed of my blog, or just to check the ten last blog posts from optimiced in RSS format.)
Can we prevent this from happening?
I searched the Internet for some time, until finally I dropped on the WordPress Support forum, where the solution was found, and the thread itself, titled “Prevent indexing of feed pages”, was marked as ‘resolved’.
Here’s the way to do it – you must use a robots.txt
file.
What is robots.txt
?
As the name itself suggests, robots.txt
* is a text file in the standard text format (.TXT), intended to use by robots:-)
But not all robots, of course (for example, Roomba doesn’t count;-), but only by the search machines (spiders), like Google, Live Search (until recently MSN Search), Yahoo!, Alta Vista and all other search (ro)bots.
Read more