In July of 2011 Twitter and Google ended an agreement, which previously gave Google access to crawl the content on this social media powerhouse. For more than a year now, Google and Twitter have been fighting over the power of indexing and crawling content. Is this battle finally over? Or did Twitter wait too long to make their content available for search engines to index?
Recently, Twitter updated its robots.txt file, which opened up millions of pages to be crawled by search engines. Representatives from Twitter have confirmed these changes, and claim that the reason behind this change is to better help users find popular pages.
It seems as if Twitter wants to reconnect with search engines. However, if this is the case, why has Twitter blocked search engine bots from indexing specific content, like users, videos and images? It seems as if Twitter is playing hard to get. As you can see below, Twitter is allowing search engines to crawl some of its content like tweets and hashtags, while blocking the rest:
For example, if you wanted to find a page about football, you would search #football, and pages about football will become available in search result.
Twitter is both revamping their search options, making it more efficient and allowing search engines to crawl and index some of their content. But do search engines like Bing and Google want anything to do with Twitter’s search results?
Obviously, Twitter being one of the biggest social media sites in the world, adds a lot of value when its pages are showing up on Google and other search engines’ index. Even with this said, there is still a big question in search engines’ minds on whether or not to even index Twitter’s search results. Over the years, Google has made it clear, that they have no intention in crawling or indexing search result pages on their own search results. Why would Twitter page results be any different? Bing still hasn’t really made clear what their philosophy is on indexing Twitter. I am sure we will have more answers from both Bing and Google in the near future about this whole Twitter debacle.
Twitter and search engines obviously have a lot of work to do on their relationship. I am almost positive that in the near future, all of these problems will be completely resolved. Twitter and the search engines will come up with some agreement that will allow powerful search engines like Google and Bing, to crawl and index all of the content on Twitter.
– Joey Giorgianni
If you are interested in more SEO techniques or would like to hire us to manage your reputation online contact us at: sales@getfoundquick.com or call 857.526.0096.
Boston Web Marketing | 215 Lewis Wharf Boston | MA 02110 | www.getfoundquick.com