Getting the search engines to index all of your threads


I recall reading somewhere that something like 20% of the searches in Google are for phrases that have never been searched for before. It is one thing to target and SEO a forum (or any website) for one or a few popular keywords, it is another to get your site in a position to capture what has become known as the long tail part of the search keyword(s) traffic. This long tail is possibly a greater source of traffic than a popular keyword.

A forum is potentially well placed to capture these long tail searches due to the diversity of content that they have (all the forum topics, threads and posts). However, the first part of the challenge is to get as many of the threads and discussions indexed by the search engines, so they have a chance to rank for the long tail.

Here are some of the strategies that I use to get most, if not all threads indexed and ready to rank well for the long tail and unanticipated keyword(s). The key strategy is to get the search engine spiders all over the site, crawling deep into the site. How do you achieve that with a forum:

1. Backlinks. The more links to a site, the more often the search engines get to a site. While many like to dismiss Google’s PR as a ranking factor, there is a reasonable correlation between how often and how deep Google crawls a site with a sites PR. Link building in an important strategy to get more of a forum indexed. Web directories are part of this strategy.

2. Deep links. I get many links to every thread, by using Linkvana (see the post on treating each thread like a baby to be nurtured). Also don’t forget the potential of profiles to generate traffic. Also, the use of article submission to build deep links to forum threads should not be overlooked.

3. Sitemap.xml. I use the vBSEO site map generator to generate a daily sitemap to Google, Yahoo! and Bing.

4. Tags. These are often overlooked as a source of indexing; traffic; and their usefulness to members if you turn them into a good (eg) ‘Table of Contents’. Get deeplinks to the tags page and each individual tags page (this will help the search engine spiders get to the individual threads). Link to the tags or ‘Table of Contents’ from the top and not bottom nav bar, as its suggested the search engines pay less attention to the footer. More on the use of tags.

5. Archive. In vBulletin, I have this activated, but set to ‘no’ for the (eg) Display Simple View of Threads. This avoids the duplicate content issues and further helps the indexing of threads. This makes the archive function like a good old fashioned site map.

6. Site map. I still like the old style site map that has sort of been overtaken by the sitemap.xml. They still do help search engines, and more importantly, the user find their way around the site. In vBulletin, I use Logicians web templates to create individual pages and create a (eg) site map from it and link to it from the nav bar.

7. Statistics. I use Dreams Forum Statistics add on for my forums and link from the nav bar to the (eg) stats pages with the 50 most popular threads, most replied and most viewed threads on the forum. This is a good page for visitors, but for search engines it further helps the deep crawling of the site and the indexing of those threads. I also get deep links to this page wherever possible.

8. Related posts. I use two strategies here. In vBulletin, I have the Show Similar Threads activated, so there are some more links for the search engines to follow and crawl deep. I also have a moderator post as early as possible in a thread with a message with a post with links to other related threads and a link to the tags page. This serves a couple of purposes: it good for the site visitor, making the thread a more valuable resource; and it further helps the search engine crawler get around.

It all really comes down to making the site more accessible through multiple pathways for the search engine crawler to get all over the site. Most of the strategies are also useful to the site visitor.

, , ,

One Response to Getting the search engines to index all of your threads

  1. Hammer Electric March 15, 2011 at 12:50 PM #

    Thanks for this information – it was posted in the Google Webmaster’s forum by SEO101 and I will use some of these techniques.

Leave a Reply