At times at events, by email and just in casual conversations I may get asked about the changing of a website rankings in the Google search results. I often redirect these questions to an excellent document in the Google Webmaster Help Center titled “my site isn’t doing well in search” but I would like to take this opportunity to also share my thoughts on this topic on this personal blog.

Questions I tend to go like:

• My website is not ranking for my primary keyword anymore, why?
• I must have received a penalty because my website dropped in rankings, can you tell me why?
• Why is my website not ranking for keyword X while I was number one for years on this keyword?

Of course I consider it to be a positive activity to evaluate your website and see if you can be found and by which search queries you are found by your (potential) customers, however it is important to keep a few things in mind.

No Guaranteed Rankings

Yes, you read it right! There are no guaranteed rankings. Just because you were able to rank high for a certain keyword for a certain period, does not mean you are entitled to that position in the future. Just keep in mind that most major search engines will not give any website a fixed position but let their algorithms do all the hard work ;)

Internet is in Continuous Flux

Another thing to remember is that the Internet is growing every minute with possibly thousands, if not hundreds of thousands or more, new documents and media files. A large number of existing documents will also be updated or some may have been removed again. New links and other types of relationships are created or broken down between all these different new, changed and deleted documents.

As Googler Matthew Trewhella mentioned at his presentation at A4U Expo in London (October 2009), every time Google finishes a cycle of crawling the whole Internet, it may find from 10 percent to 25 percent of new content. This is huge! Let me illustrate this through an example: You are going to the cinema on a regular basis (e.g. once per week) and every time you find that the movies you attending have an additional 10 percent of new content. By the end of the year you could be sitting in the cinema for a whole day watching just one movie.

In addition, your competitors websites most probably have changed their websites since the last time you looked at them. Hopefully your website has also added new and compelling content. Even within the confines of your industry or niche, things are in constant motion. Google, like most search engines, will try to its best to accommodate all these changes by updating its algorithms, making sure new content is indexed quickly and update the rankings of every website globally based what its aims to be the best experience for its users.

So how can you check your rankings?

Luckily Google offers a great tool for webmasters that helps you understand how Google sees your website, called the Webmaster Tools. Here you can find an Top Search Queries overview for which search queries you rank best and which search queries result in the best click-through rate. In addition the Webmaster Tools also provides an overview to the most significant keywords that Google has associated with your website. If these are not reflect the subject matter of your website, you can identify this as an area for improvement.

In addition, a few articles were published in 2009 on how to use Google Analytics to check which position you were ranking for which search queries. Track SEO rankings with Google Analytics and Track SEO rankings and Sitelinks with Google Analytics II are some of the more interesting articles, although keep in mind that these may require you to have advanced knowledge of Google Analytics.

How can you improve your rankings?

The most important things for designing your website is to focus your attention on the users and making your website indexable/crawlable for search engines.

From a technical point of view you can analyse your website by -for example- making sure you don’t violate the webmaster guidelines, avoid duplicate content (e.g. use the canonical link element), structure your website using common sense, make your website fast. For more tips read the free SEO Starters Guide from Google.

From a quality point of view make sure that you focus your efforts on making your website a pleasant and useful experience for your users. Use tools like Google Analytics and Google Website Optimizer to optimize your website and create a better user experience.

Last but not least I would also suggest you create lots of compelling content and often publish new interesting compelling content so that your website can become an authority in your market/niche and there is a good reason for visitors to return to your website.

If after this whole story you still have questions about your rankings, I suggest you come to the Google Webmaster Help Forums where many helpful and friendly webmasters are willing to help answer your questions. Or you can choose to hire a professional SEO advisor, in which case I recommend you to read Google’s article on Search Engine Optimization.

The short answer is: “to help you visitors and search engine crawlers better understand the content of your website”. This may sound like common sense, however I do find myself often surprised how many websites forget this small yet so important part of website optimization.

Last week I was a speaker on the Google Site Clinic at the A4U Expo conference in London and during this session I highlighted a few times that the websites submitted for review were not using basic structuring elements to its full advantage. Based on the conversations I had at this conference, I wanted to elaborate on why I believe structuring your website well matters.

Imagine the following: You are in university and asked by your professor to write a 60-pages paper/report on a certain subject (note: it doesn’t really matter what the subject is). You will proceed with writing down all the chapters, paragraphs and resources. Now do you structure this? Of course you do! You will use a cover, a table of content, an index, etc. You do this to make the paper/report more readable for the reader, which is in this case your professor and your rewards may be a higher grade.

So coming back to websites, why you should do the same for your website? Keeping in mind that the reader will be, among others, visitors and search engine crawlers and that your rewards may be increased sales, trust as a authority and possibly even improved rankings in search engines.

Let me try to illustrate what I mean by comparing the structure of a paper with the desired structure of a website.

So when you are design a website, you should really keep the structuring of your website in mind. Not just because this will help search engines better understand the structure of your website (and you may rank better because of this), but improved structuring can lead to increased visitor satisfaction and sales/conversions. I recommend you to read the “SEO Starters Guide” by Google and “Don’t make me think” by Steve Krug and take some time to apply these best practices to your website.

I love to hear what other resources would you recommend. Please do share your comments below.

Update [Dec 2nd, 2009]

If you can read Dutch, you may also be interested in the following article: Kopjes gebruiken voor zoekmachine optimalisatie.

In the Google Webmaster Guidelines it states “avoid links to web spammers or ‘bad neighborhoods’ on the web, as your own ranking may be affected adversely by those links” and it is important to remember that this advice applies to both your forward and back-links. It is easy to control the forward-links of your website but it is harder to check what the back-links are and to explore how these may impact your search engine rankings.

In this article I will sum up a number of ways on how you can explore the back-links to your website. Please do note that just knowing all your back-links will not guarantee a good ranking but still it may yield some interesting findings.

Check your back-links as Google knows them

There are at least three ways how you can explore your back-links in Google:

Google Webmaster Tool (specifically the Links to your Site-section) gives the best and most complete overview on how Google sees your back-links. However it is important to note that only the webmaster/owner of a website who has been verified can see these back-links. For your convenience you can even download all the links in a spreadsheet format and process them on your own computer.

The Link-operator in Google Web Search (link:yourdomain.com) shows anyone a number of the back-links of any website. It is very important to note that this overview is just a sample from a much larger set of back-links. The Link-operator in Google Web Search will never give you the total overview, just a number of back-links and even the total of this number can change every time you submit your query to Google.

Since the Link-operator only gives you a limited view of back-links, several people have found that you get a better (and more complete) sample of back-links with the following query: “yourdomain.com” -site:yourdomain.com. So let me translate this query for you: the first part means you are looking for any reference to yourdomain.com (note that the quotes are important), and the second part means that you want to ignore any reference to yourdomain.com from the domain yourdomain.com (more information on the Site-operator and other advanced operators can be found here). If a SEO or competitor performs an analysis of the back-links to your website, and does not have access to your Google Webmaster Tool account, then this will be one of the queries they will perform.

Check your back-links as Yahoo knows them

At Yahoo Search you can the Yahoo Site Explorer and some advanced queries in Yahoo Web Search to explore your back-links (as Yahoo sees them).

The Yahoo Site Explorer will give you an overview of the back-links to your website (they refer to them as InLinks). It is important to note that you do not need to be verified as the webmaster/owner of the website to see the back-links. Because this is the opposite of how Google show the back-links of websites, Yahoo Site Explorer has become very popular to SEO’s for exploring back-link profiles of websites. In addition, Yahoo Site Explorer also offers you the opportunity to download the results in a spreadsheet format on to your computer.

Alternatively you can use the following queries on Yahoo Web Search to explore the back-links of yourdomain.com: linkdomain:yourdomain.com -site:yourdomain.com – the first part finds all links to yourdomain.com and the second part means that you want to ignore the links on the website yourdomain.com (more tips can be found in this article: More Effecitve Use of Yahoo’s Linkdomain Command). You can also combine this query with the Yahoo Link-operator and see which pages within yourdomain.com are being linked too linkdomain:yourdomain.com -site:yourdomain.com -link:http://yourdomain.com -link:http://www.yourdomain.com (the following article explain this further: Link search with Yahoo’s linkdomain).

Check your back-links as Bing knows them

At Bing (formerly known as Live and MSN Search) there are two ways of exploring your back-links:

The Bing Webmaster Tools provide the best and most complete overview of back-links to your website that are known by Bing. You can get an overview of the back-links and forward-links (they refer to them as Outbound Links) per website and you can also download the complete overview in a spreadsheet format (similar to Google and Yahoo).

Unfortunately Bing discontinued the support for the Link-operator in 2007 and so far we have not seen any signal yet that it will come back any time soon (although according to some SEO’s the link-operator still gives limited results). You can try using the query links to yourdomain.com, however this will give you only a partial overview of websites linking to yourdomain.com and it contains a lot of noise (websites that may just reference your website or contain keywords from your domain name without actually linking to your website). Alternatively, it seems that the following query may also work on Bing Web Search: “yourdomain.com” -site:yourdomain.com, and it seems to work similar as it does at Google (meaning limited results). Finally, also inbody:www.youdomain.com -site:yourdomain.com could yield some interesting results (but like the previous mentioned queries on Bing, it is a partial list at best). If you know a better way (you can try to find one with Bing’s advanced operators), please do share it in the comments below!

However I do want to note that Bing did introduce a neat forward-link-operator called ‘linkfromdomain’ (e.g. linkfromdomain:yourdomain.com) which can give you some additional information about, for example, the nature of the outgoing links. The following article sums up a few of the things you can do with this Use This Live / MSN Advanced Operator to Research Links-Out.

Get more out of Google and Yahoo webmaster tools

Once you start exploring your back-links at Google, Yahoo and/or Bing you may also want to check out a recently updated FireFox add-on, released by Joost de Valk, called SEO Link Analysis which will give you more information about each back-link within the Google Webmaster Tool and the Yahoo Site Explorer interface. For me the most useful feature of this add-on is that is tells me if a back-link is no-followed and what the anchor text is. I think that information can be very useful to see why you may be ranking for certain keywords but not for others. Please note that although this add-on gives you an idea of the PageRank value of each back-link, I would urge you to ignore this (as you do not wan to suffer from PageRank-obsession).

Using alternative databases: Majestic and Linkscape

Of course the search engines can give you a pretty good profile of your back-links, but sometimes you want more (e.g. features or information). So in the last few years a few alternatives have appeared on the market, namely Majestic and Linkscape. Both databases were built and are maintained differently and can provide some interesting information. However the information in these databases are not the same as in the databases of the search engines. If you are considering spending more money on your back-link-strategies and subscribe to these services, it would be prudent to first try them out during a trial period and see which you find the most useful. I would also recommend that you read the following article as it highlights a number of the pro/cons of both services: Linkscape vs Majestic.

Build your own tool and database

If you are a little bit technical you can also explore building your own tool and database by using some of the freely available API’s, for example: Yahoo offers the best API with the Site Explorer Inbound Links API and Bing can be accessed through the Bing Search API (if you know Python, you can use this wrapper). Unfortunately the Google Webmaster Tools API does not include access to any back-link information yet, but maybe this is yet come in the future (?) so keep an eye out for this.

Alternatively you can of course also import the different spreadsheet formats, offered by the different webmaster tools of the search engines, into a database of your own and do some comparison and data-mining on the given information. This may yield some interesting results as well ;)

Summary

It can be important for a webmaster to know who is linking to your website and there are several ways of exploring these back-links:

• You can see which back-links your websites has according to how Google, Yahoo and Bing knows them (using the webmaster tools and performing specific search queries).
• You can use alternative databases (keeping in mind that these is not the same databases the search engines are using).
• You can built your own tool and database using the API’s and spreadsheets provided by the search engines.

Anything I may have missed or called wrong? Please add your thoughts to the comments below …