At times at events, by email and just in casual conversations I may get asked about the changing of a website rankings in the Google search results. I often redirect these questions to an excellent document in the Google Webmaster Help Center titled “my site isn’t doing well in search” but I would like to take this opportunity to also share my thoughts on this topic on this personal blog.

Questions I tend to go like:

• My website is not ranking for my primary keyword anymore, why?
• I must have received a penalty because my website dropped in rankings, can you tell me why?
• Why is my website not ranking for keyword X while I was number one for years on this keyword?

Of course I consider it to be a positive activity to evaluate your website and see if you can be found and by which search queries you are found by your (potential) customers, however it is important to keep a few things in mind.

No Guaranteed Rankings

Yes, you read it right! There are no guaranteed rankings. Just because you were able to rank high for a certain keyword for a certain period, does not mean you are entitled to that position in the future. Just keep in mind that most major search engines will not give any website a fixed position but let their algorithms do all the hard work ;)

Internet is in Continuous Flux

Another thing to remember is that the Internet is growing every minute with possibly thousands, if not hundreds of thousands or more, new documents and media files. A large number of existing documents will also be updated or some may have been removed again. New links and other types of relationships are created or broken down between all these different new, changed and deleted documents.

As Googler Matthew Trewhella mentioned at his presentation at A4U Expo in London (October 2009), every time Google finishes a cycle of crawling the whole Internet, it may find from 10 percent to 25 percent of new content. This is huge! Let me illustrate this through an example: You are going to the cinema on a regular basis (e.g. once per week) and every time you find that the movies you attending have an additional 10 percent of new content. By the end of the year you could be sitting in the cinema for a whole day watching just one movie.

In addition, your competitors websites most probably have changed their websites since the last time you looked at them. Hopefully your website has also added new and compelling content. Even within the confines of your industry or niche, things are in constant motion. Google, like most search engines, will try to its best to accommodate all these changes by updating its algorithms, making sure new content is indexed quickly and update the rankings of every website globally based what its aims to be the best experience for its users.

So how can you check your rankings?

Luckily Google offers a great tool for webmasters that helps you understand how Google sees your website, called the Webmaster Tools. Here you can find an Top Search Queries overview for which search queries you rank best and which search queries result in the best click-through rate. In addition the Webmaster Tools also provides an overview to the most significant keywords that Google has associated with your website. If these are not reflect the subject matter of your website, you can identify this as an area for improvement.

In addition, a few articles were published in 2009 on how to use Google Analytics to check which position you were ranking for which search queries. Track SEO rankings with Google Analytics and Track SEO rankings and Sitelinks with Google Analytics II are some of the more interesting articles, although keep in mind that these may require you to have advanced knowledge of Google Analytics.

How can you improve your rankings?

The most important things for designing your website is to focus your attention on the users and making your website indexable/crawlable for search engines.

From a technical point of view you can analyse your website by -for example- making sure you don’t violate the webmaster guidelines, avoid duplicate content (e.g. use the canonical link element), structure your website using common sense, make your website fast. For more tips read the free SEO Starters Guide from Google.

From a quality point of view make sure that you focus your efforts on making your website a pleasant and useful experience for your users. Use tools like Google Analytics and Google Website Optimizer to optimize your website and create a better user experience.

Last but not least I would also suggest you create lots of compelling content and often publish new interesting compelling content so that your website can become an authority in your market/niche and there is a good reason for visitors to return to your website.

If after this whole story you still have questions about your rankings, I suggest you come to the Google Webmaster Help Forums where many helpful and friendly webmasters are willing to help answer your questions. Or you can choose to hire a professional SEO advisor, in which case I recommend you to read Google’s article on Search Engine Optimization.

The short answer is: “to help you visitors and search engine crawlers better understand the content of your website”. This may sound like common sense, however I do find myself often surprised how many websites forget this small yet so important part of website optimization.

Last week I was a speaker on the Google Site Clinic at the A4U Expo conference in London and during this session I highlighted a few times that the websites submitted for review were not using basic structuring elements to its full advantage. Based on the conversations I had at this conference, I wanted to elaborate on why I believe structuring your website well matters.

Imagine the following: You are in university and asked by your professor to write a 60-pages paper/report on a certain subject (note: it doesn’t really matter what the subject is). You will proceed with writing down all the chapters, paragraphs and resources. Now do you structure this? Of course you do! You will use a cover, a table of content, an index, etc. You do this to make the paper/report more readable for the reader, which is in this case your professor and your rewards may be a higher grade.

So coming back to websites, why you should do the same for your website? Keeping in mind that the reader will be, among others, visitors and search engine crawlers and that your rewards may be increased sales, trust as a authority and possibly even improved rankings in search engines.

Let me try to illustrate what I mean by comparing the structure of a paper with the desired structure of a website.

So when you are design a website, you should really keep the structuring of your website in mind. Not just because this will help search engines better understand the structure of your website (and you may rank better because of this), but improved structuring can lead to increased visitor satisfaction and sales/conversions. I recommend you to read the “SEO Starters Guide” by Google and “Don’t make me think” by Steve Krug and take some time to apply these best practices to your website.

I love to hear what other resources would you recommend. Please do share your comments below.

Update [Dec 2nd, 2009]

If you can read Dutch, you may also be interested in the following article: Kopjes gebruiken voor zoekmachine optimalisatie.

Since the latest release of Google App Engine SDK, Django 1.0 is officially supported by Google App Engine and in the article Unleash Django with app-engine-patch the advantages of using App Engine Patch (which allows you to use more Django functionality on Google App Engine) are well described. However the documentation of App Engine Patch is lacking some detailed description on how to set this up this locally. So I decided to write this small tutorial, in which I will guide you setting up Google App Engine SDK with App Engine Patch on an Ubuntu machine.

Why Ubuntu and not Windows?

For me this is a clear-cut choice. I have stepped away from Windows about a year ago after I tried to use Windows Vista Business edition for 8 months and was almost ready to throw the computer out of the building. I have used Windows most of the last 15 years I have been working on computers but in the last few years I saw that Linux is finally able to replace most, if not all, functionalities I require/need from an operating system. And since I also like the open source movement, for me switching to Linux made sense.

First the Debian distribution was my preferred choice but in the last 2 years Ubuntu has become really user friendly (offering a great user interface to Linux and frequent updates). In addition Linux operating system are especially great if you need a web development environment (although it may be challenging at times with the more graphical aspects of web development, like Flash or SilverLight). If you haven’t done recently already, I recommend you to give Linux a try! Please note that most of the steps described below can be applied to most other Linux distributions as well.

Before we start…

What you will need on Ubuntu for this tutorial are:

• A Terminal (which you can find under ‘Main menu > Applications > Accessories > Terminal’)
• A Google App Engine account (apply)
• Google App Engine SDK, Python version (download)
• App Engine Patch (download)
• Python 2.5 (which is available in the Ubuntu repository)

That is basically it, no more is needed :)

Setting Up a Google App Engine Account

In order to be able to use Google App Engine, we need to set up a free account. If you haven’t done this yet I recommend you do so by going to http://appengine.google.com/ and sign in with your Google Account (you will need to create one if you don’t have one yet).

Once logged in, you have to option to ‘Create an Application’ (see button at the bottom of the page). Click the button to create an application and fill in the form. Please note that you can not change the name (Application Identifier) after you have chosen it, nor can you delete it. Also currently you can only create a maximum of 10 applications, so I recommend you create just one for testing purposes (e.g. username-test).

After you are have created the application feel free to browse a little bit through the online admin interface. This is where you will be able to monitor the performance of your application once it has gone live.

Setting up Google App Engine

We can set up Google App Engine in several different ways. Most of the set up requires you to just download the SDK and unzip it. However for App Engine Patch we want to do go one step further. We want to register it in our PATH so we can access it from anywhere else on the computer.

In order to do all of this, first download the Google App Engine SDK (you need to go for the Linux download!) and unzip it into your home directory (one of the ways you can do this is by right-clicking on the zip-file and click on ‘Extract here’).

Unzipping the SDK will create a new sub directory called google_appengine, with the SDK files in it. Note that if all user information is stored under /home/<your-username>/ then after unzipping the SDK should be accessible here:

/home/<your-username>/google_appengine/

in my case this would be:

/home/fili/google_appengine/

Now we want to make sure we all the files have the correct access rights. We could go very specific into detail here and give different file-types different access permissions or we could just do it simple. I choose for the later, so please open a terminal (you can find this in your main menu, under Applications and Accessories). Once you open the terminal you should be in your home directory, under your user account. To verify this, type: pwd. This will print out the current path you are in (see also the screenshot below).

Now type ls and verify that you have the sub directory google_appengine there.

To make sure the access permissions are correct type the following command:

chmod -R 750 ~/google_appengine/

This will set all the permissions to something we can access. Now we need to make sure we can actually access the SDK from anywhere. In order to do this we need to add the sub directory to the environment PATH. In your terminal type:

pico ~/.bashrc

Don’t worry if the file does not exist yet or if it is full of stuff. Once you have opened the file, find the first or second white-line, and add the following lines:

# set path
PATH=$PATH:$HOME/google_appengine
export PATH

Now close and save this by clicking Control-O (this will save the new lines) and Control-X (this will close the nano editor). After this you need to log out of your Ubuntu session and log in again.

After you logged out and in again, you can verify this all worked by opening another terminal and typing:

echo $PATH

This should print something similar to the screenshot below (note the /home/fili/google_appengine at the end of the line.

The hardest part has been done now ;)

Setting Up Python2.5

You probably already have Python installed on your machine but this may be the wrong version. In order to work with App Engine Patch and Google App Engine SDK we need to make sure we all use the same version of Python, for which Python 2.5 is currently the best choice.

You can see which version is primary set up on your computer by opening a terminal and type python2.5. You should then get a screen like this (Control-D or typing exit() will get you out of this screen again):

If you get an error message that Python can not be found, it means that it is not yet installed.

If you don’t have Python 2.5 installed, you will need to install it. Make sure you have an internet connection and type:

sudo apt-get update

This make sure you have the latest repository lists available. Then type this:

sudo apt-get install python2.5

Select ‘Yes’ if asked if you want to proceed and this will install Python version 2.5 on your machine. After this process has been completed, try typing again python2.5.

Setting Up App Engine Patch

We are going to set up the App Engine Patch on our Desktop, but you can put it anywhere else on the computer.

First download the latest version of App Engine Patch to your Desktop and unzip it (similar as how you unzipped the Google App Engine SDK). A new sub directory will be created called ‘app-engine-patch-sample’. You can rename this sub directory to the name of your application (in my case: ‘fili-test’).

Before we can do anything else we need to change a one small settings in app.yaml. Open a terminal and type:

cd ~/Desktop/<your-application-name>/

For me this would translate to:

cd ~/Desktop/fili-test/

Now type:

pico app.yaml

Change the name on the first line to your application name and save the changes (Control-O) and exit the editor (Control-X):

Now your App Engine Patch example folder is associated with your Application Identifier. Now you can run the Google App Engine SDK with your App Engine Patch locally by typing (inside the fili-test subdirectory):

python2.5 manage.py runserver

This will start the local development server (Control-C will allow you to stop the development server again).

If you now open your browser on http://localhost:8000/ you can see the development server running:

That is it. You are done! Now you can start developing web applications using the powerful Django framework on Google App Engine. I recommend you to also browse through these resources about Django and Google App Engine:

• Google App Engine Articles
• Django (official homepage)
• Google App Engine SDK Getting Started Guide
• Google App Engine Documentation (Python)

If you noticed any inaccuracies/errors or you are missing any steps or are running into problems, do leave a comment below!

In the Google Webmaster Guidelines it states “avoid links to web spammers or ‘bad neighborhoods’ on the web, as your own ranking may be affected adversely by those links” and it is important to remember that this advice applies to both your forward and back-links. It is easy to control the forward-links of your website but it is harder to check what the back-links are and to explore how these may impact your search engine rankings.

In this article I will sum up a number of ways on how you can explore the back-links to your website. Please do note that just knowing all your back-links will not guarantee a good ranking but still it may yield some interesting findings.

Check your back-links as Google knows them

There are at least three ways how you can explore your back-links in Google:

Google Webmaster Tool (specifically the Links to your Site-section) gives the best and most complete overview on how Google sees your back-links. However it is important to note that only the webmaster/owner of a website who has been verified can see these back-links. For your convenience you can even download all the links in a spreadsheet format and process them on your own computer.

The Link-operator in Google Web Search (link:yourdomain.com) shows anyone a number of the back-links of any website. It is very important to note that this overview is just a sample from a much larger set of back-links. The Link-operator in Google Web Search will never give you the total overview, just a number of back-links and even the total of this number can change every time you submit your query to Google.

Since the Link-operator only gives you a limited view of back-links, several people have found that you get a better (and more complete) sample of back-links with the following query: “yourdomain.com” -site:yourdomain.com. So let me translate this query for you: the first part means you are looking for any reference to yourdomain.com (note that the quotes are important), and the second part means that you want to ignore any reference to yourdomain.com from the domain yourdomain.com (more information on the Site-operator and other advanced operators can be found here). If a SEO or competitor performs an analysis of the back-links to your website, and does not have access to your Google Webmaster Tool account, then this will be one of the queries they will perform.

Check your back-links as Yahoo knows them

At Yahoo Search you can the Yahoo Site Explorer and some advanced queries in Yahoo Web Search to explore your back-links (as Yahoo sees them).

The Yahoo Site Explorer will give you an overview of the back-links to your website (they refer to them as InLinks). It is important to note that you do not need to be verified as the webmaster/owner of the website to see the back-links. Because this is the opposite of how Google show the back-links of websites, Yahoo Site Explorer has become very popular to SEO’s for exploring back-link profiles of websites. In addition, Yahoo Site Explorer also offers you the opportunity to download the results in a spreadsheet format on to your computer.

Alternatively you can use the following queries on Yahoo Web Search to explore the back-links of yourdomain.com: linkdomain:yourdomain.com -site:yourdomain.com – the first part finds all links to yourdomain.com and the second part means that you want to ignore the links on the website yourdomain.com (more tips can be found in this article: More Effecitve Use of Yahoo’s Linkdomain Command). You can also combine this query with the Yahoo Link-operator and see which pages within yourdomain.com are being linked too linkdomain:yourdomain.com -site:yourdomain.com -link:http://yourdomain.com -link:http://www.yourdomain.com (the following article explain this further: Link search with Yahoo’s linkdomain).

Check your back-links as Bing knows them

At Bing (formerly known as Live and MSN Search) there are two ways of exploring your back-links:

The Bing Webmaster Tools provide the best and most complete overview of back-links to your website that are known by Bing. You can get an overview of the back-links and forward-links (they refer to them as Outbound Links) per website and you can also download the complete overview in a spreadsheet format (similar to Google and Yahoo).

Unfortunately Bing discontinued the support for the Link-operator in 2007 and so far we have not seen any signal yet that it will come back any time soon (although according to some SEO’s the link-operator still gives limited results). You can try using the query links to yourdomain.com, however this will give you only a partial overview of websites linking to yourdomain.com and it contains a lot of noise (websites that may just reference your website or contain keywords from your domain name without actually linking to your website). Alternatively, it seems that the following query may also work on Bing Web Search: “yourdomain.com” -site:yourdomain.com, and it seems to work similar as it does at Google (meaning limited results). Finally, also inbody:www.youdomain.com -site:yourdomain.com could yield some interesting results (but like the previous mentioned queries on Bing, it is a partial list at best). If you know a better way (you can try to find one with Bing’s advanced operators), please do share it in the comments below!

However I do want to note that Bing did introduce a neat forward-link-operator called ‘linkfromdomain’ (e.g. linkfromdomain:yourdomain.com) which can give you some additional information about, for example, the nature of the outgoing links. The following article sums up a few of the things you can do with this Use This Live / MSN Advanced Operator to Research Links-Out.

Get more out of Google and Yahoo webmaster tools

Once you start exploring your back-links at Google, Yahoo and/or Bing you may also want to check out a recently updated FireFox add-on, released by Joost de Valk, called SEO Link Analysis which will give you more information about each back-link within the Google Webmaster Tool and the Yahoo Site Explorer interface. For me the most useful feature of this add-on is that is tells me if a back-link is no-followed and what the anchor text is. I think that information can be very useful to see why you may be ranking for certain keywords but not for others. Please note that although this add-on gives you an idea of the PageRank value of each back-link, I would urge you to ignore this (as you do not wan to suffer from PageRank-obsession).

Using alternative databases: Majestic and Linkscape

Of course the search engines can give you a pretty good profile of your back-links, but sometimes you want more (e.g. features or information). So in the last few years a few alternatives have appeared on the market, namely Majestic and Linkscape. Both databases were built and are maintained differently and can provide some interesting information. However the information in these databases are not the same as in the databases of the search engines. If you are considering spending more money on your back-link-strategies and subscribe to these services, it would be prudent to first try them out during a trial period and see which you find the most useful. I would also recommend that you read the following article as it highlights a number of the pro/cons of both services: Linkscape vs Majestic.

Build your own tool and database

If you are a little bit technical you can also explore building your own tool and database by using some of the freely available API’s, for example: Yahoo offers the best API with the Site Explorer Inbound Links API and Bing can be accessed through the Bing Search API (if you know Python, you can use this wrapper). Unfortunately the Google Webmaster Tools API does not include access to any back-link information yet, but maybe this is yet come in the future (?) so keep an eye out for this.

Alternatively you can of course also import the different spreadsheet formats, offered by the different webmaster tools of the search engines, into a database of your own and do some comparison and data-mining on the given information. This may yield some interesting results as well ;)

Summary

It can be important for a webmaster to know who is linking to your website and there are several ways of exploring these back-links:

• You can see which back-links your websites has according to how Google, Yahoo and Bing knows them (using the webmaster tools and performing specific search queries).
• You can use alternative databases (keeping in mind that these is not the same databases the search engines are using).
• You can built your own tool and database using the API’s and spreadsheets provided by the search engines.

Anything I may have missed or called wrong? Please add your thoughts to the comments below …

Nowadays websites contain a lot of elements, like images, videos, CSS style sheets, Javascript (including AJAX), etc. And with HTML 5 being supported by the newer browsers, it seems that the number of elements in a single page is only likely to increase in the near future. With that in mind, it is important to look at the current design of websites. Most of which are not optimized on the front-end at all.

Steve Souders, writer of the excellent books ‘High Performance Web Sites‘ and the soon to come out ‘Even Faster Web Sites‘, is stating that users are primarily waiting for websites to load because of the lack of optimization on the front-end. With a few simple best practices and minimal code changes, your website can load super-fast for your users. Remember: speed does matter! I am not going to repeat everything that Steve has been promoting, instead I am including (below) some of Steve’s videos where he is sharing a lot of his knowledge..

At the end of 2007 Steve Souders left Yahoo and joined Google where he continued working on optimizing websites. And a few days ago, on June 3rd, Google published a new Firefox extension called Page Speed and made it available as open source to the developer community! I highly recommend you to check this out as this tool has some really cool new features!

I hope you find this tool as useful as I have found it overtime, and do leave me your thoughts in the comments below. Thanks!

Videos

And now, as promised the videos. Each will take time to watch fully, but if you are in website development the time invested in watching these videos will be absolutely worth it!

High Performance Web Sites and YSlow

Life’s Too Short – Write Fast Code (part 2)

Nicole Sullivan: Design Fast Websites

Update June 9th: Even Faster Websites

On Google I/O, Steve Souders also presented his findings on how to built even faster websites.