1. The analytics data is not available in real-time, you have to wait until midnight passes the next day to see yesterday’s data
2. The data is not yours – it’s Google’s – and for all of the clean image that Google has (”don’t be evil”), forgive me when I say that I prefer to have my own copy of my website trend and visitor data.
3. Google Analytics works by including a JavaScript snippet in your pages, but search engine spiders (”crawlers”) don’t execute JavaScript when they load your site’s pages, so their visits aren’t logged by Google Analytics. So when spiders from GoogleBot, Inktomi (Yahoo), Bing (Microsoft’s new search engine), Ask and Baidu crawl your site, you know absolutely zero about it from looking at your GA reports.

To track these visitors that in the vast majority of cases go totally unnoticed, you need a server-side scripting package that logs all visits to a database, including search engine spiders. There are a few free ones out there, and one of the best of these is CrawlTrack – freely available to download at http://www.crawltrack.net/

The install for CrawlTrack is reasonably straightforward, and can be completed in a few minutes. All you need to do is upload the CrawlTrack files to a new directory on your hosting account, create a new database on your hosting account’s MySQL server, include a piece of CrawlTrack PHP code in your website code (in a header file, or another appropriate file, as indicated by the CrawlTrack documentation) and go through the automated installation procedure.

CrawlTrack does not interfere with Google Analytics, you can run them side-by-side no problem, and enables you to see which spiders are crawling your site and how often. Here’s a screenshot of the “Crawlers” activity view from one of my new websites just taken today.

This data shows the percentage of visits from certain spiders. A table at the bottom shows how often each spider is crawling your site, broken down by each individual spider. Note: a “Visit” is one individual page load by the spider.

You can click on the name of the spider (e.g. GoogleBot) to drill down into detailed crawl stats for that specific robot. This view will give you individual crawl stats for each page of the site crawled, as well as stats into what percentage of your site pages were not crawled.

This is great data for SEO as it shows the regard by which the search engines are giving to your site. This is very useful if you are publishing new content on your site, or updating it, and you want to see when the new content is indexed by the search engines, therefore enabling you to monitor the effectiveness of site changes, e.g. you update your site with a special offer, and wait patiently for enquiries to come in. But the thing is that you don’t know when the search engines will index the new content. With CrawlTrack, you can see exactly when this new content is indexed, therefore giving you the ability to correlate the offer with the response more accurately.

With CrawlTrack, you will also be able to see trends in your website crawl data that indicates interest from the search engines because of SEO activities like inbound link-building.

Then comparing your CrawlTrack data with your Google Analytics data may give you interesting correlates.

All comments welcome!