1. The analytics data is not available in real-time, you have to wait until midnight passes the next day to see yesterday’s data
2. The data is not yours – it’s Google’s – and for all of the clean image that Google has (”don’t be evil”), forgive me when I say that I prefer to have my own copy of my website trend and visitor data.
3. Google Analytics works by including a JavaScript snippet in your pages, but search engine spiders (”crawlers”) don’t execute JavaScript when they load your site’s pages, so their visits aren’t logged by Google Analytics. So when spiders from GoogleBot, Inktomi (Yahoo), Bing (Microsoft’s new search engine), Ask and Baidu crawl your site, you know absolutely zero about it from looking at your GA reports.

To track these visitors that in the vast majority of cases go totally unnoticed, you need a server-side scripting package that logs all visits to a database, including search engine spiders. There are a few free ones out there, and one of the best of these is CrawlTrack – freely available to download at http://www.crawltrack.net/

The install for CrawlTrack is reasonably straightforward, and can be completed in a few minutes. All you need to do is upload the CrawlTrack files to a new directory on your hosting account, create a new database on your hosting account’s MySQL server, include a piece of CrawlTrack PHP code in your website code (in a header file, or another appropriate file, as indicated by the CrawlTrack documentation) and go through the automated installation procedure.

CrawlTrack does not interfere with Google Analytics, you can run them side-by-side no problem, and enables you to see which spiders are crawling your site and how often. Here’s a screenshot of the “Crawlers” activity view from one of my new websites just taken today.

This data shows the percentage of visits from certain spiders. A table at the bottom shows how often each spider is crawling your site, broken down by each individual spider. Note: a “Visit” is one individual page load by the spider.

You can click on the name of the spider (e.g. GoogleBot) to drill down into detailed crawl stats for that specific robot. This view will give you individual crawl stats for each page of the site crawled, as well as stats into what percentage of your site pages were not crawled.

This is great data for SEO as it shows the regard by which the search engines are giving to your site. This is very useful if you are publishing new content on your site, or updating it, and you want to see when the new content is indexed by the search engines, therefore enabling you to monitor the effectiveness of site changes, e.g. you update your site with a special offer, and wait patiently for enquiries to come in. But the thing is that you don’t know when the search engines will index the new content. With CrawlTrack, you can see exactly when this new content is indexed, therefore giving you the ability to correlate the offer with the response more accurately.

With CrawlTrack, you will also be able to see trends in your website crawl data that indicates interest from the search engines because of SEO activities like inbound link-building.

Then comparing your CrawlTrack data with your Google Analytics data may give you interesting correlates.

All comments welcome!

I spent a few hours doing the basic SEO work on this site, the same as I would do on any site we develop. At a minimum, the title and description tags are carefully written to target the desired keywords and of course the page content must be relevant and contain the keywords that the site is targeting also. There are of course, lots of other on-page SEO factors, and we took care of those also.  Within a week of launching the site, it is on the #1 spot of Google.ie for the keyphrase “dating coaching” (check out the screenshot below).

The keyphrase “dating coaching” does not have much competition in Ireland, so this was not such a difficult task.  The keyword “life coaching” is much more competitive so more effort will be required to get the site ranking for this keyword, more content is needed, and also a long-term manual inbound link-building campaign will be very useful in this regard.  “Dating Coaching” is a low-volume keyphrase (i.e. very few people actually search using those keywords in Ireland) but “Relationship Coaching” is much higher, and “Life Coaching” much higher volume again, so in this example the competition follows the search volume, which is usually (but not always) the case.

More updates coming soon, do subscribe and leave comments also, all appreciated.

So anyway, my friend mentioned to me that he wanted to get an AdWords campaign going in order to get some traffic to the site and generate more sales as business from the website have been slow. I remarked “ah, ok, well how many visitors are you getting to the site per day at the moment?” to which he looked at me perplexed and replied “don’t know!”. So we hopped up to the computer and logged into his Google Analytics account.

We were presented with the Google Analytics dashboard:

Ok, now correct me if I am wrong, but 1535 visits and over 7000 pageviews in the past month is not bad going, given that the site has not had any inbound link building done or any real SEO of any kind. So I suggested to my friend that we should first concentrate on getting more sales from the existing traffic before we go throwing money into Google AdWords when the conversion rate is abysmal.  

So what does he do to improve the conversion rate?  Well, we can see that most of the traffic is from the UK, coming in through the parked .co.uk domain. So the first thing my friend is going to do is to take a look at his pricing, taking a good look at his main competitor’s pricing for the products on the most trafficked product pages on his site. He can see which are his most viewed products by going to Content -> Top Content.  So an experiment would be to slash the prices of the top 5 or so viewed products and then see if sales on those products pick up.

But there is more that can be done of course, using Google Analytics more advanced features.  So, from a scientific metrics point of view, it will be useful for us, using his most trafficked product pages as a starting point, to use Google Analytics’ Goals and Funnels to chart the purchase path (the series of pages starting at the product page on through to the order “Thank You” page), and see using the Funnel Visualisation tool which pages along the way are causing the user to abort the purchase process. If most are aborting on the product page itself then either the page needs to be changed and re-tested, or the price needs to be reduced, or perhaps both.  Other pages along the way to the “Thank You” page may be leaky also. We will use the modify and re-test approach for these also until we are happy that the conversion rate is reasonably good. There is no limit to what you can do with the right tools and the commitment to using them.  We can also use a website optimisation tool like Google Website Optimizer to test variations of the pages on the site, analyse,  improve and re-test. 

But one thing is for sure, we can improve the conversion rate using Analytics data and tools. And this is the best approach at this point. Once we have the conversion rate improved to a satisfactory level we can then consider sending more traffic to the site with organic SEO methods and Google AdWords.  I’ll write an update post at a later date to tell you how we are faring on this project.  

Update: Part 2 of series of articles is available here http://webreach.ie/blog/checkout-goal-and-funnel-analytics-where-are-we-losing-sales-on-our-online-store/

Please feel free to leave comments if you have any feedback.