Clicks and Heatmap
In this post, I explain how I implemented Cannoli, a click heatmaps on top of Ruwa.
http://github.com/rosario/cannoli
Many thanks to David, the heatmap code is taken and modified from his blog. Without his contribution it wouldn’t have been possible.
Click heatmaps are really useful to visualize information that standard analytics tools may hide. Heatmaps allow you to visualize exactly where users are clicking. You can see which parts of the website are not used, and improve those parts used more.
Here’s an example of an click heatmap:
Clicks information can be stored in a different server (as you can see that’s a blog in blogpost), once the administrator is logged in, a small clip on the bottom left corner will show the heatmap when clicked.
Ruwa
Ruwa is a web analytics tool written in Ruby on Rails 2.2 and it is still under development. Ruwa tracks visitors, stores browser information and handles the connection between visited pages and visitors. The heatmap code depends on Ruwa, however it is easy to be ported to other web analytics like Piwik. Ruwa tracking code is the same as piwik:
<script src='http://yourserver.name/javascripts/piwik.js' type='text/javascript'></script>
<script type='text/javascript'>
pkBaseURL = "http://yourserver.name/javascripts/"
piwik_action_name = document.location.href;
piwik_idsite = IDSITE;
piwik_url = pkBaseURL + "log.js";
action_kind = 0;
piwik_log(piwik_action_name, piwik_idsite, piwik_url, action_kind);
</script>
<script src='http://yourserver.name/javascripts/clickmap.js' type='text/javascript'></script>This javascript code needs to go into every page. The function piwik.js collects visitor’s information and sends it to a log.js action. The IDSITE is important and it is assigned by Ruwa when a project is created. Every project has a project_id, and a project_name. The name of the project must be the same of the host where we want to run the tracking code.
We also need to add a div tag on the page. This div will be a reference point for all the clicks. To mark this point we will use this div tag: <div id="prova"></div>. This is useful when you use a centered layout in your page. If you are not sure where to put this div, you can put it right after the <body> tag.
Let’s have a look at the clickmap action:
def clickmap
if session[:user_id]
a = Action.find(session[:action_id])
name = a.url_id
heatmap = '#{SERVER_NAME}/images/#{name}.png'
overlay_style = "<style type='text/css'>@import url'#{SERVER_NAME}/stylesheets/overlay.css');</style>"
render :action => "overlay.js"
else
@click_url = "#{SERVER_NAME}/javascripts/click.js"
render :action => "clickmap.js"
end
endThis action renders two different javascript code snippets depending on the session[:user_id] variable. The variable session[:user_id] is set when the admin is logged. This way it’s possible to show the heatmap or save clicks having the same javascript code installed in a web page.
Collecting clicks and Cross site scripting
The file clickmap.js contains the javascript responsible for collecting clicks and sending this information to a remote server (but you can also install Ruwa in your own server). Let’s have a look at this two javascript functions included clickmap.js:
function getMouseXY(e) {
calculate_offsets()
if (IE) {
tempX = event.clientX document.body.scrollLeft
tempY = event.clientY document.body.scrollTop
} else {
tempX = e.pageX
tempY = e.pageY
}
tempX-=xOffset;
tempY-=yOffset;
if ((tempX >= 0) && (tempY >=0)) {
var url = '<%= @click_url %>?x='tempX"&y="+tempY;
saveclick(url);
}
return true;
}
function saveclick(url) {
var headID = document.getElementsByTagName("head")[0];
var newScript = document.createElement('script');
newScript.type = 'text/javascript';
newScript.src = url;
headID.appendChild(newScript);
}The function calculate_offsets will look for the reference point and it will setup the right offsets. The function getMouseXY calculates the position of the clicks. The variable @click_url holds the url of a server (it can be a remote server). We are going to send the click position to that server. For example, the url could look like this:
http://codynamix.com/javascripts/click.js?x=200&y=150
This means we are going to use this url to send the click position to codynamix.com. How? Well, we need to use a trick. The function saveclick creates a script inside the webpage. We set the src of the script to that url. This is a very simple way to achieve a sort of cross-site scripting.
Storing the clicks
On the server side, using Rails conventions, we are calling the click action. One option would be to store everything in a single file:
class JavascriptsController < ApplicationController
def click
x = params[:x]
y = params[:y]
myfile = File.new("/clicks.txt", "a+")
myfile.puts x + " " y
myfile.close
end
endHowever, using Ruwa we get more flexibility, and save other information such as the name of webpage where the click was done. We could also track the clicks of a particular visitor and so on. The click action is defined:
class JavascriptsController < ApplicationController
def click
a = session[:action_id]
v = session[:visitor_id]
c = Click.create(:action_id => a, :visitor_id => v, :x => params[:x], :y => params[:y])
render :text => ""
end
end
Ruwa provides two variables, the first is the action_id associated with the webpage we’re tracking, the second variable is visitor_id, that corresponds to a the visitor browsing that page. We simply create an entry that stores the relationship between the visitor, the webpage visited and coordinates of the click.
Basics of a Heatmap generation
I assume you have RMagick, we need it to create the heatmap. Each page has got a set of clicks. In this implementation clicks are stored in a Click table:
create_table "clicks", :force => true do |t|
t.integer "action_id"
t.integer "visitor_id"
t.integer "x"
t.integer "y"
t.datetime "created_at"
t.datetime "updated_at"
end
The ReadClick class is responsible of storing the clicks in a suitable way so that they can be processed. If you want to read clicks from a file you want to modify this code:
class ReadClicks
def initialize(cs)
@data = []
for c in cs
@data.push(Point.new(c.x, c.y))
end
end
def coords
xMax=0
yMax=0
coords = Array.new
@data.each do |line|
coords.push(line)
xMax=line.x if line.x>xMax
yMax=line.y if line.y>yMax
end
return Log.new(xMax,yMax,coords)
end
endThe clickmap image will be generated in the Image class:
class Image
def self.graymap(points)
file = ReadClicks.new(points)
pagedata = file.coords
# Create click-dot, colorized
intensity = (100-(100/pagedata.reps))/100.to_f
click_image =Magick::Image.read(DOTIMAGE).first.colorize(intensity,intensity,intensity,'white')
# Create overlay image
halfwidth = DOTWIDTH/2
image = Magick::Image.new(pagedata.xhalfwidth, pagedata.yhalfwidth) { self.background_color = 'transparent'}
pagedata.list.each do |coords|
image.composite!(click_image,coords.x-halfwidth,coords.y-halfwidth, Magick::MultiplyCompositeOp)
end
image = image.negate
return image
end
endThe graymap function reads the clicks and return a graymap. This is useful because it store the intensity of each points. The next step is to color the graymap:
class Image
def self.create_heatmap(heatmap_file,points)
if points.empty?
system("touch #{heatmap_file}")
else
image = graymap(points)
colorimage = Magick::Image.read(COLORIMAGE).first
imagelist = Magick::ImageList.new
imagelist << image.clut_channel(colorimage)
imagelist.fx("A*0.8",Magick::AlphaChannel).blur_image.write(heatmap_file)
end
end
endThe heatmap can be now created. To create a heatmap for a particular action:
class Action < ActiveRecord::Base
def create_heatmap
heatmap_file = "#{RAILS_ROOT}/public/images/#{url_id}.png"
Heatmap::Image.create_heatmap(heatmap_file,clicks)
end Improvements
The heatmap code is taken from David. I made few changes to it and I rewrote some parts in RMagick. However, I did not see any sensible improvements. A faster solution is to write some C++ code and directly manipulate the PNG images, I’ll show how to do that in the next blog post.
Overlay the heatmap
At this point we can track clicks from a webpage and send this information to our remote server. We can also generate a heatmap for each page visited. We now have to find a way to overlay the heatmap over the page.
The heatmap will appear on top of the webpage where the tracking script is installed and only the administrator is allowed to see the heatmap. Every time the administrator is logged into Ruwa, the session[:user_id] variable is set.
Remember that the clickmap action is this:
def clickmap
if session[:user_id]
a = Action.find(session[:action_id])
name = a.url_id
heatmap = '#{SERVER_NAME}/images/#{name}.png'
overlay_style = "<style type='text/css'>@import url'#{SERVER_NAME}/stylesheets/overlay.css');</style>"
render :action => "overlay.js"
else
@click_url = "#{SERVER_NAME}/javascripts/click.js"
render :action => "clickmap.js"
end
endWhen the user is logged in, we will render overlay.js. Let’s have a look at same part of it, (you can find the complete source code, on github):
heatmap: function( tab_options) {
document.write("<%= @overlay_style %>");
var hstyle = this.heatmap_style()
this.tab_options = {};
this.tab_options.color = '#4368ff';
this.tab_html = '<a href="#" id="clip" style="background-color:'this.tab_options.color'">HEATMAP</a>';
this.overlay_html = '<div id="heatmap_panel" style="display:none" onclick="DISPLAY.hide();return false">' '<img src="<%= @heatmap %>" style="'hstyle'" >'
'<div id="gray_panel"></div>' + '</div>';
document.write(this.tab_html);
document.write(this.overlay_html);
this.gId('clip').onclick = function() { DISPLAY.show(); return false; }
},
set_position: function() {
this.scroll_top = document.documentElement.scrollTop || document.body.scrollTop;
this.scroll_height = document.documentElement.scrollHeight;
this.client_height = window.innerHeight || document.documentElement.clientHeight;
this.gId('gray_panel').style.height = this.scroll_height+"px";
},
show: function() {
this.set_position();
this.gId('heatmap_panel').style.display = "block";
},
hide: function() {
this.gId('heatmap_panel').style.display = "none";
},
gId: function(id) {
return document.getElementById(id);
}
document.write('<script type="text/javascript" charset="utf-8">DISPLAY.heatmap({});</script>');The function heatmap overlays the heatmap image, it uses heatmap_style to create a CSS with the correct position. The <div id="prova"\></div> reference point is used for that again. We show a little clip on the left side of the webpage, once we click on it a gray transparent background appears together with the heatmap. The @heatmap image path is set in the clickmap action.
Conclusions
You should have a working click heatmap service installed. If you are interested you’re welcome to clone the repository and hack around, I’ll be happy to include any improvement.
You can download the complete code here:
http://github.com/rosario/cannoli
