Update (8 May 2011): I revisited this a year later to make a version that you can try online with your own Latitude data
Update (15 Jan 2012): A fixed version of the code to handle the new Google Latitude file format.
Google Latitude is starting to get very interesting. The new dashboard lets you see some graphs of how much time you spend at work, home, and out and about, and a list of your most visited places.
You can also see a Google Map with your 500 latest updates added as pushpins.
I had a random idea while looking at it this evening – why don’t they let you see all your updates on a map, in a heatmap that shows where you’ve been?
Naturally, once I had the idea, I had to give it a quick try.
From the Google Latitude dashboard, you can export your history of location updates as a KML file. I downloaded my history, and wrote a short, hacky Python script to parse it, and generate a heat map to overlay on a Google map.
In this post, I’ll show the sorts of results it can generate, and share my script, in case any other Latitude users fancy giving it a go.
I’ve been using Google Latitude for several months now, so I’ve got a reasonable chunk of data to play with.
If I draw smaller points on a map, it’s easier to see the individual points:
I’ve made the script configurable – the images in this flickr set show the effects of tweaking some of these parameters.
For example, playing with colours, the size of the dots added to the map, the opacity of the generated image, and so on.
I also added a parameter to let you group locations that are near each other into single groups – it makes a difference to the type of heat map you get out of it.
Another of the things I tried was making the density of marks on the heatmap proportional to the time spent at a location, rather than the number of updates made there (by comparing the timestamps of subsequent updates, and assuming that the time in between was spent at that location).
I wasn’t sure whether the number of updates at a location is proportional to the time I spent there – so it’s another thing to play with.
It’s interesting to see the clusters of marks around places that I regularly go to – such as points on the morning school run, or home and work. I’m not claiming that my script is 100% reliable or accurate, but on the whole the results look pretty believable 🙂
Here is my script – it’s a quick-and-dirty hack, and far from optimal. But it’s a fun snippet to play with.
################################################################################
# Google Latitude Heat Map
#
# Dale Lane (dale.lane@gmail.com)
# 28 May 2010
################################################################################
# needs:
# heatmap - from http://jjguy.com/heatmap/
# KML file exported from Google Latitude
################################################################################
# two parts to the script:
# 1) parsing KML downloaded from http://www.google.com/latitude/apps/history/view
# 2) adding lat/lon points extracted from KML to a heatmap
################################################################################
# warning and disclaimer:
# Provided as-is.
# This was a very quick bit of fun on a quiet evening - it's almost certainly
# got a ton of bugs and misunderstandings in it. Take it's results with a
# pinch of salt!
################################################################################
# imports
import xml.parsers.expat
import heatmap
################################################################################
#
# VALUES TO TWEAK THE BEHAVIOUR OF THE SCRIPT
#
################################################################################
LATLON_ACCURACY = 6 # how many decimal places should we round the lat/lon
# values to?
# reduce this to group similar locations together into
# a single larger group
BOUNDS_LAT_MAX = 90 # range of latitude to include
# set to 90 to get everything
BOUNDS_LAT_MIN = -90 # range of latitude to include
# set to -90 to get everything
BOUNDS_LON_MAX = 180 # range of longitude to include
# set to 180 to get everything
BOUNDS_LON_MIN = -180 # range of longitude to include
# set to -180 to get everything
# for example, in some of my maps, I set the bounds to draw a map of
# locations in and around Eastleigh
# BOUNDS_LAT_MAX = 51.033190218589546
# BOUNDS_LAT_MIN = 50.95929172950454
# BOUNDS_LON_MAX = -1.3039398193359375
# BOUNDS_LON_MIN = -1.4261627197265625
ADJUST_DENSITY_FOR_TIME = False # False - we add every location to the map once
# for every time it was recorded by
# Google Latitude
# True - we add each location to the map a number
# of times that is proportional to the
# total amount of time spent there
DENSITY_ADJUST_HOURS = 12 # if ADJUST_DENSITY_FOR_TIME is True, then we will
# add each location to the heatmap once for this
# number of hours spent at the location
# e.g. if this number is 12, we will add the
# location to the heatmap once for every 12 hours
# spent there in total
HEATMAP_DOT_SIZE = 2 # how large should points plotted on the heatmap be?
# value is in pixels
# 2 is the smallest possible value
HEATMAP_DOT_OPACITY = 210 # do you want to be able to see the map through the
# heatmap overlay?
# change opacity between 0 and 255 to adjust
GOOGLE_LATITUDE_FILE = "kml.kml" # file name of KML from Google Latitude
################################################################################
# globals
################################################################################
# points we will plot on the heat map
pts = []
class MyKMLParser:
# which XML element are we on?
currentelement = None
# the timestamp (in milliseconds since epoch) of the last placemark
previoustime = None
# the timestamp (in milliseconds since epoch) of the current placemark
latesttime = None
# the latitude of the current placemark
latestlat = None
# the longitude of the current placemark
latestlon = None
# two-dimensional array of time spent at locations
# timespent[lat][lon] = totalTimeInMilliseconds
timespent = {}
# XML parser function - StartElementHandler
def start_element(self, name, attrs):
latestelement = str(name)
if latestelement == "Data":
# there are multiple data elements - it's the 'name' attribute which
# identifies these element
self.currentelement = attrs["name"]
elif latestelement != "value":
# value elements are always within a uniquely named element, so
# we ignore the 'value' tag - it's the parent that tells us what
# this
self.currentelement = latestelement
# XML parser function - EndElementHandler
def end_element(self, name):
global ADJUST_DENSITY_FOR_TIME, pts
if name == "Placemark":
if self.latestlat != None and self.latestlon != None:
if ADJUST_DENSITY_FOR_TIME == True:
#
# calculate the time difference between this placemark
# and the previous one
#
# default to 10 minutes - for no particularly good reason
valueMilliseconds = 10 * 60 * 1000
if self.previoustime != None:
valueMilliseconds = self.previoustime - self.latesttime
self.previoustime = self.latesttime
#
# store the time spent in this location
#
if self.latestlat not in self.timespent:
self.timespent[self.latestlat] = {}
if self.latestlon not in self.timespent[self.latestlat]:
self.timespent[self.latestlat][self.latestlon] = 0
self.timespent[self.latestlat][self.latestlon] += valueMilliseconds
else:
#
# plot this point - we don't care how about how long was
# spent here
#
pts.append((self.latestlon, self.latestlat))
if name != "Data":
self.currentelement = ""
# XML parser function - CharacterDataHandler
def char_data(self, data):
global LATLON_ACCURACY, BOUNDS_LAT_MAX, BOUNDS_LAT_MIN, BOUNDS_LON_MAX, BOUNDS_LON_MIN
if self.currentelement == "coordinates":
#
# location data - looks like -1.355025,50.973697,0
# so we split it on commas, and take the first and second value
#
# initialise values
self.latestlat = None
self.latestlon = None
tmplatestcoordinates = str(data)
coordbits = tmplatestcoordinates.split(',')
try:
if len(coordbits) == 3:
lat = float(coordbits[1])
lon = float(coordbits[0])
if lon < = BOUNDS_LON_MAX and lon >= BOUNDS_LON_MIN and lat < = BOUNDS_LAT_MAX and lat >= BOUNDS_LAT_MIN:
self.latestlat = round(lat, LATLON_ACCURACY)
self.latestlon = round(lon, LATLON_ACCURACY)
except:
noop = 0
elif self.currentelement == "timestamp":
#
# timestamp - looks like 1274989863180
# the value is in milliseconds since the epoch
#
try:
newlatesttime = int(data)
# sanity check - is it in the range I expect?
# these are hard-coded values for the times I know that I've
# been using latitude
if newlatesttime > 1254355200000 and newlatesttime < 1275043497000:
self.latesttime = newlatesttime
except Exception, exc:
noop = 0
################################################################################
#
# parse the KML
#
################################################################################
parser = MyKMLParser()
p = xml.parsers.expat.ParserCreate()
p.StartElementHandler = parser.start_element
p.EndElementHandler = parser.end_element
p.CharacterDataHandler = parser.char_data
p.ParseFile(open(GOOGLE_LATITUDE_FILE))
################################################################################
#
# draw the extracted points on a heat map
#
################################################################################
# if we are adjusting the density based on how long was spent at locations,
# we need to do this now
# if not, we added points to the heat-map store as we parsed the KML, so there
# is nothing else to do
if ADJUST_DENSITY_FOR_TIME == True:
# collect the points to draw
for lat in parser.timespent:
for lon in parser.timespent[lat]:
# how long did we spend at this location?
# so that the heat map reflects the time spent at locations, we
# add a point multiple times
timeSpent = parser.timespent[lat][lon] / (1000 * 60 * 60 * DENSITY_ADJUST_HOURS)
if timeSpent > 0:
for n in range(timeSpent):
pts.append((lon, lat))
else:
pts.append((lon, lat))
print str(len(pts)) + " points to plot"
# draw the heatmap using the awesome http://jjguy.com/heatmap/
hm = heatmap.Heatmap()
hm.heatmap(pts,
"google-latitude-heatmap.png",
scheme='pbj',
opacity=HEATMAP_DOT_OPACITY,
dotsize=HEATMAP_DOT_SIZE)
hm.saveKML("google-latitude-heatmap.kml")
Finally, I have to say a big thanks to the author of the library I used to generate the heatmaps – which is the only way I was able to hack this together in an hour or so.
.
Update (15-Jan-2012): Since writing this post, Google changed the format of the files exported by Latitude. I didn’t have a chance to rewrite my code to work with the new file format, but Daniel Drucker fixed it to make his heatmaps and was kind enough to share his version of the code with me:
#!/usr/bin/python
import time
LATLON_ACCURACY = 8 # how many decimal places should we round the lat/lon
# values to?
# reduce this to group similar locations together into
# a single larger group
BOUNDS_LAT_MAX = 90 # range of latitude to include
# set to 90 to get everything
BOUNDS_LAT_MIN = -90 # range of latitude to include
# set to -90 to get everything
BOUNDS_LON_MAX = 180 # range of longitude to include
# set to 180 to get everything
BOUNDS_LON_MIN = -180 # range of longitude to include
# set to -180 to get everything
ADJUST_DENSITY_FOR_TIME = True # False - we add every location to the map once
# for every time it was recorded by
# Google Latitude
# True - we add each location to the map a number
# of times that is proportional to the
# total amount of time spent there
DENSITY_ADJUST_HOURS = 24 # if ADJUST_DENSITY_FOR_TIME is True, then we will
# add each location to the heatmap once for this
# number of hours spent at the location
# e.g. if this number is 12, we will add the
# location to the heatmap once for every 12 hours
# spent there in total
GOOGLE_LATITUDE_FILE = "kml" # file name of KML from Google Latitude
################################################################################
# globals
################################################################################
# points we will plot on the heat map
pts = []
previoustime = None
latesttime = None
latestlat = None
latestlon = None
# two-dimensional array of time spent at locations
# timespent[lat][lon] = totalTimeInMilliseconds
timespent = {}
i=0
datalines=[]
for line in open(GOOGLE_LATITUDE_FILE,'r'):
if 'gx:coord' in line:
(lon,lat,x) = line.replace('<gx:coord>','').split(' ')
lat = round(float(lat), LATLON_ACCURACY)
lon = round(float(lon), LATLON_ACCURACY)
i += 1
if 'when' in line:
t = time.mktime(time.strptime(line[6:25],'%Y-%m-%dT%H:%M:%S'))
i += 1
if i==2:
datalines.append(dict(lat=lat,lon=lon,t=t))
i=0
for thisone in datalines:
latestlat = thisone['lat']
latestlon = thisone['lon']
latesttime = thisone['t']
if ADJUST_DENSITY_FOR_TIME == True:
#
# calculate the time difference between this placemark
# and the previous one
#
# default to 10 minutes - for no particularly good reason
valueMilliseconds = 10 * 60 * 1000
if previoustime != None:
valueMilliseconds = previoustime - latesttime
previoustime = latesttime
#
# store the time spent in this location
#
if latestlat not in timespent:
timespent[latestlat] = {}
if latestlon not in timespent[latestlat]:
timespent[latestlat][latestlon] = 0
timespent[latestlat][latestlon] += valueMilliseconds
else:
# plot this point - we don't care how about how long was
# spent here
pts.append((latestlon, latestlat))
################################################################################
#
# draw the extracted points on a heat map
#
################################################################################
# if we are adjusting the density based on how long was spent at locations,
# we need to do this now
# if not, we added points to the heat-map store as we parsed the KML, so there
# is nothing else to do
if ADJUST_DENSITY_FOR_TIME == True:
# collect the points to draw
for lat in timespent:
for lon in timespent[lat]:
# how long did we spend at this location?
# so that the heat map reflects the time spent at locations, we
# add a point multiple times
timeSpent = timespent[lat][lon] / (1000 * 60 * 60 * DENSITY_ADJUST_HOURS)
if timeSpent > 0:
for n in range(int(timeSpent)):
pts.append((lon, lat))
else:
pts.append((lon, lat))
# draw the heatmap
print "var coordinates = ["
for point in pts[:-1]:
print "[ " + str(point[1]) + ", " + str(point[0]) + " ],"
print "[ " + str(pts[-1][1]) + ", " + str(pts[-1][0]) + " ]"
print "];"
Tags: geolocation, google, latitude, location, python
[…] at work last month, and getting distracted (like the goldfish that I am!) with other bits and pieces, I kinda forgot about this code after starting it last […]
[…] About a year ago, I noticed that the new Google Latitude dashboard had a way to export your history – it let you download a KML file containing locations that Google Latitude has captured you at. […]