I really need to stop blogging about CurrentCost stuff, this is starting to look like an obsession 🙂
But before I do that, I thought I’d share my experiences in writing a new CurrentCost parser.
What is this all about?
On the off-chance that anyone following me hasn’t already heard about CurrentCost a million times already, CurrentCost meters are domestic electricity monitors that tell you how much electricity your household is using. And they have a connection underneath that lets you grab the data from them.
As well as getting your current electricity usage, you can also get historical usage totals out of the serial port, which I’ve used in a variety of ways, such as in my Python CurrentCost software.
CurrentCost have now brought out a new model of their monitor – the CC128. The data format is different to the current models, so I wanted to try and write a new parser that could handle both the existing model and the new CC128 data.
Connecting
First things first, connecting to the meter.
I used to do this in Python to get data from the meter:
ser = serial.Serial(port='COM20', timeout=5)
line = ser.readline()With the CC128, I need to use:
ser = serial.Serial(port='COM20',
baudrate=57600,
bytesize=serial.EIGHTBITS,
parity=serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
timeout=3)
line = ser.readline()Getting history data – how it used to work
My old CurrentCost meter outputs a line of XML every six seconds or so. Most of the time it includes history data, but not always. But it was good enough for my app to keep reading updates from the meter until it gets one with history data – at worst, the user waits for about 15 seconds until the app has the data it needs.
Getting history data – how it works now
The CC128 doesn’t include history data in every (or nearly every) update. The draft doc on the CurrentCost website says:
“History messages are output starting at 1 minute past every odd hour, ie 17:01, 19:01 etc”
so you can potentially have to wait quite a while to get history update (or be connected at the right time!).
Alternatively, you can force the meter to output the history data by
“pressing and holding the Down and OK buttons together until the LED flashes”
The LCD display changes to just displaying a single number, flashing 1 to 9 repeatedly until it has finished outputting the history.
This is a bit of a pain for me with my app, as I get the data from the meter remotely so needing to be near enough to the meter to press the buttons is less than ideal.
How it works now – what it looks like
I’ve already blogged about what history data you can get from the CC128, and the draft doc for the CC128 XML gives a specification of the data.
What I didn’t realise from reading it was that you don’t get a single line of XML with the whole history data in it. You get a series of XML updates, each of which contains a sliding, overlapping window into the history data.
First you get two-hourly history, then daily history, then monthly history data. Each set of history data is returned in chronological order – from the oldest data to the most recent.
For example, my meter has been running for 14 days since I last reset it. So the oldest two-hourly history data is h324 – the electricity used from 326 hours to 324 hours ago.
So the first line of two-history data contains the two-hourly history data:
h324, h322, h320, h318, ... h230, h228, h226
The second line of history data received contains the two-hourly history data:
h322, h320, h318, h316, ... h228, h226, h224
The third line of history data received contains two-hourly history data:
h320, h318, h316, h314, ... h226, h224, h222
And so on, until the end of the two-hourly history data:
h102, h100, h098, h096, ... h008, h006, h004
And then the daily history data:
d014, d013, d012, d011, d010, ... d003, d002, d001
I would then receive monthly data, except my meter hasn’t been running for a month since last reset, so there isn’t any monthly history data yet.
Notice that you only receive XML for the data you have – unlike the old meter which receives the full XML in every history update, with zeroes for data not yet available.
If you want to see an example, here is my history data for the last couple of weeks.
How do you use the CC128 history data
The history data is big. Very big.
This explains why we don’t get history data every 6 seconds!
Getting just fourteen days of history data (i.e. 112 updates of two-hourly history, 1 update of daily data, 0 updates of monthly data) takes about a minute ten minutes to be output, and produces about 1MB of XML.
For my interactive app which displays a progress bar while it’s waiting to receive a complete copy of the history, this means that once the meter history is full, downloading history data could take a few minutes quarter-of-an-hour or longer.
Still, there are benefits to this approach. The overlap in the history data means that we can handle losing a few updates without losing any data. And in some ways it makes more sense to output a new copy of the history after it has changed – outputting months of history data every 6 seconds is a little redundant.
Parsing the data
I’ve updated my Python parser so that it now can handle receiving XML from either the old or the new CurrentCost meters. It can be downloaded from http://code.google.com/p/currentcost/.
.
Update (10th Feb): I massively underestimated the time it takes to download the history. I was going by the timestamps contained in the XML I was downloading – but it turns out that the CurrentCost meter’s clock stops while it is outputting history data. What I said took only a minute took more like ten minutes. It also resulted in my CurrentCost clock being ten minutes slow(er) once this was complete – every time I downloaded the history.
Update (18th Feb): I got a new CC128 last week and now it’s been running for a week, I took a look at the XML. The new CC128 has a later software version (0.11) than the one I was using when I wrote this post (0.09). The XML produced by the newer version is a little different. There are a couple of new items included in there, for example, there is now a <dsw> (days-since-wipe) item in addition to the previous <dsb>. More importantly though, it looks like v0.11 doesn’t overlap the history data in the way that 0.09 did. This is great to see.
Tags: currentcost, python
Great Work Dale! Are you planning on tackling the .NET parsing code to update that for the cc128 too?
@Jamie – Probably not.
When we were initially planning HomeCamp, there was the intention to make it not just discussing the issues, but also part-hackday where people would try out new ideas. With this in mind, we didn’t want people to spend most of the day just getting data out of the meter and not have time to do anything with it.
This was the reasoning behind trying to make source available in a variety of languages.
With this incentive gone, I think I might just leave it at Python and hope someone else comes along and ports it to another language 🙂
Can your CurrentCost software be run in a client/server mode? What I mean is having the data collection “server” running on the PC that the CurrentCost is connected to, but having a client application that can be installed on a different PC that can access the server over the LAN (or even the WAN) and display the data?
The reason I ask is the machine I have my CurrentCost plugged into doesn’t have a screen or keyboard attached to it, and using VNC isn’t ideal.
@Paul – Yes, in fact that is how I run the app. My CurrentCost meter is still connected to my slug – a low-powered linux device that doesn’t have any x-server. It runs a Perl script which is receiving the CurrentCost updates, and publishes them to a really small message broker running on the slug.
My CurrentCost software can subscribe to the message broker and receive the data via it – which means I can run the software and collect data from anywhere on the Internet.
@dale – great. I don’t have a slug any more (sold it ages ago to another CurrentCost user) and will be connecting my CurrentCost to a little Viglen MPC-L machine that I’m going to install XP onto.
So how would I go about running it like I want with the Viglen as a server?
Dale
I’ve just got a CC128 with v0.11 and the manually initiated download of history data doesn’t appear to work. Pressing down and OK buttons together results in the centre column of the bar chart cycling upwards for a period before the screen then reverts to normal (not the flashing 1-9 you described above) . Apart from live data I appeared to get no history into your app so used hyperterminal to monitor. With that I get the 6 sec updates but nothing else other than the odd spurious character or so usually indicative of a wrong baud rate
Its now 20:58 and I have your app running in “stay connected” mode so lets see what happens…………………..
Yes that works but I do seem to have a period missing through last night. no idea why though.
I’ve now moved my CC128 from my office PC that gets switched on and off an on to a small Linux box that stays on all the time. I have the linux box continually logging the output. I do not want to use the message broker.
I’ve FTP’d the file over to the PC but ‘export history’ ‘import XML’ function on your software gives me the error ‘Received invalid data’. I get the same thing if no file name is provided as I get when I put in the full file path.
What am I doing wrong?
@Pete
The ‘export history’ function exports to CSV and was provided for people wanting to export their data for use in other software such as MS Excel.
The ‘import XML’ function is for importing XML data, in the format received from CurrentCost meters. There is no relation between the two.
Hope that helps.
Dale
Thanks I confused matters by referencing the ‘export history’ tab which contains the ‘import XML’ function. I am talking about imports only
Looking at your code I’ve now realised that import XML is only for testing by typing or pasting a line of XML into the text box. I’m not sure if it will take multiple lines but it didn’t error when I tried.
I want to be able import the multiple lines captured in a logfile captured on a different PC. That way I can get round the MQTT issue and FTP the file between PC’s. I’ve never written Python before but I may soon have to try and modify the function getDataFromXML(self, event) to open a file and read it line by line through the parser. It shouldn’t be that hard.
Pete
@Pete – Ah, I see what you mean.
Yes, as you say – the import XML was added for testing the app when I don’t have a CurrentCost meter to hand 🙂
Please feel free to submit any changes you make to http://code.google.com/p/currentcostgui/source/checkout – someone else might want a similar feature, and if your code is in the app then you wont have to port it into any future releases.
Kind regards,
D
I’ve got it working but its a bit of a hack right now, using your code as an example and learning python as I go.
The history works fine now but the 6second lines seem to get discarded, and if they don’t there doesn’t currently seem to be a way of displaying them as ‘live’.
Once I’ve tidied things up and If I’m happy with the result I’ll post the code. But don’t hold your breath its slow going at the moment.
Pete
[…] Dale Lane’s Blog on how to get data from a CurrentCost device […]
[…] http://www.dannytsang.co.uk/index.php/recording-currentcost-data-viewing-it-on-the-web/ https://dalelane.co.uk/blog/?p=456 http://code.google.com/p/currentcost/ http://code.google.com/p/energyathome/ […]
[…] MQTT with Mosquitto and ZeroMQ. Although MQTT seems widely used for this type of application ( see Dale Lane, Open Source Medicine, Andy Piper and The House That Twitters ) I decided in the end to use ZeroMQ […]
[…] MQTT seems widely used for this type of application ( see Dale Lane, Open Source Medicine, Andy Piper and The House That Twitters ) I decided in the end to use ZeroMQ […]
Any chance of you adding support for multiple sensors?
ie: “Sensor 0” refers to the “Whole House” and Sensors 1 to 9 are Appliance Sensors.
Live data:
CC128-v0.150000321:12:2276.600007710113001444
CC128-v0.150000321:12:2776.610148810112501070
Hi Dean – Sorry, this was a couple of years ago now, I’m not working on this code any more.
Kind regards, D
[…] Lane’s Java XML parser: https://dalelane.co.uk/blog/?p=456 (its python but it might be of some […]