Tuesday, April 13, 2010

Uploading data to Google Powermeter (with C#)

In my last post, I described how I had registered a device to Google Powermeter. I've had a chance since to try uploading data, using the API instructions.

Data is sent to Powermeter using the HTTP POST command. If you're not familiar with this, this is a way of sending data to a URL without it being visible to the user, often utilised when submitting web-forms. However, web requests are not restricted to web-browsers, but can be made by any program with an internet connection. This means we can send data to Google Powermeter from any device capable of retrieving the raw data and accessing the internet.

The first step (as outlined previously) is to get the security information for your account by registering your device (you can delete and add as many devices as you like).

If you haven't saved the security data, then it can be accessed via the "Activation Information" setting from your Settings page, and it will look something like this:

hash=8e8abf8262aeb7bab4e24afc7beeb18c701c04ba,6c258c9f62fa8a137eba7fa35b5f58802fca2e56,d48ef4db13304c49d36fb81b20cabbe147f8ce34&token=CI2JxY-FxDk6_EJGPXejIkF&path=/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home

We're interested in two pieces of data from here:
  • "token" - this is a unique id number for this particular registered device, which you will need to include in every request
  • "path" - this will form the core of the url will send all your data to
For now, we will be following the advice of the Google engineers, and only send "Durational" data, eg the total amount of electricity used between two specified points in time, in kilowatt hours. Make sure that when you create your device registration, you include at least one "durational variable" (eg dvars=1) when you make your original request:
https://www.google.com/powermeter/device/activate?mfg=GibbonEnergy&model=Gibbonex3000&did=1234&dvars=1

You can include as many durational variables as you like (eg submeters within your house, such as individual application monitor plugs), and these will be referred to as "d1", "d2" etc when you send your data. However, at the moment, it seems to rely on your common sense and guesswork to know which one is which.

OK, so we've registered the GibbonEx3000 meter, and want to start sending data to it. I'm a C#/.net developer, and all this is probably 200 times easier in some other language and there's probably better ways to write it in C# even, but this is how I did it - there's Python sample on the Google site.

First, you need to retrieve the data you want to send. Google only want this in 15 minute blocks, none of this "measuring every second" business. Some say that only by recording data at this level of detail can you understand what is really going on in your household consumption, but since there are 86,400 seconds in day, recording all this data for the number of people Google are aiming to reach is going to be a big headache (I've clocked up 3,876,373 records using my Current Cost since October 2009).

In the documentation, Google give this example, of 0.312 kWh being consumed between 3pm and 3:15pm on 21st May, 2008 (you have other ways of defining the time period, either a start or end date-time, with the duration). This is the data we want to send to Powermeter. I use a Linq query to generate these pieces of information from my raw data, there's no right or wrong way to do this, just whatever is accurate for your system and ends up looking exactly like this.

<entry xmlns="http://www.w3.org/2005/Atom" meter="http://schemas.google.com/meter/2008">
<meter:starttime uncertainty="1.0">2008-05-21T15:00:00.000Z</meter:starttime>
<meter:endtime uncertainty="1.0">2008-05-21T15:15:00.000Z</meter:endtime>
<meter:quantity uncertainty="0.001" unit="kW h">0.312</meter:quantity>
</entry>



However, we also have to supply two other pieces of information, the name of the meter (with the duration variable) in the url, and the unique key as part of the message header.

The url is built using the Google base address, the "path" as specified in the "Activation Information" and the Id of the particular duration variable we are going to use. In the documentation, this is given as the example:
https://www.google.com/powermeter/feeds/user/user-ID/security-zone/variable/variable-ID/durMeasurement

"user-id" and "security-zone" are the same 20 digit number (eg "23492384698234029827", and "variable-ID" is the duration variable (eg "d1") appended to the main device identifier (giving you something like this: "GibbonIndustries.GibbonEx3000.Home.d1")

To build this in C#, I've used the following code to generate the URL:

string userId = "23492384698234029827";
string deviceId = "Communergy.CurrentCost128.Home";
string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);
This gives me something like:
https://www.google.com/powermeter/feeds/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home.d1/durMeasurement


You will also want to introduce your security token into your code at this point. In my example it's a hard-coded string, but you probably want to include this as a config option. Be warned, you must get this exactly right, and for some reason the security code is incredibly difficult to copy and paste (perhaps because it contains hyphens, so likes to split itself across lines). It took me about 15 goes over the course of an extremely frustrating hour to do this correctly. If you get it wrong, you will get a "401 - not authorised" error. Enjoy ...

Either way, I pass the constructed url, xml and token into my main method, which actually makes the request.

First, we convert the xml into bytes, and create a new HttpWebRequest, and set the length to the number of bytes in the converted XML:


//turn the xml into bytes
var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
//create HttpWebRequest
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Method = "POST";
httpWebRequest.ContentLength = bytes.Length;


We then set the content-type of the message to "ATOM", and add the token to the "Authorization" header, in the following format:
Authorization: AuthSub token="AUTHENTICATION_TOKEN"
You must include the double quotes around the token.


//set the required content type
httpWebRequest.ContentType = "application/atom+xml";
//set the token to the authorization header
string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
httpWebRequest.Headers.Add("Authorization", authSubHeader);


We then write the actual xml bytes to the HttpWebRequests request stream, and call the Response to submit the data:


//write the xml bytes to the request stream
var requestStream = httpWebRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
{
//Get response stream
using (var responseStream = httpWebResponse.GetResponseStream())
{
using (var xmlReader = new XmlTextReader(responseStream))
{
response = xmlReader.ReadOuterXml();
}
}
}
httpWebResponse.Close();


If all goes well, there is no response, but you can excitedly rush to your Powermeter iGoogle gadget to see the data graphed, and your settings page will report "Status: Currently uploading data"


The code (posted in full below) is just a static sample, using hard-coded values, but is really all you need to get started - you will need to move your user Id and token values into a config file, and generate your request XML on the fly from your real data, and you probably want to do something about the ".d1" part as well.

I'm currently working on device-independent piece of client code, that currently reads Current Cost meters (Wattson/DIY Kyoto coming soon), which was pretty-much hard-coded to a dedicated server.

With the release of the Google API, I was forced to reconsider my design, and now use a much more flexible approach using the Command pattern, with different "Export Providers" being injected into the application controller. This allows any number of providers to be added, so Pachube, AMEE, Wattvision and hopefully Microsoft Hohm all await. If you're interested, it's all the source is available at http://communergy.codeplex.com/, but it's really not ready yet ..

Here it is:


static void Main()
{
string userId = "23492384698234029827";
string deviceId = "Communergy.CurrentCost128.Home";
string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);


string auth = "CI2JxY-FxDk6_EJGPXejIkF";
string xmlEntry = @"

2010-04-21T13:30:00.000Z
2010-04-21T13:45:00.000Z
0.642
";

var ret = PostXml(url, xmlEntry, auth);
}


public static string PostXml(string url, string xml, string authToken)
{
string response = string.Empty;
try
{
//turn the xml into bytes
var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
//create HttpWebRequest
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Method = "POST";
httpWebRequest.ContentLength = bytes.Length;

//set the required content type
httpWebRequest.ContentType = "application/atom+xml";
//set the token to the authorization header
string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
httpWebRequest.Headers.Add("Authorization", authSubHeader);

//write the xml bytes to the request stream
var requestStream = httpWebRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
{
//Get response stream
using (var responseStream = httpWebResponse.GetResponseStream())
{
using (var xmlReader = new XmlTextReader(responseStream))
{
response = xmlReader.ReadOuterXml();
}
}
}
httpWebResponse.Close();
}
catch (WebException we)
{

throw new Exception(we.Message);
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
return response;
}

7 comments:

Justin said...

Awesome Work!

Cassandra said...

I was having some problems until I changed the case of the tags in the xml packets to match the documentation.

meter:starttime => meter:startTime

I also found it was better to read the responseStream with a System.IO.StreamReader().

Thanks for helping me get started!

Joe Kwiatkowski said...

Absolutely fantastic! Thanks very much.

Wonderful to think of the GPM API being pioneered from a camper van in Bedford.

Toby said...

I've found since the best way to do a post is with the HttpClient object from the Microsoft.Http library:


using (HttpClient client = new HttpClient())
{
client.TransportSettings.ConnectionTimeout = new TimeSpan(0, 0, 4);
client.TransportSettings.ReadWriteTimeout = new TimeSpan(0, 0, 4);
RequestHeaders headers = new RequestHeaders();
headers.ContentType = contentType;
headers.Add("Authorization", authSubHeader);
HttpContent content = HttpContent.Create(bytes);
HttpResponseMessage resp = client.Send(HttpMethod.POST, url, headers, content);
return resp.StatusCode;
}

Ox And Wegehead said...

Thank you for posting your work here and also on the powermeter forum. I have decided to take the Python plunge and just go that way to take advantage of the libraries.
Also, I don't think without your post I would have known that powermeter was now open. That's really cool - I had given it up when I saw it was just for energy utilities.

Ox And Wegehead said...

Made it work - just had to read, re-read- and re-read some more. Hacking around in a RAD called ATEasy (which is lot like VB6) I got it to work!
Thank you for helping me get started - anyone who has done training knows just get the confidence level with a success and the students usually start zooming along!

Toby said...

Congrats Mr Ox - like you say, you need to keep reading and checking, and you'll get it working (eventually)

Google have recently published a sample implementation, which might help others:
http://code.google.com/apis/powermeter/docs/powermeter_sample_implementation.html