Uploading data to Google Powermeter (with C#)

In my last post, I described how I had registered a device to Google Powermeter. I've had a chance since to try uploading data, using the API instructions.

Data is sent to Powermeter using the HTTP POST command. If you're not familiar with this, this is a way of sending data to a URL without it being visible to the user, often utilised when submitting web-forms. However, web requests are not restricted to web-browsers, but can be made by any program with an internet connection. This means we can send data to Google Powermeter from any device capable of retrieving the raw data and accessing the internet.

The first step (as outlined previously) is to get the security information for your account by registering your device (you can delete and add as many devices as you like).

If you haven't saved the security data, then it can be accessed via the "Activation Information" setting from your Settings page, and it will look something like this:

hash=8e8abf8262aeb7bab4e24afc7beeb18c701c04ba,6c258c9f62fa8a137eba7fa35b5f58802fca2e56,d48ef4db13304c49d36fb81b20cabbe147f8ce34&token=CI2JxY-FxDk6_EJGPXejIkF&path=/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home

We're interested in two pieces of data from here:

• "token" - this is a unique id number for this particular registered device, which you will need to include in every request
  
• "path" - this will form the core of the url will send all your data to

For now, we will be following the advice of the Google engineers, and only send "Durational" data, eg the total amount of electricity used between two specified points in time, in kilowatt hours. Make sure that when you create your device registration, you include at least one "durational variable" (eg dvars=1) when you make your original request:
https://www.google.com/powermeter/device/activate?mfg=GibbonEnergy&model=Gibbonex3000&did=1234&dvars=1

You can include as many durational variables as you like (eg submeters within your house, such as individual application monitor plugs), and these will be referred to as "d1", "d2" etc when you send your data. However, at the moment, it seems to rely on your common sense and guesswork to know which one is which.

OK, so we've registered the GibbonEx3000 meter, and want to start sending data to it. I'm a C#/.net developer, and all this is probably 200 times easier in some other language and there's probably better ways to write it in C# even, but this is how I did it - there's Python sample on the Google site.

First, you need to retrieve the data you want to send. Google only want this in 15 minute blocks, none of this "measuring every second" business. Some say that only by recording data at this level of detail can you understand what is really going on in your household consumption, but since there are 86,400 seconds in day, recording all this data for the number of people Google are aiming to reach is going to be a big headache (I've clocked up 3,876,373 records using my Current Cost since October 2009).

In the documentation, Google give this example, of 0.312 kWh being consumed between 3pm and 3:15pm on 21st May, 2008 (you have other ways of defining the time period, either a start or end date-time, with the duration). This is the data we want to send to Powermeter. I use a Linq query to generate these pieces of information from my raw data, there's no right or wrong way to do this, just whatever is accurate for your system and ends up looking exactly like this.

<entry xmlns="http://www.w3.org/2005/Atom" meter="http://schemas.google.com/meter/2008">
<meter:starttime uncertainty="1.0">2008-05-21T15:00:00.000Z</meter:starttime>
<meter:endtime uncertainty="1.0">2008-05-21T15:15:00.000Z</meter:endtime>
<meter:quantity uncertainty="0.001" unit="kW h">0.312</meter:quantity>
</entry>

However, we also have to supply two other pieces of information, the name of the meter (with the duration variable) in the url, and the unique key as part of the message header.

The url is built using the Google base address, the "path" as specified in the "Activation Information" and the Id of the particular duration variable we are going to use. In the documentation, this is given as the example:

https://www.google.com/powermeter/feeds/user/user-ID/security-zone/variable/variable-ID/durMeasurement

"user-id" and "security-zone" are the same 20 digit number (eg "23492384698234029827", and "variable-ID" is the duration variable (eg "d1") appended to the main device identifier (giving you something like this: "GibbonIndustries.GibbonEx3000.Home.d1")

To build this in C#, I've used the following code to generate the URL:

string userId = "23492384698234029827";
string deviceId = "Communergy.CurrentCost128.Home";       
string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);

This gives me something like:
https://www.google.com/powermeter/feeds/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home.d1/durMeasurement


You will also want to introduce your security token into your code at this point. In my example it's a hard-coded string, but you probably want to include this as a config option. Be warned, you must get this exactly right, and for some reason the security code is incredibly difficult to copy and paste (perhaps because it contains hyphens, so likes to split itself across lines). It took me about 15 goes over the course of an extremely frustrating hour to do this correctly. If you get it wrong, you will get a "401 - not authorised" error. Enjoy ...

Either way, I pass the constructed url, xml and token into my main method, which actually makes the request.

First, we convert the xml into bytes, and create a new HttpWebRequest, and set the length to the number of bytes in the converted XML:

//turn the xml into bytes
var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
//create HttpWebRequest
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);         
httpWebRequest.Method = "POST";
httpWebRequest.ContentLength = bytes.Length;

We then set the content-type of the message to "ATOM", and add the token to the "Authorization" header, in the following format:

Authorization: AuthSub token="AUTHENTICATION_TOKEN"

You must include the double quotes around the token.

//set the required content type
httpWebRequest.ContentType = "application/atom+xml";
//set the token to the authorization header
string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
httpWebRequest.Headers.Add("Authorization", authSubHeader);

We then write the actual xml bytes to the HttpWebRequests request stream, and call the Response to submit the data:

//write  the xml bytes to the request stream
var requestStream = httpWebRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();             
{
  //Get response stream
  using (var responseStream = httpWebResponse.GetResponseStream())
  {                     
      using (var xmlReader = new XmlTextReader(responseStream))
      {
          response = xmlReader.ReadOuterXml();
      }
  }
}
httpWebResponse.Close();

If all goes well, there is no response, but you can excitedly rush to your Powermeter iGoogle gadget to see the data graphed, and your settings page will report "Status: Currently uploading data"


The code (posted in full below) is just a static sample, using hard-coded values, but is really all you need to get started - you will need to move your user Id and token values into a config file, and generate your request XML on the fly from your real data, and you probably want to do something about the ".d1" part as well.

I'm currently working on device-independent piece of client code, that currently reads Current Cost meters (Wattson/DIY Kyoto coming soon), which was pretty-much hard-coded to a dedicated server.

With the release of the Google API, I was forced to reconsider my design, and now use a much more flexible approach using the Command pattern, with different "Export Providers" being injected into the application controller. This allows any number of providers to be added, so Pachube, AMEE, Wattvision and hopefully Microsoft Hohm all await. If you're interested, it's all the source is available at http://communergy.codeplex.com/, but it's really not ready yet ..

Here it is:

static void Main()
{
   string userId = "23492384698234029827";
   string deviceId = "Communergy.CurrentCost128.Home";          
   string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);


   string auth = "CI2JxY-FxDk6_EJGPXejIkF";
   string xmlEntry = @"
   
       2010-04-21T13:30:00.000Z
       2010-04-21T13:45:00.000Z
       0.642
   ";        

   var ret = PostXml(url, xmlEntry, auth);
}


public static string PostXml(string url, string xml, string authToken)
{
   string response = string.Empty;
   try
   {
       //turn the xml into bytes
       var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
       //create HttpWebRequest
       var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);          
       httpWebRequest.Method = "POST";
       httpWebRequest.ContentLength = bytes.Length;

       //set the required content type
       httpWebRequest.ContentType = "application/atom+xml";
       //set the token to the authorization header
       string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
       httpWebRequest.Headers.Add("Authorization", authSubHeader); 
             
       //write  the xml bytes to the request stream
       var requestStream = httpWebRequest.GetRequestStream();
       requestStream.Write(bytes, 0, bytes.Length);
       requestStream.Close();
       var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();              
       {
           //Get response stream
           using (var responseStream = httpWebResponse.GetResponseStream())
           {                      
               using (var xmlReader = new XmlTextReader(responseStream))
               {
                   response = xmlReader.ReadOuterXml();
               }
           }
       }
       httpWebResponse.Close();
   }
   catch (WebException we)
   {
            
       throw new Exception(we.Message);
   }
   catch (Exception ex)
   {
       throw new Exception(ex.Message);
   }
   return response;
}