Uploading data to Google Powermeter (with C#)

In my last post, I described how I had registered a device to Google Powermeter. I've had a chance since to try uploading data, using the API instructions.

Data is sent to Powermeter using the HTTP POST command. If you're not familiar with this, this is a way of sending data to a URL without it being visible to the user, often utilised when submitting web-forms. However, web requests are not restricted to web-browsers, but can be made by any program with an internet connection. This means we can send data to Google Powermeter from any device capable of retrieving the raw data and accessing the internet.

The first step (as outlined previously) is to get the security information for your account by registering your device (you can delete and add as many devices as you like).

If you haven't saved the security data, then it can be accessed via the "Activation Information" setting from your Settings page, and it will look something like this:

hash=8e8abf8262aeb7bab4e24afc7beeb18c701c04ba,6c258c9f62fa8a137eba7fa35b5f58802fca2e56,d48ef4db13304c49d36fb81b20cabbe147f8ce34&token=CI2JxY-FxDk6_EJGPXejIkF&path=/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home

We're interested in two pieces of data from here:

• "token" - this is a unique id number for this particular registered device, which you will need to include in every request
  
• "path" - this will form the core of the url will send all your data to

For now, we will be following the advice of the Google engineers, and only send "Durational" data, eg the total amount of electricity used between two specified points in time, in kilowatt hours. Make sure that when you create your device registration, you include at least one "durational variable" (eg dvars=1) when you make your original request:
https://www.google.com/powermeter/device/activate?mfg=GibbonEnergy&model=Gibbonex3000&did=1234&dvars=1

You can include as many durational variables as you like (eg submeters within your house, such as individual application monitor plugs), and these will be referred to as "d1", "d2" etc when you send your data. However, at the moment, it seems to rely on your common sense and guesswork to know which one is which.

OK, so we've registered the GibbonEx3000 meter, and want to start sending data to it. I'm a C#/.net developer, and all this is probably 200 times easier in some other language and there's probably better ways to write it in C# even, but this is how I did it - there's Python sample on the Google site.

First, you need to retrieve the data you want to send. Google only want this in 15 minute blocks, none of this "measuring every second" business. Some say that only by recording data at this level of detail can you understand what is really going on in your household consumption, but since there are 86,400 seconds in day, recording all this data for the number of people Google are aiming to reach is going to be a big headache (I've clocked up 3,876,373 records using my Current Cost since October 2009).

In the documentation, Google give this example, of 0.312 kWh being consumed between 3pm and 3:15pm on 21st May, 2008 (you have other ways of defining the time period, either a start or end date-time, with the duration). This is the data we want to send to Powermeter. I use a Linq query to generate these pieces of information from my raw data, there's no right or wrong way to do this, just whatever is accurate for your system and ends up looking exactly like this.

<entry xmlns="http://www.w3.org/2005/Atom" meter="http://schemas.google.com/meter/2008">
<meter:starttime uncertainty="1.0">2008-05-21T15:00:00.000Z</meter:starttime>
<meter:endtime uncertainty="1.0">2008-05-21T15:15:00.000Z</meter:endtime>
<meter:quantity uncertainty="0.001" unit="kW h">0.312</meter:quantity>
</entry>

However, we also have to supply two other pieces of information, the name of the meter (with the duration variable) in the url, and the unique key as part of the message header.

The url is built using the Google base address, the "path" as specified in the "Activation Information" and the Id of the particular duration variable we are going to use. In the documentation, this is given as the example:

https://www.google.com/powermeter/feeds/user/user-ID/security-zone/variable/variable-ID/durMeasurement

"user-id" and "security-zone" are the same 20 digit number (eg "23492384698234029827", and "variable-ID" is the duration variable (eg "d1") appended to the main device identifier (giving you something like this: "GibbonIndustries.GibbonEx3000.Home.d1")

To build this in C#, I've used the following code to generate the URL:

string userId = "23492384698234029827";
string deviceId = "Communergy.CurrentCost128.Home";       
string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);

This gives me something like:
https://www.google.com/powermeter/feeds/user/23492384698234029827/23492384698234029827/variable/GibbonIndustries.GibbonEx3000.Home.d1/durMeasurement


You will also want to introduce your security token into your code at this point. In my example it's a hard-coded string, but you probably want to include this as a config option. Be warned, you must get this exactly right, and for some reason the security code is incredibly difficult to copy and paste (perhaps because it contains hyphens, so likes to split itself across lines). It took me about 15 goes over the course of an extremely frustrating hour to do this correctly. If you get it wrong, you will get a "401 - not authorised" error. Enjoy ...

Either way, I pass the constructed url, xml and token into my main method, which actually makes the request.

First, we convert the xml into bytes, and create a new HttpWebRequest, and set the length to the number of bytes in the converted XML:

//turn the xml into bytes
var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
//create HttpWebRequest
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);         
httpWebRequest.Method = "POST";
httpWebRequest.ContentLength = bytes.Length;

We then set the content-type of the message to "ATOM", and add the token to the "Authorization" header, in the following format:

Authorization: AuthSub token="AUTHENTICATION_TOKEN"

You must include the double quotes around the token.

//set the required content type
httpWebRequest.ContentType = "application/atom+xml";
//set the token to the authorization header
string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
httpWebRequest.Headers.Add("Authorization", authSubHeader);

We then write the actual xml bytes to the HttpWebRequests request stream, and call the Response to submit the data:

//write  the xml bytes to the request stream
var requestStream = httpWebRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Close();
var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();             
{
  //Get response stream
  using (var responseStream = httpWebResponse.GetResponseStream())
  {                     
      using (var xmlReader = new XmlTextReader(responseStream))
      {
          response = xmlReader.ReadOuterXml();
      }
  }
}
httpWebResponse.Close();

If all goes well, there is no response, but you can excitedly rush to your Powermeter iGoogle gadget to see the data graphed, and your settings page will report "Status: Currently uploading data"


The code (posted in full below) is just a static sample, using hard-coded values, but is really all you need to get started - you will need to move your user Id and token values into a config file, and generate your request XML on the fly from your real data, and you probably want to do something about the ".d1" part as well.

I'm currently working on device-independent piece of client code, that currently reads Current Cost meters (Wattson/DIY Kyoto coming soon), which was pretty-much hard-coded to a dedicated server.

With the release of the Google API, I was forced to reconsider my design, and now use a much more flexible approach using the Command pattern, with different "Export Providers" being injected into the application controller. This allows any number of providers to be added, so Pachube, AMEE, Wattvision and hopefully Microsoft Hohm all await. If you're interested, it's all the source is available at http://communergy.codeplex.com/, but it's really not ready yet ..

Here it is:

static void Main()
{
   string userId = "23492384698234029827";
   string deviceId = "Communergy.CurrentCost128.Home";          
   string url = string.Format("https://www.google.com/powermeter/feeds/user/{0}/{0}/variable/{1}.d1/durMeasurement", userId, deviceId);


   string auth = "CI2JxY-FxDk6_EJGPXejIkF";
   string xmlEntry = @"
   
       2010-04-21T13:30:00.000Z
       2010-04-21T13:45:00.000Z
       0.642
   ";        

   var ret = PostXml(url, xmlEntry, auth);
}


public static string PostXml(string url, string xml, string authToken)
{
   string response = string.Empty;
   try
   {
       //turn the xml into bytes
       var bytes = System.Text.Encoding.ASCII.GetBytes(xml);
       //create HttpWebRequest
       var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);          
       httpWebRequest.Method = "POST";
       httpWebRequest.ContentLength = bytes.Length;

       //set the required content type
       httpWebRequest.ContentType = "application/atom+xml";
       //set the token to the authorization header
       string authSubHeader = string.Format("AuthSub token=\"{0}\"", authToken);
       httpWebRequest.Headers.Add("Authorization", authSubHeader); 
             
       //write  the xml bytes to the request stream
       var requestStream = httpWebRequest.GetRequestStream();
       requestStream.Write(bytes, 0, bytes.Length);
       requestStream.Close();
       var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();              
       {
           //Get response stream
           using (var responseStream = httpWebResponse.GetResponseStream())
           {                      
               using (var xmlReader = new XmlTextReader(responseStream))
               {
                   response = xmlReader.ReadOuterXml();
               }
           }
       }
       httpWebResponse.Close();
   }
   catch (WebException we)
   {
            
       throw new Exception(we.Message);
   }
   catch (Exception ex)
   {
       throw new Exception(ex.Message);
   }
   return response;
}

Setting up Google Powermeter

Google Powermeter is attracting a lot of attention in the home energy monitoring arena. Just the sheer reach and power of the Google brand are going to ensure it will be a major player.

So far, Google's focus has been on tie-ins with major energy utilities, whose customers already have wired-in smart meters in their homes. So far, what it has not dealt with are the millions such as myself who have internet-enabled monitors (either through a home pc, or with their own internet connection).

However, this changed last week with the announcement that their now-legendary Powermeter API will be publically available for all to use.

After a few false starts, and some patient help from the Powermeter team, I managed last night to register as a user. You don't need to be a registered provider (such as a utility), but you are responsible for uploading the data yourself to the web. It's an API, not a specific implementation for a particular monitor. However, a Python client library is supplied, once you've got your data out of your device.


The Getting Started page is pretty clear, once you get the hang of it. The documentation is aimed squarely at developers, who are writing a gateway to the API for their particular device or service, not standard users who just want to upload their data.

Before you do start, you will need a Google account (eg for Google Mail), and be logged in.

The API documents describe how to create a webpage for your website that your users will use to enroll to Powermeter. Once logged in, you have to create a webpage that will create a URL to post to Powermeter. This must take this format:
https://www.google.com/powermeter/device/activate?mfg=GibbonEnergy&model=Gibbonex3000&did=1234&cvars=1

Check the "Parameter details" section for details of what the different parameter values mean. You can customise the process by adding a return URL, otherwise you will stay within the Google pages.

Opening the link (which normally you would build within the page, but you can follow the link above or one like it) takes you to the "Activation" page



Confirm your details (including adding a geographic location), and if you have not specified a return URL, you will be presented with this page:



The "Authorization Information" is the unique key you will need to start uploading your data. If you specified a return URL, this will POSTed back to you in a particular format, so for automatic registrations you will be able to capture and store this. In this situation, take a copy of this manually, so you can use it in your API calls.


What you have to do now is master the upload API, but this doesn't look too complex - it's just wrapping up the standard SI units of time and energy, getting that data from your device in the first place is the hard bit.

Once your device has been activated, you have a set of options for how you want to display the data, including setting energy reduction targets:




Google have built an iGoogle gadget which you can place on your page and share with others:



(this probably looks a lot better once you start adding data ...)


So, there you go, that's how you activate a device on Powermeter, without being a customer of a subscribing utility. Anyone with a Google Account can do it.

I'll post again once I manage to upload data.

Weather station setup

The starting point of the whole system is a Maplin Weather Station. There's a few different ones on the market, but unless you're really serious, I think the critical thing your going to need is a wireless transmission from the outdoor weather station to your indoor base-station, and a USB connection from the base-station to your PC. The actual weather stations themselves to vary wildly in quality and performance, typical features to look out for are:

* temperature
* humidity
* rainfall
* wind-speed

Ideally, the thermometer should be inside a stevenson screen, to ensure readings don't get distorted by sunlight, and you want to make sure that you position your wind-gauge as high as possible to avoid wind-turbulance:





To fix mine, I got a 1.8m aerial from Screwfix, along with a mounting kit (that I can't find now), and attached it to my shed wall with the biggest coach-bolts I could find:



Early experiments siting the external thermometer/transmitter suggested that the plastic stevenson screen was absolutely useless, as I could identify the points at which the sun went behind, then re-appeared from the abandoned office block at the end of my street, by the sharp falls and rises in temperature.

Luckily, there's a massive rat-infested ivy swarming over my shed, and I managed to tuck the transmitter into there, keeping it sheltered from the heat:




This transmits a signal to the supplied base station, connected to an old laptop with a USB cable:


The supplied Easyweather software seems to be universally regarded as useless, not least because a memory leak guarantees it will crash if left running for any length of time, not ideal for this kind of software.

Instead, the entire world seems to love Cumulus from Sandaysoft:



With very little customisation, this can export your weather data to any number of formats, eg:
* a standard website
* twitter
* WeatherUnderground
* PWS Weather

Weather data, just like energy consumption data, always makes more sense when brought together and compared, and the last 2 sites particularly make an excellent job of this.

But come on, just uploading to somebody else's site just isn't exciting enough, no matter how well designed and useful it is - how do we get the raw data out so we can deal with it ourselves?

(to be continued ...)

From console to communergy

It's been almost a year since I wrote anything here. I got distracted, and I gave the link to a few people and never heard anything from them ever again - not quite the intention.

Anyway, I've been working on producing a personal home energy tracking system, since I acquired a Maplin Weather Station, and started to wonder how some of the things I've been learning on Open University environment and technology courses could be put to some use. I'd done a couple of spreadsheet and footprint exercises, but what has really been happening in our household energy consumption?

And how did I get from here:




to here



I've put this off for about a year, but I'll try and go through all the stages, not least so I don't forget why I've made some of the decisions that seemed to make sense at the time. In the meantime, you can see my personal home-monitoring site here , and the source code for the multiple-user version is here.

I'll try to stop complaining about Slough Borough Council's approach to renewable energy as well.

Another wind turbine planned for Slough

lock your doors, an application is in for another baby-eating wind-turbine in Slough!

Well, almost - there's an application on the Slough Council planning website

INSTALLATION OF A 40M GUYED WIND MONITORING MAST WHICH WILL BE REQUIRED TO MONITOR SITE CONDITIONS FOR A PERIOD OF 3-12 MONTHS 

The average wind-speed at this site, just north of Wexham Hospital, is about 6.4 m/s, which on a 3MW turbine means about 5GWh p/year. I don't know who is putting this application in, but at least they seem to be more serious and prepared than the last lot in Cippenham

Speaking with some of the councillors on the planning committee since the fiasco of the last hearing, there still isn't a policy on renewable energy in Slough, but they suggested that I raise this with the Scrutiny committee

maybe

Slough's energy use ...

Slough has approximately 45,000 households.

Each household uses approximately 4239 kWh each year in electricity (reference to come).

This means that Slough uses about 191,000,000kWh each year (or 191,000MWh, or 191GWh)

A 3MW wind turbine in Slough will produce about 3972MWh a year, or 4GWh. This is approximately 2% of the current electricity requirement for the town.

A household solar panel in the Thames Valley will produce about 2MWh each year, requiring approimately 2,000 such installations to produce the same amount of electricity as 1 3MW wind turbine, or 95,500 to meet the total requirement - over double the actual number of households.

That's the plain facts of electricity use in Slough.

Arrest Climate Change

Whilst visiting Slough Police Station (not "visiting") last night, I spotted this:

"burning fossil fuels has consequences" .... sounds much more sinister when it comes from The Plod ...