Tuesday 26 September 2017

Using twitters user timeline API in Node JS

Having continued to create my dashboard that allows me to access information thats useful to me in a single, easy to access location, I began to think about how I could get my twitter timeline to display on screen (messages that I post and messages people I follow post).

Normally I like to try to just do a curl GET of the webpage I'm looking to use but twitter requires an oauth key to do even a get on your own profile. *sigh*.

Since I've started developing Twitter have changed their entire web based content to a flash new website, a few new APIs and a new "feel".
New Twitter Site

The first thing we need to become a twitter developer is to create an application on https://apps.twitter.com which requires a login and for your account to have a linked mobile number. You'll need to give the application a name and a website address where the application will later ve accessible (this doesn't have to be real and can be changed later).

You will then be given application settings information complete with access token, and security keys, specifically within the "Keys and access tokens" tab.
Application Settings
-----------------------------------------------------
What I specifically wanted to do in my dashboard was display a few of the latest tweets from my home timeline which was surprisingly difficult to figure out as most use cases seem to be around mass production of tweets and making "bots" and a lot of the documentation for node module implementation on node js didn't really describe what was happening.

It's worth noting that when thinking about how I wanted to achieve this I had to consider twitters rate limiting for per access token usage. Which comes to 15 calls per application/access token per 15 minute window, meaning realistically we could make one call a minute.

Initially I just wanted to get set up and the easiest way to do so seemed to be with an existing node module called (wait for it...) twitter npm. https://www.npmjs.com/package/twitter. Following their instructions I took a plain old Node Js application and added the following code using the oauth tokens we get in our application settings:
var Twitter = require('twitter');
var client = new Twitter({
      consumer_key: 'XXX',
      consumer_secret: 'XXX',
      access_token_key: 'XXX',
      access_token_secret: 'XXX'
    });
At this point there are two options for receiving tweets from our timeline and this is to use the Stream API or GET API.

STREAM: Creates a single connection that receives data as it is pushed from twitter, this allows us to process the data in real time and not make continuous calls to twitter.

GET: Consists of standard HTTPS GET calls as and when we want to get information from twitter.

Whilst streaming gives us the most up to date data from twitter, it comes with additional complexity of having an open connection between your server and twitter that must be secured and kept alive which will generally require a bit more thought than an evenings coding. GET on the other hand suits my purpose well and allows me to call and handle data as and when I want.

For fullness I will show how to implement both:
----------------------------------------------------------------------------------------------------------
STREAM User Timeline:
client.stream('user', {track: 'GallagherAiden'}, function(stream) {
      stream.on('data', function(tweet) {
        console.log("got the following twitter message: " + JSON.stringify(tweet));
        var createdTime = tweet.created_at;
        createdTime = createdTime.substring(0,16);
        var tweetStream = createdTime + " " + tweet.user.name + " : " + tweet.text;
      });
      stream.on('error', function(error) {
        if(error){
          log.dashErr(error);
        }
         console.err("Could not get any twitter feeds");
        });
      });
    });
Here we use the twitter npm ".stream" functionality with the client defining that we want a "user"s stream and that that users handle is "GallagherAiden".

We then use "stream.on" to handle the incoming data which I then stringify and make suitable for my needs. i.e. in the following format:
Tue Sep 26 19:04 Gary Lineker - Gorgeous finish from @GarethBale11 gives Real Madrid the lead in Dortmund.
----------------------------------------------------------------------------------------------------------
GET User Timeline:
client.get('statuses/home_timeline', function(error, tweets, response) {
      if(error){
       console.log(error);
      }
var thisTweet = tweets[0];
var createdTime = thisTweet.created_at;
createdTime = createdTime.substring(0,16);
var tweetGet = createdTime + " " + thisTweet.user.name + " - " + thisTweet.text;
console.log(tweetGet);

Here we use 'statuses/home_timeline' which responds either with the tweets, an error or the response. This relates directly to the home timeline of the user who is linked to the application. There is no way to alter this.
The above code would give the same output as the stream in terms of formatting.
----------------------------------------------------------------------------------------------------------

The output of a single tweet is extremely large! 
{
    "created_at": "Mon Sep 25 06:22:53 +0000 2017",
    "id": 912200715552124900,
    "id_str": "912200715552124928",
    "text": "Photographing the Peak District in autumn \u2013 in pictures https://t.co/mHSVxMpNt2",
    "truncated": false,
    "entities": {
      "hashtags": [],
      "symbols": [],
      "user_mentions": [],
      "urls": [
        {
          "url": "https://t.co/mHSVxMpNt2",
          "expanded_url": "https://trib.al/hCIqP6P",
          "display_url": "trib.al/hCIqP6P",
          "indices": [
            56,
            79
          ]
        }
      ]
    },
    "source": "<a href=\"http://www.socialflow.com\" rel=\"nofollow\">SocialFlow</a>",
    "in_reply_to_status_id": null,
    "in_reply_to_status_id_str": null,
    "in_reply_to_user_id": null,
    "in_reply_to_user_id_str": null,
    "in_reply_to_screen_name": null,
    "user": {
      "id": 87818409,
      "id_str": "87818409",
      "name": "The Guardian",
      "screen_name": "guardian",
      "location": "London",
      "description": "The need for independent journalism has never been greater. Become a Guardian supporter: https://t.co/EWg9aqA7PQ",
      "url": "https://t.co/c53pnmnuIT",
      "entities": {
        "url": {
          "urls": [
            {
              "url": "https://t.co/c53pnmnuIT",
              "expanded_url": "https://www.theguardian.com",
              "display_url": "theguardian.com",
              "indices": [
                0,
                23
              ]
            }
          ]
        },
        "description": {
          "urls": [
            {
              "url": "https://t.co/EWg9aqA7PQ",
              "expanded_url": "http://gu.com/supporter/twitter",
              "display_url": "gu.com/supporter/twit\u2026",
              "indices": [
                89,
                112
              ]
            }
          ]
        }
      },
      "protected": false,
      "followers_count": 6791634,
      "friends_count": 1113,
      "listed_count": 54290,
      "created_at": "Thu Nov 05 23:49:19 +0000 2009",
      "favourites_count": 152,
      "utc_offset": 3600,
      "time_zone": "London",
      "geo_enabled": false,
      "verified": true,
      "statuses_count": 378671,
      "lang": "en",
      "contributors_enabled": false,
      "is_translator": false,
      "is_translation_enabled": true,
      "profile_background_color": "FFFFFF",
      "profile_background_image_url": "http://pbs.twimg.com/profile_background_images/704160749/ff996aa3bc2009a2f9b97cdd43e8b5b7.png",
      "profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/704160749/ff996aa3bc2009a2f9b97cdd43e8b5b7.png",
      "profile_background_tile": false,
      "profile_image_url": "http://pbs.twimg.com/profile_images/877153924637175809/deHwf3Qu_normal.jpg",
      "profile_image_url_https": "https://pbs.twimg.com/profile_images/877153924637175809/deHwf3Qu_normal.jpg",
      "profile_banner_url": "https://pbs.twimg.com/profile_banners/87818409/1491899430",
      "profile_link_color": "005789",
      "profile_sidebar_border_color": "FFFFFF",
      "profile_sidebar_fill_color": "CAE3F3",
      "profile_text_color": "333333",
      "profile_use_background_image": false,
      "has_extended_profile": false,
      "default_profile": false,
      "default_profile_image": false,
      "following": true,
      "follow_request_sent": false,
      "notifications": false,
      "translator_type": "regular"
    },
    "geo": null,
    "coordinates": null,
    "place": null,
    "contributors": null,
    "is_quote_status": false,
    "retweet_count": 1,
    "favorite_count": 3,
    "favorited": false,
    "retweeted": false,
    "possibly_sensitive": false,
    "possibly_sensitive_appealable": false,
    "lang": "en"
  }
 ----------------------------------------------------------------------------------------------------------
I have implemented this in a way that calls from the server using a "setInterval" ever 60 seconds which keeps me within my rate limit:
setInterval(function(){
  getTwitterFeed();
}, 60000);
I then do a web client GET of this data in order to populate my dashboard. I'm currently working with the last five tweets and will probably post how I'm doing that in a later post.

For now. Adieu. I hope you enjoyed reading, feel free to tell me where I'm wrong and how I can improve. you can email me at aiden.g@live.co.uk  or leave me a comment.

Twitter Feed on my dashboard, reloading every sixty seconds

Tuesday 19 September 2017

National Rail Enquiries Darwin Lite Webservice and Node JS

Getting train times for my local dashboard

Recently I decided I wanted to make a dashboard using Node JS that would provide quick information that I look for regularly. These might be;
  - Share prices I'm interested in
  - Latest football scores and upcoming matches for my football team (Burton Albion)
  - A sub section of my twitter feed

What I definitely wanted was to be able to click onto my dashboard and quickly find out the next trains I can get back to Winchester as I've been commuting to and from London recently and leave for home at differing times. There can occasionally be delays and I'd like to see if there's any delays so I can leave early or later to avoid a rush.

So where to look? The obvious place to try was a quick google of network rail api's which took me to: http://www.nationalrail.co.uk/46391.aspx

The three API options were:
Online Journey Planner - This deals more with planning routes, calculating ticket prices etc. Which since I follow the same route everyday and buy a daily travel card, it doesn't really fit my need.

Knowledgebase - This provides information on engineer disruption, station data such as facilities and service disruption. Whilst the latter is helpful the others were less so, as my stations are always the same and I would be aware of engineer work on the journey to London.

Darwin - In its own words "GB rail industry’s official train running information engine, providing real-time arrival and departure predictions, platform numbers, delay estimates, schedule changes and cancellations." which fits my requirements perfectly!

To use the Darwin SOAP API's you need an authentication token which you get by registering here:

This limits you to 5000 requests / hour which is around 5 million over 28 days. This cap can be removed by contacting infoservices@nationalrail.co.uk though it's extremely unlikely I will hit that!

You can find all the documentation for the API's here:

*NOTE*: You must follow the brand guidelines if you are using the APIs attributing the data to network rail. Brand Guidelines & Logos

So how did I use it?

Great Question!
Now I've not used SOAP much so did the obvious thing and imported the supplied WSDL (v2017-02-02) (https://lite.realtime.nationalrail.co.uk/OpenLDBWS/wsdl.aspx?ver=2017-02-02) into SOAPUI (v5.3.0).

This gave me all the services I can use as well as example XML formats for input to a REST API.



This gives me the HTTP address to make the calls to, the valid XML the service requires and lets me test with the data I will be using.
       <typ:TokenValue> : This is supplied by Network Rail after registration and is required
      <ldb:numRows> : This denotes the number of responses to return.
       <ldb:crs>  : The first of these which is required states the departure station, in my case Waterloo  
        or WAT in network rail world. Full list: http://www.nationalrail.co.uk/stations_destinations/default.aspx 
                          : The second is used for destination station and more can be added for other 
       destination.
      <ldb:timeOffset> : This is how much after the current time you want the search to look for.
      <ldb:timeWindow> : Denotes amount of time to get results for

The output is also very straight forward based on SOAP UI:

Making the request in Node JS

In our node JS application we can achieve the equivalent of this call by using the "request" node module: https://www.npmjs.com/package/request

var request = require("request"); 
var postOptions = {
      url: 'https://lite.realtime.nationalrail.co.uk/OpenLDBWS/ldb10.asmx',
      headers: {
        'Content-Type': 'text/xml'
      },
      body: data/waterlooToWinchester.xml,
      method: 'POST'
    };
    request(postOptions, function cb(err, res, body) {
      if(err) {
        console.error(err);
      } else { 
console.log(body);
}  
Where "data/waterlooToWinchester.xml" is the same XML shown in the SOAPUI request example.

This outputs:
<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><GetDepBoardWithDetailsResponse xmlns="http://thalesgroup.com/RTTI/2017-02-02/ldb/"><GetStationBoardResult xmlns:lt6="http://thalesgroup.com/RTTI/2017-02-02/ldb/types" xmlns:lt="http://thalesgroup.com/RTTI/2012-01-13/ldb/types" xmlns:lt4="http://thalesgroup.com/RTTI/2015-11-27/ldb/types" xmlns:lt5="http://thalesgroup.com/RTTI/2016-02-16/ldb/types" xmlns:lt2="http://thalesgroup.com/RTTI/2014-02-20/ldb/types" xmlns:lt3="http://thalesgroup.com/RTTI/2015-05-14/ldb/types"><lt4:generatedAt>2017-09-19T22:44:55.6600574+01:00</lt4:generatedAt><lt4:locationName>London Waterloo</lt4:locationName><lt4:crs>WAT</lt4:crs><lt4:filterLocationName>Winchester</lt4:filterLocationName><lt4:filtercrs>WIN</lt4:filtercrs><lt4:platformAvailable>true</lt4:platformAvailable><lt6:trainServices><lt6:service><lt4:std>23:05</lt4:std><lt4:etd>On time</lt4:etd><lt4:operator>South Western Railway</lt4:operator><lt4:operatorCode>SW</lt4:operatorCode><lt4:serviceType>train</lt4:serviceType><lt4:length>5</lt4:length><lt4:serviceID>0rdRe8ywmDifl5VO1N1Q4Q==</lt4:serviceID><lt5:rsid>SW927700</lt5:rsid><lt5:origin><lt4:location><lt4:locationName>London Waterloo</lt4:locationName><lt4:crs>WAT</lt4:crs></lt4:location></lt5:origin><lt5:destination><lt4:location><lt4:locationName>Poole</lt4:locationName><lt4:crs>POO</lt4:crs></lt4:location></lt5:destination><lt6:subsequentCallingPoints><lt6:callingPointList><lt4:callingPoint><lt4:locationName>Woking</lt4:locationName><lt4:crs>WOK</lt4:crs><lt4:st>23:31</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Basingstoke</lt4:locationName><lt4:crs>BSK</lt4:crs><lt4:st>23:51</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Micheldever</lt4:locationName><lt4:crs>MIC</lt4:crs><lt4:st>00:03</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Winchester</lt4:locationName><lt4:crs>WIN</lt4:crs><lt4:st>00:11</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Eastleigh</lt4:locationName><lt4:crs>ESL</lt4:crs><lt4:st>00:22</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Southampton Airport Parkway</lt4:locationName><lt4:crs>SOA</lt4:crs><lt4:st>00:27</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>St Denys</lt4:locationName><lt4:crs>SDN</lt4:crs><lt4:st>00:32</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Southampton Central</lt4:locationName><lt4:crs>SOU</lt4:crs><lt4:st>00:37</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Totton</lt4:locationName><lt4:crs>TTN</lt4:crs><lt4:st>00:43</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Brockenhurst</lt4:locationName><lt4:crs>BCU</lt4:crs><lt4:st>00:54</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>New Milton</lt4:locationName><lt4:crs>NWM</lt4:crs><lt4:st>01:02</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Christchurch</lt4:locationName><lt4:crs>CHR</lt4:crs><lt4:st>01:09</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Pokesdown</lt4:locationName><lt4:crs>POK</lt4:crs><lt4:st>01:13</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Bournemouth</lt4:locationName><lt4:crs>BMH</lt4:crs><lt4:st>01:17</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Branksome</lt4:locationName><lt4:crs>BSM</lt4:crs><lt4:st>01:23</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Parkstone (Dorset)</lt4:locationName><lt4:crs>PKS</lt4:crs><lt4:st>01:26</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint><lt4:callingPoint><lt4:locationName>Poole</lt4:locationName><lt4:crs>POO</lt4:crs><lt4:st>01:30</lt4:st><lt4:et>On time</lt4:et><lt4:length>5</lt4:length></lt4:callingPoint></lt6:callingPointList></lt6:subsequentCallingPoints></lt6:service></lt6:trainServices></GetStationBoardResult></GetDepBoardWithDetailsResponse></soap:Body></soap:Envelope>
This isn't much good to me at least, as I prefer to use JSON to access the data at the web client side. To do this there is a neat little node module called "xml2js": https://www.npmjs.com/package/xml2js

The below snippet parses the xml string and removes the prefixes from the xml tags to make it more readable and accessible as JSON.

var parseString = require('xml2js').parseString;
var stripPrefix = require('xml2js').processors.stripPrefix;
parseString(body, { tagNameProcessors: [stripPrefix] }, function (err, result) {
            if(err){
              console.log(err);
            }else{
              var currentJourney = result.Envelope.Body[0].GetDepBoardWithDetailsResponse[0].GetStationBoardResult[0].trainServices[0].service[0];
                  var train = {
                    "trainTime": currentJourney.std[0],
                    "status": currentJourney.etd[0],
                    "departingFrom": currentJourney.origin[0].location[0].locationName[0],
                    "goingTo": currentJourney.destination[0].location[0].locationName[0]
                  };
}, function afterJourneysProcessed(err) {
                if(err) {
                  console.log(err);
                }else{
                  return(train);
                }
});
At the client side we can handle this "train" variable by parsing the JSON and then setting the html based on the JSON input.

dashboard.html
<p id="nexttrains1"></p>
dashboard.js

var example = JSON.parse(this.responseText);
        //train 1
        var trainOutput1 = example[0].departingFrom + " to " + example[0].goingTo + "<br>";
        trainOutput1 = trainOutput1 + "<b>Departing at:</b> " + example[0].trainTime + " <b>Current Status:</b> " + example[0].status;
        document.getElementById("nexttrains1").innerHTML = trainOutput1;
This returns a single train journey. In my own code I also included an array called "calling points" and returned five journeys. This is quite a bit more code and a little too much for a single blog (since I've already added a lot already) but I can pass it over on request. Email me if you'd like to see the full code with promises and client to server code!

What does it look like? 


Anything you would do differently?
Any questions?

Feel free to email or comment and I'll gladly get back to you!