NiFi - Collecting data from bitly
by Lewis
Posted on January 1, 2018 at 12:00 PM
In this article I describe how I use NiFi to cllect data from the url shortening service bitly and store this collected data in a DynamoDB table.
Recently I released two Alexa skills to the Amazon Skill Store. As exciting as this was, I needed to promote my skills to ensure that I received maximum engagement. As part of my release, I promoted the Skills on social media sites such as LinkedIn, Facebook, Twitter & Instagram. As I had never promoted an Alexa skill before, I was interested in which links would be clicked on what social media site. In order to help me track this data, I looked at a variety of methods - I could use Google Analytics, I could have put clicks through a funky redirect and recorded the click myself, or the option I finally settled on, I could use a URL shortening service. This would also have other benefits - for example saving space on my promotional Tweets. There are many URL shortening services available (Ow.ly, Bit.ly to name just two) but I settled on Bitly as I have used them before and am familiar with their API.
Setting up the link in bitly
After you have signed up for free your bitly account, you will need to set up your first link. Navigate to bitly and login, you will be redirected to the "Bitlinks page";
From here, press the Create button in the top right corner to be presented with this pop out
Now paste your target url, in my case the Amazon Skill link, into the target box and press Create. Your shortened url is created and you are directed to a further page where you can refine the link shortening. By default bitly will create a shortened url using random characters, and its these random characters that can be changed. According to this link, customised urls get 34% more traffic but not only that, a shortened URL with a custom back link just looks more professional. My shortened URLs look like this;
Using the bitly API
bitly has an API that amongst other things can be used to extract the data recorded about your shortened link. The current version of the bitly API as of writing is version 4, however the example below is using version 3. The main difference between the two versions is the authentication, version 3 allows you to authenticate by passing the access token as a query parameter, while version 4 removes that altogether in preference of the standard Authorisation: Bearer header.
Authenticating with the bitly API is pretty standard and they offer four ways to authenticate. These are OAuth Web Flow, Exchange a username\password, HTTP Basic or a Generic Access Token. For my requirements I'll be using the Exchanging a username\password flow.
When successfully calling the authentication end point, it returns the Access token in a JSON message;
{"access_token": "e663e30818201d28dd07803e57333bed4f15803a"}
This access token is then used in the subsequent API calls. Now we know how we authenticate against the API, lets identify the end points to use. One of my requirements is to see which Social platform gets me the most clicks, so the "referring domain" end point looks good. I'll also grab the "referring countries" data as my Alexa skills are multi lingual so will be interested to see if the non English versions get picked up, and to corroborate the previous metrics, I'll grab the click data as well. The endpoints I will use then are;
| Title | Endpoint |
|---|---|
| Click data | https://api-ssl.bitly.com/v3/link/clicks?access_token=<access token>&rollup=false&link=<bitly url> |
| Referring Domains | https://api-ssl.bitly.com/v3/link/referring_domains?access_token=<access token>&rollup=false&link=<bitly url> |
| Referring Countries | https://api-ssl.bitly.com/v3/link/countries?access_token=<access token>&rollup=false&link=<bitly url> |
Each of these end points will return a JSON message that I can either parse, manipulate or record into DynamoDB.
Now that we have the end points we require & we know the authentication method we can implement this into a NiFi flow.
NiFi
[TBC]