18 Dec 2014

Survey Monkey API Ruby Part 1

I had an opportunity to access the Survey Monkey API and I found very few Ruby examples so I decided to post my own here, below is simple version without OAuth but it has page polling and optional field requests handled with splat variables.


1:  require 'httparty'  
2:  require 'certified'  
3:  require 'pp'  
4:  class SurveyMonkey  
5:   include HTTParty  
6:   attr_accessor :access_token, :base_uri, :api_key  
7:   attr_reader :result  
8:   def initialize(access_token, base_uri,api_key)  
9:    @access_token = access_token  
10:    @api_key = '?api_key=' + "#{api_key}"  
11:    @base_uri = base_uri  
12:    @result = {}  
13:   end  
14:   def get_survey_list(start_date, end_date, *fields)  
15:    current_page = 1  
16:    params_hash = {}  
17:    params_hash[:start_date] = "#{start_date}"  
18:    params_hash[:end_date] = "#{end_date}"  
19:    params_hash[:fields] = fields  
20:    params_hash[:page] = current_page  
21:    while true  
22:      @result = HTTParty.post("#{@base_uri}" + '/v2/surveys/get_survey_list' + "#{@api_key}",  
23:      #:debug_output => $stdout,  
24:      :headers => {'Authorization' => "bearer #{@access_token.to_s}", 'Content-type' => 'application/json'},  
25:      :body => params_hash.to_json  
26:     )  
27:     if @result["data"]["page"] == current_page  
28:      current_page += 1  
29:     else  
30:      break  
31:     end  
32:    end   
33:   end  
34:   def get_survey_details(survey_id)  
35:    @result = HTTParty.post("#{@base_uri}" + '/v2/surveys/get_survey_details' + "#{@api_key}",  
36:     #:debug_output => $stdout,  
37:     :headers => {'Authorization' => "bearer #{@access_token.to_s}", 'Content-type' => 'application/json'},  
38:     :body => {:survey_id => "#{survey_id}"  
39:     }.to_json  
40:    )  
41:   end  
42:   def get_respondent_list(survey_id, *fields)  
43:    params_hash = {}  
44:    params_hash[:survey_id] = "#{survey_id}"  
45:    params_hash[:fields] = fields  
46:    @result = HTTParty.post("#{@base_uri}" + '/v2/surveys/get_respondent_list' + "#{@api_key}",  
47:     #:debug_output => $stdout,  
48:     :headers => {'Authorization' => "bearer #{@access_token.to_s}", 'Content-type' => 'application/json'},  
49:     :body => params_hash.to_json  
50:    )  
51:   end  
52:   def get_responses(survey_id, respondent_ids, *fields)  
53:    params_hash = {}  
54:    params_hash[:survey_id] = "#{survey_id}"  
55:    params_hash[:respondent_ids] = respondent_ids  
56:    params_hash[:fields] = fields  
57:    @result = HTTParty.post("#{@base_uri}" + '/v2/surveys/get_responses' + "#{@api_key}",  
58:     #:debug_output => $stdout,  
59:     :headers => {'Authorization' => "bearer #{@access_token.to_s}", 'Content-type' => 'application/json'},  
60:     :body => params_hash.to_json  
61:    )  
62:   end  
63:   end  
64:  start_date = 'YYYY-MM-DD 00:00:00'  
65:  end_date = 'YYYY-MM-DD 00:00:00'  
66:  base_uri = 'https://api.surveymonkey.net'  
67:  access_token = 'XX register with developer API to obtain access token XX'   
68:  api_key = 'XX avaiable via Survey Monkey for your survey (not developer) account XX'  
69:  #create the object  
70:  xx = SurveyMonkey.new(access_token, base_uri, api_key)  
71:  #method call with parameters  
72:  xx.get_survey_list(start_date, end_date)  
73:  #print results to console  
74:  puts xx.result["data"]  

If a user can skip a question you cannot guarantee that the json tree will contain the data elements to traverse tree so you will have to retrieve each users complete data tree and then load them into normalised database tables.

An example of Oauth using the Google API is here.

To follow;
  • Error Handling

27 Nov 2014

debian 7 encrypted partitions setup and keyfile

How to setup Luks on Debian 7 here

 https://dl.dropboxusercontent.com/u/40638984/how_to_lvm_debian_wheezy.pdf

 Make sure to allow  a root partition of 300mb, see note.


Debian w/LUKS root+key file on USB
There are a few advantages to booting a system onto an encrypted root volume using LUKS keyfiles instead of passphrases. Using keyfiles, booting to an encrypted root volume can once again be automatic - no more entering a passphrase at the console during each boot cycle. Alternatively, the key-containing volume can be removed after boot and kept secure, so that the system can only be rebooted by the owner.

Following this guide should be quite safe. It's possible to maintain secondary access via the original passphrase still in LUKS's keystore, and relatively trivial to restore the original crypttab configuration. Still, it's best to have complete backups before you begin! I'll give some troubleshooting tips following the setup instructions.

NOTE: All bets are off unless you're running Debian GNU/Linux version 7.4 ("Wheezy") with a LUKS-encrypted root partition for which you possess the passphrase. You'll also need a spare USB thumb drive formatted as FAT32.

Setup
First, plug in your FAT32 or ext2 formatted USB drive and create within it a file of random content, about four kilobytes should do nicely:

dd if=/dev/urandom of=/media/THUMBDRIVE/keyfile bs=1024 count=4

Next, add this key to the LUKS keystore:

sudo cryptsetup luksAddKey /dev/sda1 /media/THUMBDRIVE/keyfile

Now the container can be decrypted using the key file. Next we'll need to set up crypttab to inform LUKS about where this key can be found. To do that, first lets get the USB drive's UUID, so we can reliably refer to it in our configuration:

ls /dev/disk/by-uuid/ | grep sda

Next, we'll need to direct LUKS to use this keyfile upon booting. Edit /etc/crypttab and make the root partition's line look like this:

sda5_crypt UUID=<LUKS Container UUID> /dev/disk/by-uuid/<USB drive UUID>:<USB key file name> luks,keyscript=/lib/cryptsetup/scripts/passdev

The real magic is in that last bit about the passdev keyscript. Because the decryption must take place so early in the boot process, USB disks are not normally mounted. In the past, we'd had to carefully write our own scripts that would find and mount the USB volume, then include that script in the initramfs. The developers of cryptsetup have included a solution, these days: passdev. Supplied a block device path (we use by-uuid because it will remain consistent across hardware changes) and a file path, passdev will temporarily mount the device and stream the file to STDOUT.

Finally, lets prepare the new, USB keyfile-aware initramfs. We'll have to direct initramfs to load some new modules into future builds. We'll need these in order for the kernel to access USB devices so early in the boot process. Edit /etc/initramfs-tools/modules and add the following lines:

vfat
fat
nls_cp437
nls_iso8859_1
nls_utf8
sd_mod
scsi_mod
usb-storage
usb-hid
uhci-hcd
ohci-hcd
ehci-hcd
usbcore


If you want, you can trim this down a bit if you know what encoding is in use by your filesystem, and choosing ext2 over FAT32. Personally, I prefer to use the FAT32 format because it is the standard default of most thumb drives, allowing me to buy an off-the-shelf disk and copy the key onto it in an emergency. The downside to this is that we need to support the several encoding flavors that geographically local FAT32 typically comes in: UTF-8, iso8859_1, and codepage 437.

Now, lets rebuild our initramfs using these modules and our modified /etc/crypttab. Type this command at a shell:

update-initramfs -u

...that's it, we're done! During the next reboot, grub (or whatever bootloader you're using) will load the initramfs and stub kernel, which will source the initramfs version of /etc/crypttab, causing the initramfs version of /lib/cryptsetup/scripts/passdev to be run with correct arguments, temporarily mounting the USB disk and piping the keyfile into cryptsetup, mounting the encrypted root container, allowing the boot process to pivot away from initramfs and into your real file system.

Using the alternate passphrase
If the system can't find the keyfile at boot time, you'll find yourself at a BusyBox prompt running from within the system's initramfs. As long as your original passphrase is still loaded into a LUKS key slot (i.e. you haven't intentionally removed it), simply issue these commands at the BusyBox shell, replacing /dev/sda1 with your LUKS container's partition, sda1_crypt with your encrypted root volume's name within /dev/mapper, and entering the passphrase when prompted after issuing the first command:

cryptsetup luksOpen /dev/sda1 sda1_crypt
vgchange -ay

Restore /etc/crypttab to its working state, regenerate your initramfs, and then reboot. You should see the familiar LUKS passphrase prompt, as before we started.

Removing the alternate passphrase
It is also possible to remove the original passphrase from the LUKS keystore, leaving the keyfile as the only way of booting the system. Issue the following command and, when prompted, enter the passphrase to be removed:

cryptsetup luksRemoveKey /dev/sda1
I have posted this here because the original page is nolonger available and it should be kept because it works..

7 Jan 2013

Debian Powerpc G4 wake on lan alternative

I had been searching for hours on alternatives to wake on lan for a G4 Mac Mini I bought for a home server project, these posts
http://ubuntuforums.org/archive/index.php/t-1701534.html
http://ubuntuforums.org/archive/index.php/t-1889083.html

to a very useful linux package powerpc-utils for controlling the open firmware which can schedule the mac to boot up on week days and weekends and also turn off the very annoying and loud chime on startup.

30 Sept 2012

Class Library Templates Visual Studio 2008 Where are you

Having spent far too long in hindsight trying to find these templates for some work on functions and regular expressions I gave up and installed the express version of Visual Studio 2010.

FYI, using the command line to populate the default templates doesn't work as described here.

So why am I doing this, a new role where SQL and .net are the sole available languages got me thinking of how to implement data cleansing routines from perl in SQL.

The appendix in this book Inside Microsoft SQL Server 2008: T-SQL Programming gives you the VB code you need which is a great place to start to learn intermediate techniques such as user defined functions and CLR assemblies.

CLR Assemblies allow you to incorporate Regular Expressions into SQL which can be used to verify email address structures and are the basis of data cleaning.

3 Jul 2012

Your first API call with Perl

I find learning more interesting when I have practical example to try, ebay have kindly provided an example of how to make an API call.  Click here for the full tutorial which lists the perl modules required.

If your using a Mac or Linux platform check out these resources first;

Perlbew is an excellent version manager similar to RVM (Ruby Version Manager).

- Cpanminus is an excellent module to ensure the root perl libraries remain separate to the Perlbrew libraries.  A very helpful blog post describes how to install and setup cpanminus to handle the local perl installation and libraries you will setup using perlbew.

- Another blog post describing the modules require for https api calls and a very helpful test script to ensure your setup is working.

If this has kickstarted your interest, these excellent resources are a most to learn more about XML and perl packages and libraries.  If you spend some time surfing the web you will find an online resource for more chapters from Perl and XML by Erik T. Ray and Jason McIntosh.  You can also pick up a used copy on ebay for a few pounds. 


22 Apr 2012

Data Modelling with differing levels of data granularity

I came across this in a discussion on Linkedin's Microsoft Business Intelligence group, the initial post asked

 "Join Fact and Dimension Tables that are at Two Different Grains
I have a fact table that is based off the source system at a order level. My dimension is at a order line detail level. Would creating a bridge table work to join the fact and dimension or should I create another order dimension table at the order level?"

Click here for a Kimball Group cheat sheet with examples suggested by Assaf Mentzer which answers this question. 

6 Apr 2012

SSIS load text file name into table for ETL auditing Unicode error

The old saying still rings true, you get what you pay for, pay nothing, expect nothing, this excellent post details how to read the file name but this post will cause a unicode error because the its missing details at this part



"Be sure to change the DATA TYPE from UNICODE to STRING.  Select OK to confirm changes."  This is impossible unless you do this

Click on Show Advanced Editor





to change the data type to String (see screen shot below)







Then your data type will be string (see below)


And then you will not have unicode errors.