Spot any errors? let me know, but Unleash your pedant politely please.

Thursday 20 December 2012

Still feeling a bit stupid

I saw yesterday that requests v1.0 was out.  I revisited it in the hope that I wouldn't feel so stupid.

It still doesn't work for me. In a different way.  I tried the three signature types I think are available, but all are rejected by the server.

import requests
from requests_oauthlib import OAuth1

consumer_key        = u"that"
consumer_secret     = u"would"
access_token_key    = u"be"
access_token_secret = u"telling"
                                 
for signature_type in ('auth_header','body','query'):
    auth = OAuth1(client_key            = consumer_key,
                  client_secret         = consumer_secret,
                  resource_owner_key    = access_token_key,
                  resource_owner_secret = access_token_secret,
                  signature_type        = signature_type)
    
    r = requests.get(u'https://api.projectplace.com/1/user/me/profile.json', auth=auth)
    
    print signature_type
    print r.encoding
    print r.text
    print r.json

gives me

auth_header
ISO-8859-1
Could not properly validate "oauth_consumer_key".
>

body
ISO-8859-1
Could not properly validate "oauth_consumer_key".
>

query
ISO-8859-1
Could not properly validate "oauth_consumer_key".
>
administrators-macbook-pro-2:ProjectPlace (requests) bob$ 

Ho hum.
Being a bit lazy, I have some modules that I use that I haven't packaged. I just copy them from project to project.  One is a Java style properties thingamajig. (This is probably unPythonic, there are probably better versions of the same thing out there, but it works for me).  I have a small test in the __main__ that requires a file containing some test data.

Being a bit lazy, I got fed up of copying that test file. It occurred to me today that I should write the test file as part of the test and remove it afterwards.

Why didn't I think of this before?

if __name__ == '__main__':
    
    open('test.properties','w').write("""Param : This is a parameter value
                                          Multi Param  : multi parameter 1
                                         +Multi Param : multi parameter 2
                                         +Multi Param :  multi parameter 3""")
    
    test_properties = Properties('test.properties')
    assert test_properties.property('Param')=='This is a parameter value'
    assert test_properties.propertyAsList('Multi Param')==['multi parameter 1','multi parameter 2','multi parameter 3']
    os.remove('test.properties')


I guess the tests should be put into some kind of unit test framework, but I can't quite yet be arsed to figure that stuff out. If this horrifies you, please do point me in the right direction.

Monday 3 December 2012

Sometimes I feel stupid...

I thought I'd refactor my Python class for accessing Project Place via its REST API using the purportedly more human Requests. A bit of reading and a bit of prodding and poking suggests that when it comes to OAuth, everything sucks, including Requests.

After a bit of faffing with the refactoring, I yanked the bare minimum of code into a single file. No subroutines or classes. Not a single indent:


import requests

consumer_key        = "that"
consumer_secret     = "would"
access_token_key    = "be"
access_token_secret = "telling!"

auth = requests.auth.OAuth1(consumer_key,
                            consumer_secret,
                            access_token_key,
                            access_token_secret,
                            signature_type='auth_header')

r = requests.get('https://api.projectplace.com/1/user/me/profile.json', auth=auth)

For my trouble, I get:

Traceback (most recent call last):
  File "/Users/bob/Dropbox/Coding/Python/ProjectPlace (requests)/requests_test.py", line 19, in
    r = requests.get('https://api.projectplace.com/1/user/me/profile.json', auth=auth)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 65, in get
    return request('get', url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
    return function(method, url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 51, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
    r.send(prefetch=prefetch)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.py", line 521, in send
    r = self.auth(self)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/auth.py", line 102, in __call__
    unicode(r.full_url), unicode(r.method), r.data, r.headers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/__init__.py", line 213, in sign
    request.oauth_params.append((u'oauth_signature', self.get_oauth_signature(request)))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/__init__.py", line 65, in get_oauth_signature
    uri, headers, body = self._render(request)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/__init__.py", line 135, in _render
    headers = parameters.prepare_headers(request.oauth_params, request.headers)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/utils.py", line 28, in wrapper
    return target(params, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/parameters.py", line 55, in prepare_headers
    escaped_value = utils.escape(value)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/oauthlib/oauth1/rfc5849/utils.py", line 52, in escape
    raise ValueError('Only unicode objects are escapable.')
ValueError: Only unicode objects are escapable.

And Googling doesn't really help. I discovered that the OAuth part of Requests is being pulled out into a separate package/project. I tried 'pip'ing it, but it's not available yet.  Sigh.

I should probably mention that:

import requests
print requests.get('https://www.google.com').content

Works perfectly well. It's the OAuth stuff that appears to be FUBAR.

Wednesday 17 October 2012

Film: Friends with Benefits.

I was at the NOVOTEL Eiffel Tower, sans Internet. Technically I had NOVOTEL's 'high speed' wired connection, but Speed Test measured this at 0.07Mbps, so it may as well not exist. Add a VPN in order to get to my Netflix account from France and it would be firmly in shit dial-up territory. I partied like it was 1999, and watched a DVD instead*

Of the small selection of movies to hand, I picked Friends with Benefits…

The movie opens with Justin Timberlake in LA breaking up with someone and Mila Kunis breaking up with someone else in New York.

Kunis plays a high flying creative industries head-hunting agent. Timberlake is a high-flying creative that Kunis places with a magazine in New York.  They become friends. Neither is romantically interested in the other but come to a mutually beneficial agreement to engage in sport fucking.

There are several sex scenes played mostly for amusement rather than eroticism. Timberlake's arse is shown but not his cock or balls.  Kunis' arse and side-boobs are shown, but her nipples and mons are strictly taboo.  I don't really mind this. I don't think it's necessary to be particularly graphic in other to tell the story - on the other hand, strategically placed sheets, oral strictly under the sheets just made me aware that these were actors. The obvious method of concealment took me out of the movie.

It would have been a good opportunity to emphasise the use of condoms, but that was possibly too icky. There wasn't any evidence of any fluids - I'm sorry, but a man comes up for air from between a woman's legs, from under a sheet, and there's no sweat or saliva or ladylube on his chin/lips/nose?  Again, it just seemed to icky-avoiding.

The scene that annoyed me most was one in which Timberlake's cunnilingus wasn't doing anything for Kunis, she gave him instructions, which her followed to her satisfaction. There was a gag implying that he licked her anus, which was what she's intended,  but anything visually icky was avoided.  This open communication, with one person given pleasure-giving instructions by another was welcome, but flawed. It started by ridiculing the man's technique (which may well have exactly what his previous partner preferred).  When it came to the reciprocal blow job, Timberlake attempted to express his preference, but this evidently wasn't necessary as Kunis proceeded to immediately give him the best fellatio of his life.  While there's probably some truth to the old adage that there's no such thing as a bad blow job, it would've been a better scene had both had learn each other's likes and dislikes.  It felt a little like man-mocking, which made the movie feel very much like the chick-flick it is.

Despite the sanitised sport-fucking, Kunis' character is romantic, besotted with a Hollywood romantic ideal. Kunis curses a poster of a Katherine Heigl movie, at which point, Friends With Benefits pretty much turns into a Katherine Heigl movie and predictable rom-commery ensues: Kunis falls for Timberlake; he's too stupid too see it; She gets the hump; Timberlake still wants to hump; He misses her, figures out he loves her and then everyone figures out how to help him win her back.

It's a Sony movie, and as with some other Sony movies, product placement is far from subtle.  I need to double check, but I suspect that the Katherine Heigl movie referenced at the point that Friends With Benefits jumps the shark is yet more product placement.

Intimacy, it is not.

*The DVD had actually been ripped and converted to h.264. I don't want you to think I actually carried the media with me.

Saturday 22 September 2012

iPhone 5

There seem to be two takes on the iPhone 5. One from the people who 'get' Apple (Gruber, Siegler et al) is that it's the best iPhone yet and we should all upgrade as soon as we can.  The other is that it's boring and disappointing and the most valuable company in the history of the world has seriously lost its mojo.

The truth is that it's a significant evolutionary release that I'd buy now if the cost meant nothing to me.  I've owned four phones in 15 years. I bought the first one because my wife was pregnant. I bought the second one because I was curious about bluetooth, it was second hand and it was similar to price of a replacement battery for the first phone.  I bought the third one because the second had been stolen and I was between jobs, so needed to be available to take calls from agents.

I bought the iPhone 4 because my iPod Touch (bought when my original 5GB iPod developed a fault after 7 years)  had taught me that a phone keypad was the wrong way to type on a handheld device, and, well, because I wanted one.

This hopefully establishes that I don't have a history of buying/replacing gadgets on a whim.  That hasn't changed with the iPhone 5.

As an iPhone 4 user/owner, it used to puzzle me that I'd still see people apparently happy with their inferior older model iPhones.  This no longer puzzles me at all.  I now understand that they didn't feel the need to upgrade because their 'old' phones simply didn't feel in any way broken to them.  That's exactly how I feel about the 4.  It's still an awesome device. It's not in anyway broken.

I wonder what the 5S/6 will be like next year?


Friday 14 September 2012

Joined up something

Following on from openpyxl win, I had, or at least found, more a little more spare time between writing tests to work a little more on producing progress reports from Excel files.

The thing that triggered this was discovering that Project Place, the document/project repository we use has an API. I thought I'd do a little digging.  I thought it would be cool not just to produce a report from the files, but to fetch the files automatically first.

I should state that the people at Project Place were great, and didn't give me any grief for being a total n00b at this kind of thing.

Initially, using a browser based client to run examples, it all looked pretty simple…

I decided to stick with Python rather than add another layer of unnecessary learning. The first hurdle was OAUTH. I gave up for a few days, but it kept bugging me, so I read more, understood more and managed to get through the Oauth process. The following code is pretty much the same as examples on the available online. There's a slight tweak to disable SSL certificate validation. It's probably not the right thing to do, and you may not need it (I didn't on OS-X on ADSL, but it worked around a problem on Windows on a corporate network through a firewall). The other differences are that oath properties are read from and written to a Java style properties file.

import urlparse
import local_oauth2 as oauth
import json
from Properties import Properties

consumer_properties = Properties('consumer.properties')

request_token_url = 'https://api.projectplace.com/initiate'
access_token_url  = 'https://api.projectplace.com/token'
authorize_url     = 'https://api.projectplace.com/authorize'

consumer = oauth.Consumer(consumer_properties.property('Key'),
                          consumer_properties.property('Secret'))

client = oauth.Client(consumer=consumer,
                      disable_ssl_certificate_validation=True)

# Step 1: Get a request token. This is a temporary token that is used for 
# having the user authorize an access token and to sign the request to obtain 
# said access token.

resp, content = client.request(request_token_url, "GET")
#print 'Response:',resp
#print 'Content:',content
if resp['status'] != '200':
    raise Exception("Invalid response %s." % resp['status'])

request_token = dict(urlparse.parse_qsl(content))

#print "Request Token:"
#print "    - oauth_token        = %s" % request_token['oauth_token']
#print "    - oauth_token_secret = %s" % request_token['oauth_token_secret']
#print 

# Step 2: Redirect to the provider. Since this is a CLI script we do not 
# redirect. In a web application you would redirect the user to the URL
# below.

print "Go to the following link in your browser:"
print "%s?oauth_token=%s" % (authorize_url, request_token['oauth_token'])
print 

# After the user has granted access to you, the consumer, the provider will
# redirect you to whatever URL you have told them to redirect to. You can 
# usually define this in the oauth_callback argument as well.
accepted = 'n'
while accepted.lower() != 'y':
    accepted = raw_input('Have you authorized me? (y/n) ')
oauth_verifier_url = raw_input('What is the URL that you were redirected to? ')

oauth_verifier = oauth_verifier_url.split("&oauth_verifier=")[-1]

#print "oauth_verifier:%s"%oauth_verifier

# Step 3: Once the consumer has redirected the user back to the oauth_callback
# URL you can request the access token the user has approved. You use the 
# request token to sign this request. After this is done you throw away the
# request token and use the access token returned. You should store this 
# access token somewhere safe, like a database, for future use.
token = oauth.Token(request_token['oauth_token'],
                    request_token['oauth_token_secret'])
                    
token.set_verifier(oauth_verifier)
client = oauth.Client(consumer = consumer,
                      token    = token,
                      disable_ssl_certificate_validation=True)

resp, content = client.request(access_token_url, "POST")
access_token = dict(urlparse.parse_qsl(content))

resp, content = client.request(request_token_url, "GET")
#print 'Response:',resp
#print 'Content:',content
#print 'Access Token:',access_token

#print "Access Token:"
#print "    - oauth_token        = %s" % access_token['oauth_token']
#print "    - oauth_token_secret = %s" % access_token['oauth_token_secret']
#print
#print "You may now access protected resources using user_access.properties." 
#print

user_access_properties=file('user_access.properties','w')
user_access_properties.write('Key    : %s\n'%access_token['oauth_token'])
user_access_properties.write('Secret : %s'%access_token['oauth_token_secret'])
user_access_properties.close()

local_oauth2 is pretty much the same as python-oauth2, but with the addition of the disable_ssl_certificate_validation parameter...

    def __init__(self,
                 consumer,
                 token=None,
                 cache=None,
                 timeout=None,
                 proxy_info=None,
                 disable_ssl_certificate_validation=False):

        if consumer is not None and not isinstance(consumer, Consumer):
            raise ValueError("Invalid consumer.")

        if token is not None and not isinstance(token, Token):
            raise ValueError("Invalid token.")

        self.consumer = consumer
        self.token = token
        self.method = SignatureMethod_HMAC_SHA1()

        httplib2.Http.__init__(self,
                               cache=cache,
                               timeout=timeout, 
                               proxy_info=proxy_info,
                               disable_ssl_certificate_validation=disable_ssl_certificate_validation)

The Project Place class, which is still a work in progress, and certainly not fully tested, looks like this:

#!/usr/bin/env python
# encoding: utf-8
"""
TalkToProjectPlace.py

Created by Hywel Thomas on 2012-07-19.
Copyright (c) 2012 Jupiterlink Limited. All rights reserved.
"""

#import sys
#import os
import local_oauth2 as oauth
import json

class BadValue(Exception):
    def __init__(self, message):
        self.message = message
        
    def __str__(self):
        return self.message

class BadParameterCombination(Exception):
    def __init__(self, message):
        self.message = message
        
    def __str__(self):
        return self.message

class ContainerNotFound(Exception):
    def __init__(self, message):
        self.message = message
        
    def __str__(self):
        return self.message
        
class NotImplemented(Exception):
    def __init__(self, message):
        self.message = message
        
    def __str__(self):
        return self.message

class ProjectPlace(object):

    def __init__(self,
                 consumer_key,
                 consumer_secret,
                 access_token_key,
                 access_token_secret,
                 format='json',
                 proxy_info = None,
                 disable_ssl_certificate_validation = False):

        self.consumer_key        = consumer_key
        self.consumer_secret     = consumer_secret
        self.access_token_key    = access_token_key
        self.access_token_secret = access_token_secret
        
        self.baseURL = "https://api.projectplace.com/1/"
        self.format=format     

        consumer = oauth.Consumer(key=consumer_key,
                                       secret=consumer_secret)
        access_token = oauth.Token(key=access_token_key, 
                                        secret=access_token_secret)
    
        self.client = oauth.Client(consumer   = consumer,
                                   token      = access_token,
                                   proxy_info = proxy_info,
                                   disable_ssl_certificate_validation = disable_ssl_certificate_validation) 


    def request(self, request, **parameters):
        
        full_request = u'%s%s.%s'%(self.baseURL,request,self.format)
        
        if len(parameters)>0:
            full_request="%s&%s"%(full_request,"&".join(['%s=%s'%(parameter,parameters[parameter]) for parameter in parameters]))

        resp, content = self.client.request(uri=full_request,
                                            method='GET')
        if resp['status']=='200':
            return content
        else:
            print '\n\n'
            print content
            return resp


    def request_binary(self,request):
        fullRequest = u'%s%s'%(self.baseURL,request)
        resp, content = self.client.request(fullRequest)

        if resp['status']=='200':
            return content
        else:
            print '\n\n'
            return resp

    # USER methods
    
    def user_profile(self,user_id='me'):
        return self.request('user/%s/profile'%user_id)
        
    def user_coworkers(self,user_id='me'):
        return self.request('user/%s/coworkers'%user_id)
        
    def user_favorite_coworkers(self,user_id='me'):
        return self.request('user/%s/favorite-coworkers'%user_id)
        
    def user_favourite_coworkers(self,user_id='me'):
        return self.user_favorite_coworkers(user_id)  
    
    def user_projects(self,user_id='me'):
        return self.request('user/%s/projects'%user_id)
        
    def user_favorite_projects(self,user_id='me'):
        return self.request('user/%s/favorite-projects'%user_id)
        
    def user_favourite_projects(self,user_id='me'):
        return self.user_favorite_projects(user_id)
               
    def user_recent_documents(self,user_id='me'):
        return self.request('user/%s/recent-documents'%user_id)
        
    def user_avatar(self, user_id):
        # TODO - think that this request should not go through 
        # the authenticated service. Get a 401 error.
        return self.request_binary('avatar/%s/%s'%(user_id,self.access_token_key))

    def user_assignments(self,user_id='me'):
        return self.request('user/%s/assignments'%user_id)
        
    def latest_user_conversations(self,user_id='me',complete=None,count=None):
        request = 'user/%s/conversations'%user_id
        parameters={}
        if complete:
            parameters['complete'] = complete
        if count:
            parameters['count'] = count
        return self.request(request, **parameters)
        
         
    def user_document_feed(self,user_id='me'):
        return self.request('user/%s/document-feed'%user_id)
        
    def project_members_list(self,project_id):
        # Currently getting a 403
        return self.request('project/%s/members'%project_id)        


    def document_comments(self,document_id):
        #todo: figure out how to POST
        return self.request('document/%s/comments'%document_id)  

    def document_properties(self,document_id):
        #todo: figure out how to POST
        return self.request('document/%s/properties'%document_id)    

    def document_touch(self,document_id):
        #todo: figure out how to POST
        return self.request('document/%s/touch'%document_id)        
        
    def document_upload(self, document_container_id, document, **kwargs):
        document_container = document_container_contents()
        #todo: figure out how to POST
        raise NotImplemented('document_upload to Project Place has not yet been implemented')    
         
    def document_download(self, document_id, destination_filename=None, version=None):
        request = 'document/%s'%document_id
        parameters={}
        if version:
            parameters['version'] = version
        document = self.request(request, **parameters)
        if destination_filename:
            destination_file = open(destination_filename, 'wb')
            destination_file.write(document)
            destination_file.close()
        return document


    def document_versions(self,document_id):
        return self.request('document/%s/versions'%document_id) 
          
    def document_version_properties(self,document_id,version_id):
        #TODO POST
        return self.request('document/%s/versions/%s/properties'%(document_id,version_id))    
        
                        
        
    def document_container_contents(self,document_container_id=None,project_id=None):
        if document_container_id:
            return self.request('document-container/%s/contents'%document_container_id)
        elif project_id:
            return self.request('project/%s/documents'%project_id)
        else:
            raise BadValue('document_container_contents requires a document-container-id or project-id parameter')



    def project_conversations(self,
                              project_id,
                              older_than=None,
                              newer_than=None):
        request='project/%s/conversations'%project_id
        parameters={}
        if older_than and not newer_than:
            parameters['older_than'] = older_than 
        if newer_than and not older_than:
            parameters['newer_than'] = newer_than 
        return self.request(request,**parameters)
            
        
    def project_logotype(self, project_id):
        # TODO - think that this request should not go through 
        # the authenticated service. Get a 401 error.
        return self.request_binary('project-logotype/%s/%s'%(project_id,self.access_token_key))


    
    #---------
    # Higher order methods
    def project_id(self,
                   project_name,
                   user_id='me'):
        projects=json.loads(self.user_projects(user_id))
        for project in projects:
            if project['name']==project_name:
                return project['id']
                
   
    def container_id_at_path(self,
                             project_name,
                             path,
                             user_id='me'):
        
        #print 'in container_id_at_path'
        project_id=self.project_id(project_name = project_name,
                                      user_id      = user_id)
        #print "Project-ID:",project_id
        containers=self.containers(json.loads(self.document_container_contents(project_id=project_id)))
        for target_container_name in path:
            next_container = None
            for container in containers:
                 if container['name']==target_container_name:
                     next_container=container['id']
            if next_container:
                containers=self.containers(json.loads(self.document_container_contents(document_container_id=next_container)))
            else:
                raise ContainerNotFound('A container called "%s" was not found'%target_container_name)
        return next_container
            
            
    def containers(self,document_container_contents):
        return document_container_contents["containers"]

            
    def documents(self,document_container_contents):
        return document_container_contents['documents']
        
        

       
    def container_structure(self,
                            document_container_id=None,
                            container_name=None,
                            project_name=None,
                            user_id="me"):
        """
        Supply either a document_container_id if starting from a known container,
        or the project_name (and optionaly a user_id), if starting from the project
        root container.
        The method returns a nested list matching the container structure. This does
        not include files. Each node is a dictionary of 'name';'id' and 'containers'.
        """
        """the 'containters' returned in document_containter_contents looks like:
       
                    {u'container_count': 0,
                     u'name': u'f2',
                     u'versioned': True,
                     u'color': u'yellow', 
                     u'id': 707070707, 
                     u'document_count': 0}
            
           we want to add the document_containter_contents here in a new field called
           'contents'. This is done recursively to give us the whole nested structure.
           It's probably wise not to call this against a large structure (i.e. use
           container_id_at_path first)
        """
        if document_container_id and project_name:
            raise BadParameterCombination('Only one of document_container_id and project_name should be supplied to the ProjectPlace.container_structure method, not both')
        
        elif not document_container_id and not project_name:
            raise BadParameterCombination('At least one of document_container_id and project_name should be supplied to the ProjectPlace.container_structure method.')
        
        elif project_name and not document_container_id:
            project_id=self.project_id(project_name = project_name,
                                       user_id      = user_id)
            container_structure=json.loads(self.document_container_contents(project_id=project_id))
            container_structure['name']=project_name
            
        elif container_name: # document_container_id and not project_name:
            container_structure=json.loads(self.document_container_contents(document_container_id=document_container_id))
            container_structure['name']=container_name
            
        else:
            raise BadParameterCombination('A container_name must be supplied to the ProjectPlace.container_structure method when document_container_id is the parameter')

        for container in container_structure['containers']:
            container[u'contents']=self.container_structure(document_container_id=container[u'id'],
                                                            container_name=container[u'name'],
                                                            user_id=user_id)
                                                                                          
        return container_structure
       


I've not had any luck with POST,  mainly due to not having in-depth understanding of http and the oath package.  Hints would be most welcome!

Wednesday 27 June 2012

Songwriting

On Saturday, I spent a delightful afternoon with my twelve year old daughter helping her to record a song. I've dabbled in this a little in the past, but without any success.  I've owned a cassette based 4-Track, an Atari ST with a dodgy copy of Cubase. I had Cubase lite on a 486 PC with an expensive but ultimately crap sound card. I bought a full copy of Cubase on a Mac running OS9 and pretty much failed to get it do do anything. I dabbled with Soundtrack Pro and Garageband. I have Garageband and Intua Beatmaker on my phone.

I can use music software in the same way as I can use a camera: I know enough of the technical details to get by, but I am largely devoid of talent. Both musical and visual.

In photography, I learned that my wife has a natural eye for picture, but no interest in the technical aspects of the camera. We make a reasonable team. She'll give me some indication of what she wants to do, I adjust the camera, she frames the shot and presses the shutter release.

A similar thing happened with my daughter and her song. She wandered into the living room at about 1PM and announced that she had written a song. I somewhat condescendingly asked her to play it for me. She did … quite badly … stumbling over the lyrics and with far too much right-hand damping as she strummed.  What was interesting is that she'd pretty much mastered the few chords I'd taught her a while back, discovered a few more, and strung them together into two distinct sequences.

I suggested that we record it. I launched Garageband. I asked her to play the song while I figured out the tempo and then we looked for a drum pattern that matched reasonably well. I picked a simple MIDI bass pattern that went with both. We worked though the first chord sequence to adjust the pitch of the bass to match. We repeated this for the second chord sequence and spent a remarkably small amount of time adjusting the number of bars of each. We had the foundations of the demo sorted.

Next we recorded her playing the guitar into my laptop's built-in mic. The first attempt was aborted early when I could see that the signal was too weak.  I adjusted the recording level and we the first sequence again, this time successfully.

My daughter's strumming is a bit erratic. The timing and clarity aren't consistently good, but we were able to pick the best 4 bars and copy and paste these.  We did the same for the second sequence.

I was quite careful at this point not to criticise.  Had it been my song a couple of years ago, I'd have probably spent the rest of the day trying to capture a better guitar part.  I've been listening to the wonderfully prolific Billy Childish recently though, and I've come around to the idea that getting a lot of songs done roughly is far better than perfecting just a few.

Next up were the vocals. The laptop mic was used again. My daughter sang the whole thing start to finish in one take. No getting lost, no problem with the timing into and out of the bridge, finishing at the right point too.

We then spent a bit of time mucking about with the software. We put pitch correction up to the max, tried all the vocal presets, changed the tempo.  Her pitch was pretty good for the most part, but she liked the effect of the pitch correction, so we kept that.

The song began at 100bpm. Adjusting the tempo of audio tracks works best at multiple values. I ramped it up to 200bpm. "I like it like that!", she exclaimed. The guitar sounded fine, the the vocals did not.

We recorded the vocals again at 200bpm.  Again, just the one take. There are some mistakes, some practice and more takes would no doubt fix these problems, but we were enjoying ourselves too much to worry about them. I adjusted the levels a little, and we had ourselves a song.

Knowing that multi-tracking the vocals to add depth/chorusing/richness is a common technique, I un-muted the original (now playing at 2x) vocal to see how it would sound. We got a lucky. The timing was different, so it sounded like two distinct voices singing different but complementary things. With the original vocal's level reduced so that it's not fighting, we ended up with quite a cool result:

Bowser.mp3

The song is about an unloved character in the Mario games. For bit more fun, my daughter wanted pictures of the character in a slideshow along with the music.  Five minus in Google images and 5 more in Final Cut produced something she added to her Facebook page that she was delighted with.

I have an old mixer and a condenser mic.  I just ordered a mic stand and XLR cable. Hopefully we can take the song and record it again, but do a better job and take a little more care.  If she wants to to a rough job on entirely different song though, that's fine with me. Just like with my wife and the camera, my daughter can tell me what she wants to do, and I'll help out with the technical side until she can get the hang of it herself.

Thursday 17 May 2012

Syntax Highlighting

It's taken me ages to get around to adding Syntax Highlighting.  No idea why it took so long, but it's here now. w00t!

Tip of the hat to David Craft, who wrote the instructions I found and many thanks to Alex Gorbachev for the hard work.

Hex Dump

I'm off to Paris next week for a meeting. This has prompted me to investigate a minor bug in some code that was displaying "§" instead of "§". It's really just a cosmetic defect until a customer sees it. At that point, it makes me look like a careless dick.

My immediate thought was to look at the hex values in the file. The last time I did this was when I was using VAX/VMS systems. That was at least 6 years ago, but it may have been 10 years ago when I last looked inside a file in this way. 'hexdump' at the OS-X command line does the job, but not immediately in the way I wanted. I didn't bother to read the MAN pages, but decided to roll my own in Python instead:

#!/usr/bin/env python
#!/usr/bin/env python
# encoding: utf-8

import sys

WIDTH = 24

# §

def ascii(byte_value):
    if byte_value < 32:
       return {0:  'NUL',
               1:  'SOH',
               2:  'STX',
               3:  'ETX',
               4:  'EOT',
               5:  'ENQ',
               6:  'ACK',
               7:  'BEL',
               8:  'BS ',
               9:  'TAB',
               10: 'LF ',
               11: 'VT ',
               12: 'FF ',
               13: 'CR ',
               14: 'SO ',
               15: 'SI ',
               16: 'DLE',
               17: 'DC1',
               18: 'DC2',
               19: 'DC3',
               20: 'DC$',
               21: 'NAK',
               22: 'SYN',
               23: 'ETB',
               24: 'CAN',
               25: 'EM',
               26: 'SUB',
               27: 'ESC',
               28: 'FS ',
               29: 'GS ',
               30: 'RS ',
               31: 'US ',}[byte_value]
    elif byte_value < 127:
        return ' %s '%chr(byte_value)
    else:
        return '%03d'%byte_value
        
def hex_dump_line(bytes, index=0, width=WIDTH):
    if len(bytes)>0:
        hex_part = u""
        text_part = u""
        for byte_value in (ord(byte) for byte in bytes):
            hex_part += '%02X '%byte_value
            text_part += ascii(byte_value)
        print '%04X %s'%(index,hex_part)
        print '     %s\n'%(text_part)
            
def hexdumpfile(filename, width=WIDTH):
    file_to_dump = open(filename, "rb")
    byte = file_to_dump.read(1)
    bytes = []
    index = 0
    while byte != "":
        bytes.append(byte)
        if len(bytes)==width:
             hex_dump_line(bytes,index=index,width=width)
             bytes = []
        byte = file_to_dump.read(1)
        index +=1
    hex_dump_line(bytes, index=index, width=width)

if __name__=="__main__":
   hexdumpfile(filename=sys.argv[0],
               width=WIDTH)
The output looks like:

0017 23 21 2F 75 73 72 2F 62 69 6E 2F 65 6E 76 20 70 79 74 68 6F 6E 0A 23 20 
      #  !  /  u  s  r  /  b  i  n  /  e  n  v     p  y  t  h  o  n LF  #    

002F 65 6E 63 6F 64 69 6E 67 3A 20 75 74 66 2D 38 0A 0A 69 6D 70 6F 72 74 20 
      e  n  c  o  d  i  n  g  :     u  t  f  -  8 LF LF  i  m  p  o  r  t    

0047 73 79 73 0A 0A 57 49 44 54 48 20 3D 20 32 34 0A 0A 23 20 C2 A7 0A 0A 64 
      s  y  s LF LF  W  I  D  T  H     =     2  4 LF LF  #    194167LF LF  d 

This allowed me to see the '§' was encoded in the file as C2A7, and sure enough C2 is the unicode value of 'Â' and A7 is the unicode value of  '§'.   When I open the file in TextMate, I see just the '§', but in Safari's View Source, I see '§'.  It turns out that although I'm encoding correctly as UTF-8, I'd neglected to declare the encoding in the HTML . This fixes the problem:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>'

I'm not sure when I'll need to look inside a file again. Not for this issue, I hope. I'll recognise it and check the header first.

EDIT: I should've just looked in the app store. File Viewer is free and does pretty much what I wanted to do - quicker, better, prettier, etc.

Friday 23 March 2012

openpyxml win

I'm a software tester. I write and run a lot of test scripts. These scripts are usually written in Excel.

Excel feels much more appropriate than a dedicated test case management environment. It is a tester's tool, not a management tool. Its Achilles heel, however, is the fragility of inter-workbook linking. It just doesn't seem possible to create and maintain relative links for multiple users.

Using a separate summary/reporting workbook to show test progress requires quite a lot of link plumbing. This plumbing is a maintenance nightmare.

My current project is in its infancy. The shit and the fan aren't even in the same room yet. I have time to put things in place. Time to do something about the plumbing. I can't fix Excel, but I can rip out most of the troublesome plumbing.

I've had some success with Apache POI in the past (Java library for reading/writing Excel files). My Eclipse and Java skills are rusty, but my Python is not.

A little Googling gave me a few options. The first I tried was openpyxl. It worked, so I saw no reason to look beyond it.

The Test Scripts are all based on a template devised and maintained by the testers. This template has some rows at the top containing a live view of the state of the sheet (number of steps, number of passes, number of failures).

I've now reduced the plumbing to a few cells in a workbook that link to a few cells in a CSV. This CSV is written by a Python script that scans a folder and its subfolder for Excel files and uses openpyxl to scrape values from the results cells of test scripts.

If new scripts are created somewhere in the folder structure, these will show up in the CSV.

It's also possible to generate a summary at each level of folder hierarchy. That means we can create a meaningful hierarchy - say reflecting the requirements hierarchy - and easily get a summary at level.

Another benefit is that we can split long test scripts into smaller individual test cases. The Python that scrapes the results will simply find them all without any additional work.

Thursday 2 February 2012

Single Story, Television and Violet Blue

Last night, due to very bad WiFi in my hotel, I watched a few TED videos I'd downloaded in case the WiFi in my hotel room was very bad. One was just superb. It was 'The Danger of a Single Story' from Chimamanda Adichie. I'd never heard of her before, but just ordered a couple of her books on the strength of how much I enjoyed this:











It made me think of The Wire. Just as Chimamanda was stuck telling single stories in her early writing, the majority of TV is stuck telling single stories. The majority of characters in TV are superficial. One dimensional. They have a single story … good policeman … bent copper… drug dealer … guy in red shirt in away team to the planet surface.

Characters become multi-dimensional when we know more than one of their stories. I've always thought this is why The Wire is so good. Most the characters have more than one story. It's not possible to do this for all of them, of course. Some minor characters are never fleshed out. Some characters have to wait for several seasons to pass before getting more of their stories told. After this happened a couple of times, I stopped judging characters on their single-story - being patient for the reveal of a second, third … nth story.

I haven't always extended that patience in the real world, but it's something I shall endeavour to do, having thought about it in this way.


I'm going to have to consider whether this is also why I like The Killing and Borgen. I think it's why Skins and Misfits are so good - they present single-story stereotypes intially, but dedicate whole episodes to fleshing out each of the main characters by giving them additional stories.

This assumption of a single story is what's bitten Violet Blue in captioning "The Saddest Booth Babe In The World" ). I'm not going to judge her too harshly. She'd just responding to a single story in he way most of us often do. She's not doing herself any favours, however, by not reconsidering, not apologising, not learning, not growing.

A person at conference booth can be a young attractive women, with breasts in a tight t-shirt, representing herself and her work and her nerdery. A young attractive woman, with breasts in a tight t-shirt is not necessarily hired marketing eye-candy. She may be bored and tired and disillusioned. She may be struggling to overcome shyness or social awkwardness (I am imagining myself in a similar situation). She may be sitting there thinking "Fuck! I wish I'd hired a booth-babe. I suck at this!"