Exploiting Python's logical evaluation order for conditional evaluation in list comprehensions

My head is hurting slightly over this one. First I'll show the experimentation (edited!)

>>> if 'arse':5
... 
5
>>> if '':5
... 
>>> 
>>> if 'arse'==True:5
... 
>>> 'arse' and 'arse'
'arse'
>>> 'arse' and 'flange'
'flange'
>>> 'arse'==True
False
>>> a = [1,2,3,4,5]
>>> b = [4,4,4,4,5]
>>> c = [x==y for x,y in zip(a,b)]
>>> c
[False, False, False, True, True]
>>> a = [(x<y and x+1 or y) for x,y in zip(a,b)]
>>> a
[2, 3, 4, 4, 5]
>>> a = [(x<y and x+1 or y) for x,y in zip(a,b)]
>>> a
[3, 4, 4, 4, 5]
>>> a = [(x<y and x+1 or y) for x,y in zip(a,b)]
>>> a
[4, 4, 4, 4, 5]
>>> a = [(x<y and x+1 or y) for x,y in zip(a,b)]
>>> a
[4, 4, 4, 4, 5]

I then spent a little time reading about logical operator and precedence in my Python Pocket Reference, because this was messing with my head. In x and y, if x is false, y is not evaluated. If x is true in some form, then y gets evaluated. If y happens to be a string, then the evaluation of y is returned, which is a string.

What we have then is a sort of trick if:else: statement. By writing…

condition and x or y

…we're achieving the same as…

if condition:
    x
else:
    y

This is remarkably handy stuff. Especially (as shown above) when used in a list comprehension. I'm not sure that it's particularly readable, but it's certainly more concise than the code I had previoulsy. More importantly, it works too (the previous code did not). The result is this (names changed to protect the innocent), which increments the values in a list, but only up to the corresponding limits specified in another list:

counters = [counter<boundary.upper and counter+1 or counter for counter,boundary in zip(counters,boundaries)]

The code I had in place prior looked something like this:

newList=[]
for counter,limit in zip(counters, limits):
    if counter < limit.upperLimit:
        newList.append[counter+1]
    else:
        newList.append[counter]
counters=newList

Python: str * int … WTF!? … Thinks … Ah!

I was reading Guido's History of Python blog this morning. In it he mentions this :

Passing an invalid argument, for example a string, is generally caught quickly because few operations accept mixed string/numeric operands (the exception being multiplication)

WTF!?
Rushes over to Terminal...

>>> 'string' *2
'stringstring'

Why the hell would _anyone_ want to do that ?

…Thinks…

…Realise that I have some code that can be simplified with this…

indent=''
for space in range(level):
   indent += ' '

…quick check for zero case…

>>> 'string' *0
''

…changes code…

indent = ' ' * level

Python lesson 1 learned for the day.

Wolfram|Alpha. I guess this explains the 'Alpha' part

Today I wanted to convert something, and thought maybe it was the sort of thing Wolfram|Alpha would be good at, and a better starting point than Google. Converting from European shoe sized to UK shoe sizes, to see if some shoes I like the look of (but will probably not buy if I'm honest), are made in my size.

If you type a query about shoe sizes in, it probably won't know what to do with it, but it will have a link to shoe sizes and some example queries.

One of the examples happens to be "U.K. men's size 11 shoe in Japanese size". I clicked this and then changed it to "European men's size 47 shoe in U.K. size". It borked (Wolfram|Alpha isn't sure what to do with your input). Changing it to "U.K. men's size 11 shoe in European size" also made it bork.

I guess this explains the 'alpha' part of the name.

BTW, I really want to like Wolfram|Alpha, but their natural language parser just doesn't work well enough yet to allow people to get at the data.

Blogger, spaces, &nbsp; and <pre>

OK, So I've been posting bits of code of late, and Blogger buggers up the spacing. It removed leading spaces and trims multiple spaces to single spaces. This is probably good for the average blog post, but really, really bad for indent dependent Python.

I finally got around to Googling the problem. I'll repeat it here, just in case this helps someone else.

This is how I was doing it until today:

<code>
def double(x):
&nbsp;&nbsp;&nbsp;&nbsp;return x*2
</code>

But now I know about <pre></pre> and I can write this instead:

<pre>
def double(x):
    return x*2
</pre><
To show this:

def double(x):
    return x*2

Much better !

Python : Using **arg and setattr to create instance variables at runtime

I found that I need the generic/extendable **args, but I also needed those args easily visible to a method in a subclass. Initially I just passed the **args dctionary into a named parameter, but that seemed messy when I actually came to use it. Plus I need this usable by (even) less able programmers.

Here's in essence what I've done:

>>> class A(object):
...   def __init__(self, **args):
...     for key,value in args.iteritems():
...       setattr(self,key,value)
...
>>> a=A(first = 1, second = 2, name="Monty")
>>> a.first
1
>>> a.second
2
>>> a.name
'Monty'

There's a little more to it. I have checks against the existing names in case I overwrite something, for example.

Python : Instantiate nested classes in a superclass, not the direct containing class

I'm using classes for parts of the test harness that I'm writing for some functional testing. I'm writing I want the tests themselves to be fairly clean, fairly free of boilerplate etc, because some of the people writing the test have not done much or any coding before.

I have a TestCase class and I have a TestStep class. When a new test step is written, TestStep is subclassed and included in a subclass of TestCase.

One of the things I wanted to avoid doing was having to get the test writer to instantiate the TestSteps. I don't think this is possible in many languages, but Python lets me do it with its ludicrous ability to introspect and add stuff to objects at runtime.

class NewTestCase(TestCase):
    class NewTestStep(TestStep):
        def body(self):
            ...test step code goes here...
    def body(self):
        stepNewTestStep() #call instance of NewTestStep

Note that there's no stepNewTestStep = NewTestStep() anywhere, and I can add as many TestSteps as I like to the class. stepNewTestStep is not hard coded into the TestCase.

I'd considered naming the NewTestStep objectsas newTestStep rathe than stepNewTestStep, but I was already getting duplicate names in the tests that I'd written, so decided to add the explicit 'step' prefix.

To a Python newbie like me, this is a bit WTF!?, but stick with me. As ever, real Pythoneers, feel free to tell me of better, more Pythonic ways to do this. Here's how it was done…

import types
class TestCase(object):
    def __init__(self):
        for name in dir(self):
            attribute = self.__getattribute__(name)
            if type(attribute) = types.TypeType: # ignore any non classes
                for base in attribute.__bases__:
                    if 'TestStep' in base.__name__: # only interested in TestStep classes
                        setattr(self, 'step%s' % attribute.__name__, attribute())

In plain English, get the names of all the things in TestCase/NewTestCase. For each, make reference to the object of that name (__getattribute__), see if it derived from TestStep and then create a new instance of it (setattr), called 'step[ClassName]' as part of TestCase/NewTestCase.

Note that NewTestCase inherits the __init__ of TestCase, so dir(self) returns the names of everything in NewTestCase.

Python : Iterating Multiple Lists

I did this a few times...

for index in range(len(aList)):
    doSomethingWith(aList[index], blist[index])
    doSomethingElseWith(cList[index])

Or even...

for a in aList:
    doSomethingWith(a, blist[aList.index(a)])
    doSomethingElseWith(cList[aList.index(a)])

And at some point, it felt wrong. It felt, I have come to understand, unPythonic.
Google was my friend, although I didn't really look or read enough, and I ended up with something like this:

a=0
b=1
c=2
for abc in zip(a,b,c):
    doSomethingWith(abc[a], abc[b])
    doSomethingElseWith(abc[c])


And I was happy. But putting it in a few more places, it also felt like unPython. So I decided to take a better look (actually just a better more explicit simple search for exactly what I wanted), and found the Pythonic way...

for a,b,c in zip(aList,bList,cList):
    doSomethingWith(a.b)
    doSomethingElseWith(c)

Refactoring to use this idiom has been wonderful. Lines of code disappearing and the code becoming more readable too. Marvellous !

(I think I found my answer at testing reflections

Pythonic

I'm not a very Pythonic Python programmer yet, but it's getting better. The web has oodles of useful hints and tips. I'm adding stuff here that I find useful as much for me as anyone else.

I don't think yesterday was the first time I came across the term 'Pythonic', but it's the first tie it really registered. It's a nice landmark on my journey.

'Pythonic' just means doing things in Python the way a real Python programmer does them, rather than doing it Python in the way that it's normally achieved in another language. I've mentioned before that I'm a couple of degrees of separation from being a real Python programmer. I'm from an Ada background, and came to Python via Java.

Ada is a superb language. It's like a polar opposite of Python, and that's fine. I wouldn't use Python for safety critical, and I wouldn't use Ada to write automated functional tests. Right tools for the job and all that.

Edit: worth a read : What is Pythonic ?

Example of an exception class in Python

If I'm going to suggest that you define your own exceptions, I really ought to give an example:

class PlayerError(Exception):
    pass

class PlayerIsNotPlayingInExpectedDirectionError(PlayerError):
    def __init__(self, direction):
        """direction parameter should be 'forwards' or 'in reverse'"""
        if 'rev' in direction : direction = 'in reverse'
        self.direction = direction

    def __str__(self):
        return "Player was not playing %s as expected." % self.direction


Define and raise exceptions liberally.

It does the job, but this doesn't really feel like it's enough:

  if not playerIsPlayingForwards():
      raise "Player is not playing forwards'

I'm using this structure instead, liberally raising my own verbose exceptions :

  try:
      assert playerIsPlayingForwards():
  except AssertionError:
      raise PlayerIsNotPlayingInExpectedDirectionError('Forwards')

It takes a bit more time, but it's worth it. You get it back in debug time saved. You also get a nice warm fuzzy feeling when one of your explicit exceptions gets raised, highlighting a very specific problem.

Finding the name of the current function in Python. nameOfThisFunction()

As part of my testing, I need to create logs that give me some traceability for when things go wrong, evidence of testing whatever the result. Kind of like permanent debug text. I'll look at differences between log files of the same tests run on previous and new versions. It's ad-hoc testing when it's first written and it's regression testing thereafter.

One of the ways to show trace is simply to show which methods/functions are being called. Being lazy, I simply want to cut and paste something, I don't want to also have to change it. I would prefer to use the actual function name rather than hard coding into a string for printing / logging. The name might change in the future. I might forget to alter the name after pasting it.

This is what I don't want:

def testStepA(param):
    writeToLogFile('Reached function 'testStepA')
    ...

def testStepB(param):
    writeToLogFile('Reached function 'testStepB')
    ...

I want something like this instead:

def testStepA(param):
    writeToLogFile('Reached function '%s', nameOfThisFunction())
    ...

def testStepB(param):
    writeToLogFile('Reached function '%s', nameOfThisFunction())
    ...

Fortunately, inspect gives me the information I'm after:

import inspect

def nameOfThisFunction (offset=0):
    return inspect.stack()[1 + offset][3]


Handling exceptions and printing a useful trace in Python

In writing a test harness for functional automated testing, I'm expecting things to fail. In writing some of the tests, I'm trying to get things to fail, poking around the boundaries etc. WhenI have a test suite that takes 50 hours to run, that I kick off on a Friday night, what I don't want is for it to stop running five minutes after I leave the office.

I want it to carry on with the rest of the tests, but I also want some information about these failures. What exceptions were raised and where they were raised. This is how I'm doing it:

import sys
import traceback
…
try:
    myTestCase.executeTestSteps()
except:
    # handle and log exceptions
    exctype, value, trace = sys.exc_info()[:3]
    extracted_tb = traceback.extract_tb(trace)
    printAndLog("**********************');
    printAndLog(str(exctype) + ':' + str(value))
    for tb_line in extracted_tb:
        printAndLog(str(tb_line[0]) + 'line ' + str(tb_line[1]) + ' in ' + str(tb_line[2]) + ' (' + str(tb_line[3]) + ')')
    printAndLog("**********************');

    del trace

( Where myTestCase is the thing I want to run and printAndLog (you'll have to roll your own) sends to the console and to a log file)

Profit per head. Tech stock

The $210K profit per head at Google figures are interesting (as John Gruber points out).

Apple's not known for talking small profits and Microsoft pride take pride associating themselves with cheap so what's going on here ?

I have a few initial thoughts…

1 - Microsoft doesn't have stores. Although the Apple stores must bring in lot of revenue, they're also a massive overhead.

2 - Apple isn't in enterprise, and this is where Microsoft gets its profit. Money for old rope with Office and Exchange and Windows volume licenses.

3 - The data is based on 'employees', but that's not the same as staff. It probably excludes contract staff. Those companies using lots of contractors might be showing artificially high numbers using this measurement. I've no idea whether Microsoft, Google or Apple uses a lot of contact staff (I suspect not), but I bet a few of the companies listed do.

Python: count of parameters in a function

I recently had a problem with a that class had several ways to populate one of its instance variables. An instance method was called in each case, either with a parameter, (using the method provided by the class), or without a parameter, (in the case of the same method overridden in a subclass). I've since realised that there was a simpler/better way to achieve this, but no matter, Python allowed me to figure out which one to call by being able to count the parameters in the method, and then call

import inspect

def parametersInMethod(method):
    return len(inspect.getargspec(method)[0])

Allowed me to do this...

if parametersInMethod(self.populateInstanceVariable) == 0:
    self.populateInstanceVariable()
else:
    self.populateInstanceVariable(withValue)

This give a little insight into 'inspect' and also shows how methods being objects allow them to be passed as parameters into functions.

More on Python and a bit on Objective C

I can't quite put my finger on it, possibly because I've not used ObjC in anger yet, but it seems to have a similar ethos to Python. I'm probably going to ignore ObjC for a while longer, but I may dabble with PyObjC instead. Learn some Cocoa through the increasingly familiar Python, then apply that knowledge to an ObjC project.

What I love about Python so far is that just trying things seems to usually just work. It's a language that does what you want it to. It's a little dangerous, because so much is checked and can be manipulated at runtime. It's the least nanny-ish language I've come across, trusting the programmer to do the right thing and allowing the programmer to break convention when it is appropriate. It's counter-intuitive, perhaps, but the responsibility given to the programmer seems to lead to responsible programming. Treat me like an adult and I will behave like an adult kind of thing.