Skip to main content

Posts

Showing posts from June, 2009

Parallel python: destroying job_servers

Remember to destroy job servers after they are done.

My nice animation generator crashed sometime over the weekend with the following death rattle:

Traceback (most recent call last):
File "animate_array_job.py", line 168, in
tscale = tscale, dpi = dpi)
File "animate_array_job.py", line 92, in process_file
job_server = pp.Server(ppservers=ppservers)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 336, in __init__
self.set_ncpus(ncpus)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 494, in set_ncpus
range(ncpus - len(self.__workers))])
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 141, in __init__
self.start()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/pp.py", line 148, in start
shell=True)
File "/Library/Framew…

Parallel python : clusters

If the jobs are too short they may not get submitted to the remote machine and may all execute on local. pp submits jobs to remote machines only after running out of workers on local
File handles are opened on the remote machines.
You can't pass a file handle as a function argument
TypeError: can't pickle file objects

Parallel python

Among the many libraries for parallel processing in python I chose what seemed to have one of the simpler interfaces - Parallel Python.

There is one trick that is not apparent from the docs: The function you call has to be 'self sufficient'. You basically have to write up the function you call as if it were a script. So the function has to have the imports etc within itself.

This is not a biggie, but it gets confusing until you figure it out.

Original code:

# module x -----------------
import a
import b

def f1(g):
f2
f3

def f2():

def f3():

# Calling script
import x

x.f1(2)
x.f1(4)
x.f1(6)
x.f1(8)


Parallelized code
# module px ------------------------------
def pf(g)
import x <------ NOTE THIS. ALL IMPORTS AND OTHER FUNS HAVE TO BE WITHIN THIS FUN
x.f(g)

# New calling script
import pp
import px

ppservers = ()
job_server = pp.Server(ppservers=ppservers)

jobs = []
jobs.append(job_server.submit(px.pf, (2,))
jobs.append(job_server.submit(px.pf, (4,))
jobs.append(job_server.submit(px.pf, (6,))
jobs.append(j…

Matplotlib interactive plotting pads figure size

With interactive backend (eg Tk)


f = pylab.figure(figsize = (7,10))
f.get_size_inches() -> array([ 7.175, 10.175])


The saved pdf is also of this size.

It becomes extreme with smaller figures:


f = pylab.figure(figsize = (1,1))
f.get_size_inches() -> array([ 3.275, 1.175])


This last I can see as a function of getting the tool bar to fit.

With Agg the saved figure is the size you ask for

Only on wikipedia:

Reading a paper about neural circuitry. Came across the word 'resonant circuitry'. Looked it up on wikipedia. Knew it had to do with band pass circuitry. Looked that up on wikipedia. Read in that entry that Thomas Pynchon claimed the character Kilroy was based on the circuit diagram for a band pass filter. You be the judge.



Any hoo, went to Pynchon's page on wikipedia and started to read. On the entry for his novel V, read about yoyodyne corp. Hey I know that from the GPL. Went to the (fictional) webpage for yoyodyne corp... Wait. Whaa!? What's all thsi? What's the time? Where am I? What am I doing here? Wasn't I just reading a paper on neural circuitry?

Sourceforge community choice awards

So I voted as follows:

KeePass Password Safe for Best Project (I would have liked matplotlib in the list)
eViacam for Best New Project
matplotlib for Best Project for Academia
Audacity for Best Project for Multimedia (mencoder, if it was on sf, would have won mine)
OpenOffice.org for Best Project for Government
KeePass Password Safe for Most Likely to Change the Way You Do Everything (I would have liked matplotlib in the list)

I would have liked to see Matplotlib in the categories 'Best Project' and 'Most Likely to Change the Way You Do Everything' too, but KeePass is a good choice.

I didn't vote in a category when I didn't know any of the choices. eViacam is a mistake. I tried to get rid of it but couldn't cancel that vote. I picked it at first even though I knew nothing about it because I thought it was a good idea for a project, but later decided it wasn't right to vote on a project I haven't used.

Efficient pickling in python

Use the highest protocol you have available - which will usually be binary. So as example:
import cPickle
import pylab

z = pylab.random((10,10))
z ndarray 10x10: 100 elems, type `float64`, 800 bytes
len(cPickle.dumps(z)) -> 2371
len(cPickle.dumps(z, cPickle.HIGHEST_PROTOCOL)) -> 934

z = pylab.random((50,50))
z ndarray 50x50: 2500 elems, type `float64`, 20000 bytes
len(cPickle.dumps(z)) -> 55586
len(cPickle.dumps(z, cPickle.HIGHEST_PROTOCOL)) -> 20134
For some reason, I thought the default was to use the most efficient protocol, but it isn't.

cPickle.dump?
Type: builtin_function_or_method
Base Class:
String Form:
Namespace: Interactive
Docstring:
dump(obj, file, protocol=0) -- Write an object in pickle format to the given file.

See the Pickler docstring for the meaning of optional argument proto.
Hence my slow processing and bloated data files...
Note that load figures out the protocol automatically...

3D in matplotlib

One of the frustrating things about matplotlib is its lack of 3D plotting. If matplotlib were a mediocre library it wouldn' hurt. But it is a GREAT library and produces really good looking plots. So the lack of 3D plotting sometimes makes me want to go out and strangle somebody. There has however always been a furtive 3D plotting feature in matplotlib, and apparently, its being worked on some more. Yeehah! No more futzing with pyVtk.

Important links:
Docs for 3D features
Installing the svn code (that has all this goodness)

Drawing circles using matplotlib

Use the pylab.Circle command

import pylab #Imports matplotlib and a host of other useful modules cir1 = pylab.Circle((0,0), radius=0.75, fc='y') #Creates a patch that looks like a circle (fc= face color) cir2 = pylab.Circle((.5,.5), radius=0.25, alpha =.2, fc='b') #Repeat (alpha=.2 means make it very translucent) ax = pylab.axes(aspect=1) #Creates empty axes (aspect=1 means scale things so that circles look like circles) ax.add_patch(cir1) #Grab the current axes, add the patch to it ax.add_patch(cir2) #Repeat pylab.show()

Modifying a firefox plugin

I dislike the indistinct icons in the cookie manager plugin CSlite.

In Firefox/extensions/{00084897-021a-4361-8423-083407a033e0}/ (the directory for this particular extension) we find chrome/cookiesafe.jar

From a hint in a thread here, we learn that the .jar file is just a .zip file, and we can extract that to find skin/classic/cookiesafe and a whole bunch of .gifs there.

We can then change those gifs and change the icons that are used in the application.

CS Lite

I was using the Permit Cookies extension for firefox, but it hasn't been updated for a few cycles of firefox updates. It works fine, but I started to look for something else. I found CS lite which has a more convenient interface (a popup menu from the status bar). The only gripe I have so far is that the icons in the status bar are not very clean.

Searching for a cheap graphics card that will play Crysis

Downloaded and started playing the Crysis demo. On my integrated graphics MoBo, naturally I had to set everything down to low, but even then it can get choppy when in the jungle. And the graphics look so crappy I think - hmm, maybe I'll just play Far Cry. Tom's hardware has a great page where you can compare the performance of various cards with various games at various settings when searching for a cheap graphics card for crysis.

Verizon FiOS 20/5 serivice over wireless

From other parts