Skip to main content


Showing posts from 2011

The "page flash" on the kindle

The "page flash" on the Kindle 4 - the sudden turning of all the pixels black and then white, before the page is displayed - occurs every 6 page turns. It seems to be necessary because a certain amount of 'dirt' accumulates inbetween - bold letters (and also normal text) seem to leave traces pixels around the edges of the letters after they have been wiped after a page turn. The 'flash' wipes these pixels out and makes the page look crisp again.

I don't know if this is a flaw in the driver software, or a shortcoming in the e-ink display itself (that leaves the traces).

Quick state machine in Python

SM = { 'INTERTRIAL': {'TRIALSTART': 'TSTART0'}, 'TSTART0': {'TRIALSTART': 'TSTART1'}, 'TSTART1': {'TRIALSTART': 'INTRIAL'}, 'INTRIAL': {'TRIALEND': 'TEND0'}, 'TEND0': {'TRIALEND': 'TEND1'}, 'TEND1': {'TRIALEND': 'INTERTRIAL'} } nextstate = SM[currentstate].get(input,currentstate)

matplotlib installation

sudo make installlibpng (
sudo make installsudo easy_install matplotlib

Changing how numpy arrays are printed

Use numpy.set_printoptions. One of the things I like most about pylab (and numpy) is this flexibility, especially the summarization feature, which, when the array is very large, prints ellipses (...) instead of clobbering you with lines and lines of output. I like to set the threshold low (default is 1000) and I do this with pylab.set_printoptions(threshold=10)

Mac OS X install pytables and h5py

Install tables - need NumPy version 1.6Get NumPy from sourceforge and install - need Python 2.7Install python 2.7 on Lion, open new terminal (or refresh path)curl | pythoncurl | pythonsudo pip install ipythonsudo pip install tablesneed numexpr > 1.4.1, Download anc ompile numexpr -> wants to compile using gcc-4.2sudo ln -s gcc gcc-4.2sudo pip install cythonGet HDf5 from and compileCopy the hdf5 folder whereever you wantpython build --hdf5=/path/to/hdf5 (from the unzipped source of h5py)sudo python install --hdf5=/usr/local/hdf5/ (in the unzipped dir of pytables) In contrast, to get h5py working on Ubuntu:

sudo apt-get install libhdf5-serial-devsudo easy_install h5py

Dealing with strict key checking for SSH


While using sftp (or some other ssh related service) the following warning shows up and we can't use the service:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is b0:a8:eb:30:ce:1a:0e:6a:4d:7a:6b:3a:0a:c6:27:60. Please contact your system administrator. Add correct host key in /Users/xxx/.ssh/known_hosts to get rid of this message. Offending key in /Users/xxx/.ssh/known_hosts:28 RSA host key for has changed and you have requested strict checking. Host key verification failed. Connection closed

Remove the key by doing
sed -i "28 d" ~/.ssh/known_hosts

We kno…

Mac OS Lion

Things I like
New tab in terminal opens in same folderPreview can make pdfs by stringing msic documents together I don't like
Preview is sooo sluggishPreview messes up some transparency in pdfsNo save as (gotta duplicate etc etc)

++i OR i++ ?

From discussions on stack overflow we have:
++i does not expect to return a value while i++ might need to return a value.A modern compiler, for simple data types (i.e. integer) will treat the two the same if the context is right e.g. in a for loopFor an object (e.g. an interator), however, ++i will be faster than i++ because a compiler cannot optimize away the creation of a temporary object for i++

How to get file creation time on Mac OS X (64 bit)

QFileInfo.created() on POSIX systems (like Mac OS X) returns the last modified time.This is annoying because on Mac OS X you know the system (e.g. finder) has access to the actual creation time.Turns out that BSD systems (like Mac OS X) have an extension to sys/stat.h that contains the file creation time. The recipe, then is (shown using the QT framework):
#include <QtCore> #include <sys/stat.h> int main(int argc, char *argv[]) { QString fname("/Users/kghose/Sandbox/ChhobiTest/2005/2005_07_10/MVI_0693.AVI"); QDateTime thedatetime; struct stat64 the_time; stat64(fname.toStdString().c_str(), &the_time); thedatetime.setTime_t(the_time.st_birthtimespec.tv_sec); qDebug() << thedatetime; }

Income inequality

I don't deal with political topics here. But there were some thoughts I had on this fad going around on income inequality.

If you take a look at this report by the economist the claim is that while the income disparity between 99% of the people in the US hasn't changed much that between the 99% and the 1% has changed a lot.

I can see in my mind's eye some person or persons with a definite political agenda sitting at their software and moving a slider control. They first split the population 50/50 and grunt with dissatisfaction. They move it to 25/75. Still not so good. Then they get all excited and move it all the way to 1/99. Yay! Look at THAT income disparity.

I have a few concerns about this graph. Perhaps someone can enlighten me?
If I took the top 1% of a distribution MOST measures I would make using that 1% would be biased in some extreme way. There is no way I could stop that, because I took an extreme part of the population to start with. I'm taking a few sample…

Use to insert images as pages into a pdf

Open image in, "print" and save as pdfOpen pdf you want to add toOpen the pages paneDrag and drop the image pdf you just created into the pages pane (in whatever sequence you want it) will show the pdf to be "edited"Save the pdf You can directly drag and drop the image file from the Finder into the pages pane. This will, however, add a white border round the image.

Export display from Linux to Mac

From here
On client Mac startup and check preferences->Security->"Allow connections from network clients"Open an xterm from the menussh into host machine with the X flag (from the xterm)
ssh y.y.y.y -X -luserEverything works (even with firewall on), no need to set DISPLAY etc.

Installing ubuntu (11.10)

New HDD was cut into two partitions and win xp was installed on one.Ubuntu was installed off a USB stick "Install alongside Win XP" was chosen. This correctly used the empty (unformatted) partition for Linux, with appropriate swap space choice.Ssh needs to be installed - can be installed through the package manager. I used a metapackage to install both client and server. The changes took effect immediately (no restart or manually starting server). I went from not being able to ssh into the machine from my mac to being able to.Python 2.6 comes installed sudo apt-get python-pipsudo pip install ipythonsudo apt-get install python-matplotlibThe Unity Launcher cannot be configured to be at the bottom. Idiots. sudo apt-get install aptitudeinstall kubuntu-desktop via aptitude (picked kdm as display manager)Muon package manager has authentication problems (can't authenticate - Muon creator says it is a policykit problem). IdiotsUninstall KDE and go back to Unity: follow the instr…

Migrating from a 10.5 macbook to a 10.7 macbook pro

Most impressed by migration: just needed to hook up my time machine disk to the new machine in in 2 hours my WHOLE computer was transplanted onto the new mac, including applications and frameworks.When I went to run eclipse (Helios) I was asked to install java runtime, which the OS installer found and installed by itself.The fast login switching, not so useful for single user laptops and also takes up menubar space with your name can be removed from system preferences (here).Spaces threw me: from here, we see that the way to add new spaces is to go into misson control (middle click) and then move the pointer to the top right hand corner to add a new desktop.To use the existing time machine backup (so you keep your history etc), however, all the files seem to be copied over afresh - so you get a new snapshot which uses a lot of space - the first new backup is not incremental.The screen looks different - the colors are brighter - but perhaps my old macbook's lcd was just fading.Two …

Deus Ex HR

One of the few games I ever bought when it came out (thus at full price. That's how much I wanted it)Tossing a cardboard box at armed NPCs now leads the bastards to gun you down mercilessly, rather than simply yell at and berate you like they used to do in DEYou can drag a body through the vents.Immersive side-story itemsImproved dialog system much like RPGYou can sell stuff you don't need for cash - so I sell all the lethal weapons, which I don't use.  A LOT of attention has been paid to detail and atmosphere, one of the few games I would recommend to get at full price to reward the creators. If you liked Deus Ex, you will definitely enjoy this one, which has update graphics and some enjoyable new augs.

Travelling on advance parole from India to the US through Frankfurt

The person at the Lufthansa counter accepted the combo EAD/AP cardThe immigration official at Indian customs examined the card closely, looked at my expired H1-B visa stamp, asked me if they would let me in with this card (I said yes) and then let me passAt Frankfurt, I was asked if I have a visa, I said "Yes, a German visa" (I had a one year multiple entry German visa). She did not flip through the passport to actually find the visa, she just let me through after looking at the front page. Current regulations state you need a transit visa if you are on AP and are not from one of the visa waiver countries.At US immigration I was asked to fill out the white I-94 card and sent to secondary inspectionAt secondary inspection the lady asked a few questions about my specialization and then stamped the I-94 card as "DA/AOS" and let me through. She also "Cancelled Without Prejudice" my expired H1-B visa stampI did not have to show any documents other than my pass…

Latex: Pushing floats to the end of a document

The endfloat package is kind of fancy but the problem I faced with it is that I was renumbering figures in the middle of the document and the latex referencing algorithms were handling it fine, but when the figures appeared at the end of the document their numbering was screwed up.

There is an easy way, however, to push the figures to the end of the document as I wanted. From some hints here about how LaTeX decides whether there is enough space on a page to put in a figure


in the preamble does the trick, by fooling LaTeX into deciding there is NEVER enough space to put figures on a page until it is all out of text and has to just dump the figures as best as it can, one to a page.

Latex + Unicode = XeTeX

Never bothered to figure out what this xetex thing was. Turns out it can handle unicode. For texmaker just replace pdflatex command with xelatex and use a unicode aware font (e.g. \usepackage{helvet}) and things should work

migrating existing svn repo to git (and use with eclipse)

Install gitsudo gem install svn2git --source (here)svnserve -d (start readonly svnserver on local machine)mkdir new directory and cd into it svn2git svn://localhost/path/to/local/svn/repoInstall EGit and restart. (Eclipse) new pydev project ->  browse to existing folderSet up git locally (here)set up to use gitteam -> remote -> push -> (url from your project page)Notes:
git svn clone file: did not work because of a file format problem (Expected FS format '2'; found format '4' at /usr/local/git/libexec/git-core/git-svn line 1535)

small issue with hist

The left is the matplotlib plot - notice the slight rightward shift of the drawn histogram with respect to the solid black line.

The right is the pdf produced from the same plot, which is exactly where it should be.

Same deal, but with 'Agg' backend and printed to png

LyX + Mac + Math heavy document = keystone cops

LyX has a really nice online math type setting system. I have a math heavy document that I wanted to type up using LyX. Pretty soon I had the classic problem of any feedback system with feedback delay - oscillations. I would type something, it would take about a quarter of a second to update, I would move my mouse to click on a formula, it would take a quarter of a second to change state and by that time I would have clicked twice, the document would have scrolled, and the formula would change state again, because I had clicked somewhere else.

Welp. Its back to good old latex and text editors for me. I don't mind the write-compile cycle.

LyX is good for text documents, but for math, at least on mac, it is so sluggish, it is more annoying than anything.


I have found texmaker to be a very nice IDE on mac.

Tracking down backup space hogs

Problem: Even though I haven't done many file changes the Time machine incremental backup is about 350 MB per flush (resulting in many GB per day for hourly backups). I don't generate THAT much work each day.

Using Time Machine and Time Tracker to track down the culprits.


global-messages-db.sqlite (170.5 MiB)


urlclassifier3.sqlite (50.6 MiB)
Cache 78.2 (MiB)

ipython + matplotlib crashing

I've been having this problem ever since I changed to 1.0.1. It's not clear if it is ipython's fault or matplotlib's fault.

Today I got the first error message before segfault:

objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470
And Mac OS X error report:

Date/Time: 2011-07-15 21:47:54.500 -0400 OS Version: Mac OS X 10.5.8 (9L30) Report Version: 6 Anonymous UUID: 655916BF-D521-48C5-B8A3-FBC5E5FA6683 Exception Type: EXC_BAD_INSTRUCTION (SIGILL) Exception Codes: 0x0000000000000001, 0x0000000000000000 Crashed Thread: 0 Application Specific Information: objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470 Thread 0 Crashed: 0 libobjc.A.dylib 0x90595bfa _objc_error + 116 1 libobjc.A.dylib 0x90595c30 __objc_error + 52 2 libobjc.A.dylib 0x90594637 _freedHandler + 58 3 0x93faa…

Making up a GET request in bottle

Problem: I'm using bottle for a desktop browser based app. I needed to call a page view from an action. The page view normally gets some parameters encoded in a GET request.

Solution: The GET request can be faked by doing:
request.GET.append(key, value)
Since request.GETis just a dictionary.

subclipse, branching, merging etc.

Sometimes merge/branch will refuse to work, saying your resource is out of synch. Do an 'update' and then a 'commit'.After a rename or move + commit, subclipse will say resource is out of synch with the filesystem, just refresh,commit and continuepydev explorer does not show the .* project files. These need to be added to svnignore (or committed if you wish). Use the eclipse navigator to see these fileswitching a branch to be the trunk is annoying in svn: basically you have to move the folder over - so the current trunk has to be moved (possibly under branches with a new name) and the branch you want has to be moved over and renamed as trunk

Mac OS X MS Word and Equations

I like to compose my equations in LaTeX. I find it a lot faster than the click interface of MS Word's own equation editor. I type up my notes in a LaTeX format so I already have the equations and its a terrible inefficient duplication of work to hunt-n-click them into ms word.

One solution is to print a pdf of all the equations, and then use the select tool in preview to copy the equation and then paste it into the word document.

The advantage in mac os x is that the copy is in vector format, which means that the inserted equation does not behave as an image - becoming pixellated at the worst moments, but rather scales gracefully.

That's gonna save me some time.

Pickling python classes

Ok, this is the first thing in Python I have found to be annoying and nonintuitive. When you pickle a class object you need to make sure that the module for the class is explicitly in scope and imported.

I kept banging my head against this problem and only understood it after looking at this guy's blog.

When you pickle a class object included in the pickle file is a coded import statement telling the interpreter which module to import to look for the definition of the class. This leads to the following gotcha:

The following code will define a class, instantiate it and pickle it without problems

#File import cPickle class A: def __init__(self): self.x = 22 if __name__ == "__main__": a = A() cPickle.dump(a, open('obja.pkl','wb'), protocol=-1) print cPickle.dumps(a, protocol=0)
Notice the stringification of the class : it begins with i__main__
Now, we run the following code to load the object from the pickle:
import cPickle m = cPickl…

Hide the dock icon of a program when it is running

Go to the application's bundle (e.g. and open the bundle contents. Navigate to Info.plist, open it with a text editor or plist editor. Add or change an item with the key "NSUIElement" and set its value to "1" (type = string). On relaunch of the application, its icon will not appear in the dock.

This is usefull for apps like X11 which are servers and should be running in the background.

minidom.writexml handling unicode errors

Writing xml with unicode characters does not work, even with the encoding set to 'utf-8'. Always results in something like

UnicodeEncodeError('ascii', u'kr\xf6se1989', 2, 3, 'ordinal not in range(128)')

Solution (from here):
Use the codecs module and manually open the file in utf-8 mode for writing

import codecs
doc.writexml(,'wb','utf-8'), encoding='utf-8')

Compress camera video for a windows machine

mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=1:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o /dev/null MVI_0052.AVI
mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=2:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o output.avi MVI_0052.AVI

Python and word documents

From here, using only standard python modules:

import zipfile, re

docx = zipfile.ZipFile('/path/to/file/mydocument.docx')
content ='word/document.xml')
cleaned = re.sub('<(.|\n)*?>','',content)
print cleaned

But, if you want to mess around in more detail in the document, then we can use the python-docx module.

discount bug

import discount
a = discount.Markdown('')
--> Yields TypeError
TypeError: 'NoneType' object is unsubscriptable
Should return a blank.

Found that markdown or markdown2 give comparable performance, so now no longer using discount.

Using adminer on Mac OS X

adminer is a nice php based sqlite manager. I prefer the firefox plugin "sqlite manager" but it currently has a strange issue with FF5 that basically makes it unworkable, so I was looking for an alternative to tide me over.

I really don't want apache running all the time on my computer and don't want people browsing to my computer, so what I needed to do was:

Download the adminer php script into /Library/WebServer/Documents/Change /etc/apache2/httpd.conf to allow running of php scripts (uncomment the line that begins: LoadModule php5_moduleStart the apache server: sudo apachectl -k startOperate the script by going to localhostStop the server: sudo apachectl -k stop

Standalone browser based app

Perhaps my long search for a convenient way to code database frontends that work locally ("desktop app") but leverage the power of a browser as a GUI has come to a successful point. This point involves:
Python (Naturally)Bottle - a "microframework"py2app - py2exe for windowsThe only tricks making Bottle and  py2app work well together were
You need to explicitly list the template files when calling py2app, otherwise they don't get included in ResourcesI had to add a url route (which I called /quit) that called os._exit() to shut down the server (otherwise done using CTRL-C). just exit() and even SIGINT did not work. This is a bit extreme, but I think safe in this usage.I was happy to see that the created app used the Resources directory as the working directory and the packager put all the files in this directory making things very transparent and worry free.

3d wall decor

So, there are artists who paint patterns on walls that turn into a recognizable figure if you are standing at the right perspective (e.g. here).

One easy way to generate these patterns is to set up a projector with a slide of the pattern that you want projecting from the vantage point you want, and then paint according to the projected pattern.

Nikon D40 (almost) 2 years on

The D40 is an awesome camera, especially at the price I bought it ($375 with kit lens - but refurbished). It's years out of production now and has been superseded, but it does fine for my casual needs. The only two gripes I have about it, in order of importance are:

Auto white balance is sucky for incandescent lighting. So I always switch from auto to preset when I shoot indoors.Only one func button makes it awkward. I have to pick between select ISO and select WB for that button. I think select WB wins out, but it would be nice to have dedicated buttons for that.

Nikon 35mm f/1.8 AF-S One year on

This lens is very, very good. I had read on boards from people who would never take this lens off the camera. I find that most of my shooting is done with this lens. The kit lens that came with the D40 (The Nikon 18-55 mm) is very good, but the 35mm's 1.8 aperture is great.

Thunderbird bug

If new mail comes in and goes through a filter and into a folder, if you are viewing that folder, the new mail does not appear until you view a new folder and then go back to the old folder.

Installing python in user space

cd Python-2.6.6/
./configure --prefix=/home/kg46/local
make install

git clone git:// numpy
~/local/bin/python install --prefix=/home/kg46/local/

Installing matplotlib in user space

Problem:Need to install matplotlib in user space, because I'm going to use a HPC cluster and their version of everything is just a little out of date.

Solution:svn co matplotlib
cd matplotlib
python install --home=~

Now add to .bash_login the lines

And python is picking up the svn version

Spotlight : disabling and reenabling

Problem:I completely screwed up spotlight by killing the mds process AND deleting some spotlight files
Solution:From here: The following commands can be used to properly enable and disable spotlight

sudo launchctl unload -w /System/Library/LaunchDaemons/
sudo launchctl unload -w /System/Library/LaunchAgents/

sudo launchctl load -w /System/Library/LaunchDaemons/
sudo launchctl load -w /System/Library/LaunchAgents/

Important Bangla unicode rules

Consonant conjucts are made using hashanta 0x09cdHashanta is generated using hashanta + zero width non joiner 0x09cd + 0x200cThe zwnj is only needed if the next letter would form a conjunct. Otherwise, 0x09cd is understood to be a hashanta All indic scripts use the devnagari danda and double danda 0x0964,0x0965

Input Method for Bengali (Bangla) on Mac OS X

Wanted to have an IM for Bangla on the MacDidn't find any onlineFound this tutorial from Apple on how to make an IMOnly problem is that the system does not seem to support the generation of conjunctions of vowel modifiers.This means that each and every conjunction and consonant+vowel combination must be present in the file:
i.e. if we want to write কৌ (kou) we have to set kou = কৌ in the file: we can't build a rule. So this file must have all the possible combinations, like কু  কূ etc. etc.This is tedious to do by hand. I'm working on a small script to do these mix-n-matches and print out a input method file based on the rules I developed for lekho (see I knew that effort would come in useful some day!)Jamil Ahmed has put up a bangla dictionary for Firefox.Input keystrokes are case insensitive.

Google can block searches for you

Do a searchClick on the link to go to some hit (This must be done first)Hit back on the browserThe relevant hit now has a "Block" link below itClick it if you want to block search hits from that siteI was initially looking for the "Block" link and didn't find it. It is not so well advertized that you have for first visit the site, before you can block it.

Mac OS X: Sparse Bundles and backups

Sparse bundles, like read/write disk images, can be encryptedCreate a sparse bundle through the disk utilityThe sparse bundle's size on disk corresponds to its content size (which I like, but you may not like that it gives away some information about its insides)The change password function of Disk Utility (Images->Change Password) does not work through the GUI (the sparse bundle is greyed out and can not be selected). Use the command line instead:
hdiutil chpass /path/to/Encrypted.sparsebundle/
The sparse bundle is supposed to be easier on Time Machine and other backups, because it is not one monolithic structure.Resizing the sparse bundle needs to also be done through the command line (GUI greyed out) and is done using
hdiutil resize -size 1.5g /path/to/bundle
Where 1.5g is the new size. It will ask you for the password.

Rubyforge repository public key (from Ben)

Ben writes:

 I don't know if you use rubyforge much anymore, but I found that the reason for this [he could not commit or update his code repository on rubyforge] was because they suddenly started requiring a public key and password authentication is now disabled.
Thought I'd give you a heads up so you don't spend the time trying to troubleshoot on your end.  Maybe you can write a blog entry about it.  It sort of annoys me that they didn't at least notify us by email that they were changing their login setup. Blog entry plagiarized as requested.

Also see

Genealogy software: GRAMPS

GRAMPS is a nice, open source (GPLd) program written in python that allows you to build up a database of family relationships, from which you can make genealogy treesGRAMPS runs on mac os, and they say it should run on windows and linux tooIt is very intuitive and easy to useI miss the ability to see descendant family trees with siblings etc (like this), but perhaps there is a trick to do that.

Mac OS X finder not displaying thumbnails properly

Problem: Mac OS X Finder doesn't show the little thumbnails properly (where it used to before). For example, it will show thumbnails for .txt or .html files, but will only show a generic JPEG thumbnail for .jpg files
Solution: (From here) Go to (your home) > Library > Preferences folder, delete the file and the file if it exists, and then restart Finder from the force quit menu.