Thursday, December 22, 2011

View markdown on Mac OS X

Use the Quicklook plugin from here.

Simpy drop the QLMarkdown.qlgenerator to ~/Library/QuickLook or /Library/QuickLook

Now whenever you select a markdown file (.md) it will format the file and show the formatted version to you through quicklook.

Monday, December 19, 2011

The "page flash" on the kindle

The "page flash" on the Kindle 4 - the sudden turning of all the pixels black and then white, before the page is displayed - occurs every 6 page turns. It seems to be necessary because a certain amount of 'dirt' accumulates inbetween - bold letters (and also normal text) seem to leave traces pixels around the edges of the letters after they have been wiped after a page turn. The 'flash' wipes these pixels out and makes the page look crisp again.

I don't know if this is a flaw in the driver software, or a shortcoming in the e-ink display itself (that leaves the traces).

Thursday, December 15, 2011

Quick state machine in Python

SM = {
  'TEND0': {'TRIALEND': 'TEND1'},

nextstate = SM[currentstate].get(input,currentstate)

Wednesday, December 14, 2011

matplotlib installation

  1. freetype2
    sudo make install
  2. libpng (
    sudo make install
  3. sudo easy_install matplotlib

Monday, December 12, 2011

Bye bye eclipse, hello PyCharm and QT creator

I ditched eclipse for C when I found QT Creator. But I still kept it around for my python code, but it got annoying. Then I found PyCharm. I like it. Bye bye eclipse.

Changing how numpy arrays are printed

Use numpy.set_printoptions. One of the things I like most about pylab (and numpy) is this flexibility, especially the summarization feature, which, when the array is very large, prints ellipses (...) instead of clobbering you with lines and lines of output. I like to set the threshold low (default is 1000) and I do this with pylab.set_printoptions(threshold=10)

Wednesday, December 7, 2011

Mac OS Lion : install matplotlib

  1. Download source
  2. "Building mpl on Mac OS X is a nightmare" - John Hunter
  3. Use his very effective make.osx:
  4. sudo make -f make.osx PREFIX=/temp/build/dir PYVERSION=2.7 fetch deps mpl_install_std
  5. Remove old ~/.matplotlib directory (some weird warnings)

Tuesday, December 6, 2011

Mac OS X install pytables and h5py

  1. Install tables - need NumPy version 1.6
  2. Get NumPy from sourceforge and install - need Python 2.7
  3. Install python 2.7 on Lion, open new terminal (or refresh path)
  4. curl | python
  5. curl | python
  6. sudo pip install ipython
  7. sudo pip install tables
  8. need numexpr > 1.4.1, 
  9. Download anc ompile numexpr -> wants to compile using gcc-4.2
  10. sudo ln -s gcc gcc-4.2
  11. sudo pip install cython
  12. Get HDf5 from
  13. /configure and compile
  14. Copy the hdf5 folder whereever you want
  15. python build --hdf5=/path/to/hdf5 (from the unzipped source of h5py)
  16. sudo python install --hdf5=/usr/local/hdf5/ (in the unzipped dir of pytables)
In contrast, to get h5py working on Ubuntu:

sudo apt-get install libhdf5-serial-dev
sudo easy_install h5py 

Dealing with strict key checking for SSH


While using sftp (or some other ssh related service) the following warning shows up and we can't use the service:

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /Users/xxx/.ssh/known_hosts to get rid of this message.
Offending key in /Users/xxx/.ssh/known_hosts:28
RSA host key for has changed and you have requested strict checking.
Host key verification failed.
Connection closed


Remove the key by doing
sed -i "28 d" ~/.ssh/known_hosts

We know it is line 28, because that came up in the warning message.

github project page checkout for working

git clone -b gh-pages

Mac OS Lion

Things I like
  1. New tab in terminal opens in same folder
  2. Preview can make pdfs by stringing msic documents together
I don't like
  1. Preview is sooo sluggish
  2. Preview messes up some transparency in pdfs
  3. No save as (gotta duplicate etc etc)

Saturday, December 3, 2011

Reading Plexon files into Python

  1. Sample C code is at plx2csv, rather striaghtofrward reading of the plexon binary file
  2. Needed to load the data into Python, so looked at Boost.Python and pybindgen, but then
  3. decided that pickling the data from the c code and loading it into python scripts would be fine and so used
  4. The PicklingTools library.

Thursday, December 1, 2011

++i OR i++ ?

From discussions on stack overflow we have:
  1. ++i does not expect to return a value while i++ might need to return a value.
  2. A modern compiler, for simple data types (i.e. integer) will treat the two the same if the context is right e.g. in a for loop
  3. For an object (e.g. an interator), however, ++i will be faster than i++ because a compiler cannot optimize away the creation of a temporary object for i++

Tuesday, November 29, 2011

LaTeX : pretty printing the difference between two versions

Use latexdiff

e.g. latexdiff ../v01/Complete/main_v02.tex main_v02.tex > diff.tex

Then compile diff.tex (and don't forget to run bibtex, though new citations don't show up as changes)

Monday, November 28, 2011

MS word (Mac os X) Different footer/header text

  1. Insert->Break->Section break(continuous)
  2. Open formatting palette
  3. Uncheck "link to previous"
  4. Enter the new header/footer

Monday, November 21, 2011

WARNING: terminal is not fully functional (After installing Xcode on Mac OS X)

From the first answer here:
  1. This is due to xcode install messing with /usr/share/terminfo
  2. The default terminal is xterm-256color which is removed from /usr/share/terminfo/78
  3. If you have a TimeMachine backup, use that to restore /usr/share/terminfo. 
  4. If you don't, use the file provided here.

Sunday, November 20, 2011

How to get file creation time on Mac OS X (64 bit)

  • QFileInfo.created() on POSIX systems (like Mac OS X) returns the last modified time.
  • This is annoying because on Mac OS X you know the system (e.g. finder) has access to the actual creation time.
  • Turns out that BSD systems (like Mac OS X) have an extension to sys/stat.h that contains the file creation time.
The recipe, then is (shown using the QT framework):
#include <QtCore>
#include <sys/stat.h>

int main(int argc, char *argv[])
    QString fname("/Users/kghose/Sandbox/ChhobiTest/2005/2005_07_10/MVI_0693.AVI");
    QDateTime thedatetime;
    struct stat64 the_time;
    stat64(fname.toStdString().c_str(), &the_time);
    qDebug() << thedatetime;

Monday, November 7, 2011

MenuMeters (Replacement for iStat)

MenuMeters is an opensource program that does, very colorfully, what iStat does and works on Lion. The appearance is not as slick as iStat, but it is very configurable.

Thursday, October 27, 2011

Income inequality

I don't deal with political topics here. But there were some thoughts I had on this fad going around on income inequality.

If you take a look at this report by the economist the claim is that while the income disparity between 99% of the people in the US hasn't changed much that between the 99% and the 1% has changed a lot.

I can see in my mind's eye some person or persons with a definite political agenda sitting at their software and moving a slider control. They first split the population 50/50 and grunt with dissatisfaction. They move it to 25/75. Still not so good. Then they get all excited and move it all the way to 1/99. Yay! Look at THAT income disparity.

I have a few concerns about this graph. Perhaps someone can enlighten me?
  1. If I took the top 1% of a distribution MOST measures I would make using that 1% would be biased in some extreme way. There is no way I could stop that, because I took an extreme part of the population to start with. I'm taking a few samples from a tail, taking that average and comparing it with the average of the rest. What ELSE could I expect from any distribution in the real world?
  2. What kind of outcome are we looking here? A truncation of the tail? Some way to get rid of the super rich? Sure we could, but WHAT WOULD THAT BUY US? What would we be happy with. One super rich individual per country? 10? 100?
I really don't understand this sentiment and frankly I am worried by it. I am worried by a sentiment that people should not be much more financially successful than the average.

If the anger is against the bank bailouts or bank policies, take your money and put it in a different bank or a credit union! If you hate outsourcing, buy local products. If you are unhappy because you are unemployed, start a business!

Life isn't easy, and never was and never will be. Anybody who promises otherwise is lying! There may be some temporary steps the government can take to increase short term demand, but getting rid of financially successful individuals can NEVER improve conditions. That's just jealousy pure and simple.

Progressive tax rates is fine, it's an economic decision on the part of the govt. but just remember, the top 1% are usually mobile, and have international interests, so they just might move to somewhere else more economically suitable for them - and we are left with, well 99% of the people who don't take the financial risk that a lot of the 1% are taking (which got them to 1%).

If you are angry with the bonuses that the wall street guys are getting MOVE YOUR MONEY ELSEWHERE! They didn't steal the money from us, we gave it do them in expectation of higher returns. MOVE THE MONEY ELSEWHERE! It was a bad decision.

If they broke laws, shout for their prosecution. If the laws were lax, shout for better ones.

But don't hate on the rich: it's the surest way to stay poor and miserable.

We are not serfs paying rent to rich landowners anymore. We have choices. But we need initiative and courage to take them. Sure we can take less risk, but then we get less reward. If we complain about that - well that's entitlement and that just keeps us poor.

Take control, and don't hate.

Wednesday, October 26, 2011

Use to insert images as pages into a pdf

  1. Open image in, "print" and save as pdf
  2. Open pdf you want to add to
  3. Open the pages pane
  4. Drag and drop the image pdf you just created into the pages pane (in whatever sequence you want it)
  5. will show the pdf to be "edited"
  6. Save the pdf
You can directly drag and drop the image file from the Finder into the pages pane. This will, however, add a white border round the image.

Monday, October 24, 2011

iTunes: option + maximize = real maximize

iTunes is annoying because clicking "+" (maximize) moves it to mini-player mode. Holding down "option" when clicking "+" will make it work like mazimize

Thursday, October 20, 2011

Thunderbird: "Shift" sends emails in html

Holding down "shift" when you press "reply" or "forward" causes emails to be composed in HTML in thunderbird.

Probably the most useful tip I have seen.

Tuesday, October 18, 2011

Export display from Linux to Mac

From here
  1. On client Mac startup and check preferences->Security->"Allow connections from network clients"
  2. Open an xterm from the menu
  3. ssh into host machine with the X flag (from the xterm)
    ssh y.y.y.y -X -luser
  4. Everything works (even with firewall on), no need to set DISPLAY etc.

Monday, October 17, 2011

Installing ubuntu (11.10)

  1. New HDD was cut into two partitions and win xp was installed on one.
  2. Ubuntu was installed off a USB stick "Install alongside Win XP" was chosen. This correctly used the empty (unformatted) partition for Linux, with appropriate swap space choice.
  3. Ssh needs to be installed - can be installed through the package manager. I used a metapackage to install both client and server. The changes took effect immediately (no restart or manually starting server). I went from not being able to ssh into the machine from my mac to being able to.
  4. Python 2.6 comes installed
  5. sudo apt-get python-pip
  6. sudo pip install ipython
  7. sudo apt-get install python-matplotlib
  8. The Unity Launcher cannot be configured to be at the bottom. Idiots.
  9. sudo apt-get install aptitude
  10. install kubuntu-desktop via aptitude (picked kdm as display manager)
  11. Muon package manager has authentication problems (can't authenticate - Muon creator says it is a policykit problem). Idiots
  12. Uninstall KDE and go back to Unity: follow the instructions here. (This is important)
  13. (sudo apt-get autoremove because it told me to)

Wednesday, October 12, 2011

Get rid of MySQL from Mac OS X

Follow the instructions here. For Lion /etc/hostconfig seems to be going away.

Migrating from a 10.5 macbook to a 10.7 macbook pro

  1. Most impressed by migration: just needed to hook up my time machine disk to the new machine in in 2 hours my WHOLE computer was transplanted onto the new mac, including applications and frameworks.
  2. When I went to run eclipse (Helios) I was asked to install java runtime, which the OS installer found and installed by itself.
  3. The fast login switching, not so useful for single user laptops and also takes up menubar space with your name can be removed from system preferences (here).
  4. Spaces threw me: from here, we see that the way to add new spaces is to go into misson control (middle click) and then move the pointer to the top right hand corner to add a new desktop.
  5. To use the existing time machine backup (so you keep your history etc), however, all the files seem to be copied over afresh - so you get a new snapshot which uses a lot of space - the first new backup is not incremental.
  6. The screen looks different - the colors are brighter - but perhaps my old macbook's lcd was just fading.
  7. Two new bangla input methods are available and built in, though I am yet to try them properly.

Tuesday, September 27, 2011

Deus Ex HR

  1. One of the few games I ever bought when it came out (thus at full price. That's how much I wanted it)
  2. Tossing a cardboard box at armed NPCs now leads the bastards to gun you down mercilessly, rather than simply yell at and berate you like they used to do in DE
  3. You can drag a body through the vents.
  4. Immersive side-story items
  5. Improved dialog system much like RPG
  6. You can sell stuff you don't need for cash - so I sell all the lethal weapons, which I don't use. 
A LOT of attention has been paid to detail and atmosphere, one of the few games I would recommend to get at full price to reward the creators. If you liked Deus Ex, you will definitely enjoy this one, which has update graphics and some enjoyable new augs.

Saturday, September 17, 2011

Travelling on advance parole from India to the US through Frankfurt

  • The person at the Lufthansa counter accepted the combo EAD/AP card
  • The immigration official at Indian customs examined the card closely, looked at my expired H1-B visa stamp, asked me if they would let me in with this card (I said yes) and then let me pass
  • At Frankfurt, I was asked if I have a visa, I said "Yes, a German visa" (I had a one year multiple entry German visa). She did not flip through the passport to actually find the visa, she just let me through after looking at the front page. Current regulations state you need a transit visa if you are on AP and are not from one of the visa waiver countries.
  • At US immigration I was asked to fill out the white I-94 card and sent to secondary inspection
  • At secondary inspection the lady asked a few questions about my specialization and then stamped the I-94 card as "DA/AOS" and let me through. She also "Cancelled Without Prejudice" my expired H1-B visa stamp
  • I did not have to show any documents other than my passport and EAD/AP card to anyone.

Friday, August 26, 2011

Extract a clip from a video with mencoder

mencoder -ss 00:38:46 -endpos 00:05:40 -oac copy -ovc copy movie.avi -o excerpt.avi

00:38:46 is the start position and 00:05:40 is the duration

Thursday, August 11, 2011

Latex: Pushing floats to the end of a document

The endfloat package is kind of fancy but the problem I faced with it is that I was renumbering figures in the middle of the document and the latex referencing algorithms were handling it fine, but when the figures appeared at the end of the document their numbering was screwed up.

There is an easy way, however, to push the figures to the end of the document as I wanted. From some hints here about how LaTeX decides whether there is enough space on a page to put in a figure


in the preamble does the trick, by fooling LaTeX into deciding there is NEVER enough space to put figures on a page until it is all out of text and has to just dump the figures as best as it can, one to a page.

Tuesday, August 9, 2011

Latex + Unicode = XeTeX

Never bothered to figure out what this xetex thing was. Turns out it can handle unicode. For texmaker just replace pdflatex command with xelatex and use a unicode aware font (e.g. \usepackage{helvet}) and things should work

Sunday, August 7, 2011

Updating Eclipse

From here

Help->Install New Software->Add site-> (or whatever release you are going to)

Help->Check for updates

And eclipse should update itself with no fuss.

Monday, August 1, 2011

migrating existing svn repo to git (and use with eclipse)

  1. Install git
  2. sudo gem install svn2git --source (here)
  3. svnserve -d (start readonly svnserver on local machine)
  4. mkdir new directory and cd into it
  5. svn2git svn://localhost/path/to/local/svn/repo
  6. Install EGit and restart.
  7. (Eclipse) new pydev project ->  browse to existing folder
  8. Set up git locally (here)
  9. set up to use git
  10. team -> remote -> push -> (url from your project page)
  1. git svn clone file: did not work because of a file format problem (Expected FS format '2'; found format '4' at /usr/local/git/libexec/git-core/git-svn line 1535)

Wednesday, July 27, 2011

small issue with hist

The left is the matplotlib plot - notice the slight rightward shift of the drawn histogram with respect to the solid black line.

The right is the pdf produced from the same plot, which is exactly where it should be.

Same deal, but with 'Agg' backend and printed to png

LyX + Mac + Math heavy document = keystone cops

LyX has a really nice online math type setting system. I have a math heavy document that I wanted to type up using LyX. Pretty soon I had the classic problem of any feedback system with feedback delay - oscillations. I would type something, it would take about a quarter of a second to update, I would move my mouse to click on a formula, it would take a quarter of a second to change state and by that time I would have clicked twice, the document would have scrolled, and the formula would change state again, because I had clicked somewhere else.

Welp. Its back to good old latex and text editors for me. I don't mind the write-compile cycle.

LyX is good for text documents, but for math, at least on mac, it is so sluggish, it is more annoying than anything.


I have found texmaker to be a very nice IDE on mac.

Thursday, July 21, 2011

Tracking down backup space hogs

Problem: Even though I haven't done many file changes the Time machine incremental backup is about 350 MB per flush (resulting in many GB per day for hourly backups). I don't generate THAT much work each day.

Using Time Machine and Time Tracker to track down the culprits.


global-messages-db.sqlite (170.5 MiB)


urlclassifier3.sqlite (50.6 MiB)
Cache 78.2 (MiB)

Friday, July 15, 2011

ipython + matplotlib crashing

I've been having this problem ever since I changed to 1.0.1. It's not clear if it is ipython's fault or matplotlib's fault.

Today I got the first error message before segfault:

objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470

And Mac OS X error report:

Date/Time:       2011-07-15 21:47:54.500 -0400
OS Version:      Mac OS X 10.5.8 (9L30)
Report Version:  6
Anonymous UUID:  655916BF-D521-48C5-B8A3-FBC5E5FA6683

Exception Codes: 0x0000000000000001, 0x0000000000000000
Crashed Thread:  0

Application Specific Information:
objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470

Thread 0 Crashed:
0   libobjc.A.dylib                0x90595bfa _objc_error + 116
1   libobjc.A.dylib                0x90595c30 __objc_error + 52
2   libobjc.A.dylib                0x90594637 _freedHandler + 58
3       0x93faa676 CFArrayApplyFunction + 198
4               0x91a0aadc -[NSView _finalizeWithReferenceCounting] + 712
5               0x91a0a7e8 -[NSView dealloc] + 46
6           0x9524e68f NSPopAutoreleasePool + 1007
7                     0x02932580 wait_for_stdin + 561 (_macosx.m:171)
8                    0x0133bac0 call_readline + 432
9   org.python.python              0x001e04de PyOS_Readline + 254
10  org.python.python              0x0028dbfc builtin_raw_input + 412
11  org.python.python              0x00296165 PyEval_EvalFrameEx + 19429
12  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
13  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
14  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
15  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
16  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
17  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
18  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
19  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
20  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
21  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
22  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
23  org.python.python              0x002983f7 PyEval_EvalCode + 87
24  org.python.python              0x002bcf08 PyRun_FileExFlags + 168
25  org.python.python              0x002bddf3 PyRun_SimpleFileExFlags + 867
26  org.python.python              0x002cf902 Py_Main + 3122
27  org.python.python              0x00001f82 0x1000 + 3970
28  org.python.python              0x00001ea9 0x1000 + 3753

Thursday, July 14, 2011

Bug in STIX fonts

Grumble, grumble. MathML on Firefox on mac was going great guns, until I find that
gives me garbage (but
looks fine). I see from this bugreport, that this is an issue with the STIX fonts.

Making up a GET request in bottle

Problem: I'm using bottle for a desktop browser based app. I needed to call a page view from an action. The page view normally gets some parameters encoded in a GET request.

Solution: The GET request can be faked by doing:
request.GET.append(key, value)

is just a dictionary.

subclipse, branching, merging etc.

  1. Sometimes merge/branch will refuse to work, saying your resource is out of synch. Do an 'update' and then a 'commit'.
  2. After a rename or move + commit, subclipse will say resource is out of synch with the filesystem, just refresh,commit and continue
  3. pydev explorer does not show the .* project files. These need to be added to svnignore (or committed if you wish). Use the eclipse navigator to see these file
  4. switching a branch to be the trunk is annoying in svn: basically you have to move the folder over - so the current trunk has to be moved (possibly under branches with a new name) and the branch you want has to be moved over and renamed as trunk

Wednesday, July 13, 2011

Mac OS X MS Word and Equations

I like to compose my equations in LaTeX. I find it a lot faster than the click interface of MS Word's own equation editor. I type up my notes in a LaTeX format so I already have the equations and its a terrible inefficient duplication of work to hunt-n-click them into ms word.

One solution is to print a pdf of all the equations, and then use the select tool in preview to copy the equation and then paste it into the word document.

The advantage in mac os x is that the copy is in vector format, which means that the inserted equation does not behave as an image - becoming pixellated at the worst moments, but rather scales gracefully.

That's gonna save me some time.

Removing the top and right lines in the axis box in matplotlib

From here:
Where ax is an axis instance (you can get it using pylab.axis())

PS. Those lines are called spines, in case you haven't caught on

Thursday, July 7, 2011

Pickling python classes

Ok, this is the first thing in Python I have found to be annoying and nonintuitive. When you pickle a class object you need to make sure that the module for the class is explicitly in scope and imported.

I kept banging my head against this problem and only understood it after looking at this guy's blog.

When you pickle a class object included in the pickle file is a coded import statement telling the interpreter which module to import to look for the definition of the class. This leads to the following gotcha:

The following code will define a class, instantiate it and pickle it without problems

import cPickle

class A:
  def __init__(self):
    self.x = 22

if __name__ == "__main__":
  a = A()
  cPickle.dump(a, open('obja.pkl','wb'), protocol=-1)
  print cPickle.dumps(a, protocol=0)

Notice the stringification of the class : it begins with

Now, we run the following code to load the object from the pickle:
import cPickle
m = cPickle.load(open('obja.pkl'))
AttributeError                            Traceback (most recent call last)
/Users/kghose/Research/2011/Papers/SpatialIntegration/Python/Sandbox/ in ()
----> 1 m = cPickle.load(open('obja.pkl'))

AttributeError: 'FakeModule' object has no attribute 'A'


Now, say we change the original file by explicitly importing our module and instantiating the object as a class from that module (as would happen if our pickling code was in a different file from the class definition file):

import cPickle

class A:
  def __init__(self):
    self.x = 22

if __name__ == "__main__":
  import class_a
  a = class_a.A()
  cPickle.dump(a, open('obja.pkl','wb'), protocol=-1)
  print cPickle.dumps(a, protocol=0)

Now notice that the serialization begins with
which is our module name.

And when we try to load it, we don't get any errors.

Friday, July 1, 2011

Hide the dock icon of a program when it is running

Go to the application's bundle (e.g. and open the bundle contents. Navigate to Info.plist, open it with a text editor or plist editor. Add or change an item with the key "NSUIElement" and set its value to "1" (type = string). On relaunch of the application, its icon will not appear in the dock.

This is usefull for apps like X11 which are servers and should be running in the background.

Thursday, June 30, 2011

Force FireFox to reload a short cut icon

Type in the url of the icon file and load the image.

utf-8 encoded files in python

From this PeP:

Adding a line that looks like:

# This Python file uses the following encoding: utf-8

Is sufficient to avoid the

no encoding declared; see for details


Monday, June 27, 2011

minidom.writexml handling unicode errors

Writing xml with unicode characters does not work, even with the encoding set to 'utf-8'. Always results in something like

UnicodeEncodeError('ascii', u'kr\xf6se1989', 2, 3, 'ordinal not in range(128)')

Solution (from here):
Use the codecs module and manually open the file in utf-8 mode for writing

import codecs
doc.writexml(,'wb','utf-8'), encoding='utf-8')

SQL query binding for a list using Python

From here:

l=[1,2,3,4] placeholder= '?' # Works For SQLite. placeholders= ', '.join(placeholder for dummy in l) #puts in len(l) '?' marks, separated by ', ' query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders cursor.execute(query, l)

Thursday, June 16, 2011

Firefox + Mac OS X + bangla fonts = disaster

Please see this bugzilla issue.
In short:

  1. Test any bangla font you install using Mac OS X's fontbook. Do not install any font that shows warnings or errors
  2. The only Bangla font that I found to work with Firefox and not crash it on Mac OS X is at onkur.

Sources for Bangla unicode fonts

Tuesday, June 14, 2011

Strange inefficieny of pysqlite over sqlite (and apsw)

I've been happily using pysqlite to access my sqlite databases from python. Suddenly, I find, however, that there is a query that runs much, much slower in pysqlite than in apsw and the sqlite shell. I've filed a bug report but it looks like I'll be switching to apsw until this thing gets resolved.

Saturday, June 11, 2011

Another open source reference manager : Paperpile

From a blog post here, I found out about Paperpile. I downloaded and ran it, but haven't used it, since I use my own, but Paperpile has  a nice clean look. Though there is GPL code of Paperpile on github, the main website seems to indicate the author has hopes of commercializing (and closing the source) to Paperpile if things look successful.

Wednesday, June 8, 2011

Compress camera video for a windows machine

mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=1:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o /dev/null MVI_0052.AVI
mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=2:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o output.avi MVI_0052.AVI

Tuesday, June 7, 2011

Python and word documents

From here, using only standard python modules:

import zipfile, re

docx = zipfile.ZipFile('/path/to/file/mydocument.docx')
content ='word/document.xml')
cleaned = re.sub('<(.|\n)*?>','',content)
print cleaned

But, if you want to mess around in more detail in the document, then we can use the python-docx module.

Monday, June 6, 2011

discount bug

import discount
a = discount.Markdown('')
--> Yields TypeError
TypeError: 'NoneType' object is unsubscriptable
Should return a blank.

Found that markdown or markdown2 give comparable performance, so now no longer using discount.

Friday, June 3, 2011

Using adminer on Mac OS X

adminer is a nice php based sqlite manager. I prefer the firefox plugin "sqlite manager" but it currently has a strange issue with FF5 that basically makes it unworkable, so I was looking for an alternative to tide me over.

I really don't want apache running all the time on my computer and don't want people browsing to my computer, so what I needed to do was:

  1. Download the adminer php script into /Library/WebServer/Documents/
  2. Change /etc/apache2/httpd.conf to allow running of php scripts (uncomment the line that begins: LoadModule php5_module
  3. Start the apache server: sudo apachectl -k start
  4. Operate the script by going to localhost
  5. Stop the server: sudo apachectl -k stop  

Bottlepy server exits during sleep (Mac OS X)

Problem: Bottlepy server exits when computer comes out of sleep
Solution: Set "reloader" option to False.

Tuesday, May 31, 2011

Text search, sqlite bindings and Python

c.execute("SELECT * FROM entries WHERE (title LIKE ? OR body LIKE ?) order by date desc", ("%%%s%%" %text, "%%%s%%" %text))

  1. Putting %?% does not work
  2. "%?%" does not work
  3. Double percent sign (%%) due to python string substitution rules

Saturday, May 28, 2011

Standalone browser based app

Perhaps my long search for a convenient way to code database frontends that work locally ("desktop app") but leverage the power of a browser as a GUI has come to a successful point. This point involves:
  1. Python (Naturally)
  2. Bottle - a "microframework"
  3. py2app - py2exe for windows
The only tricks making Bottle and  py2app work well together were
  1. You need to explicitly list the template files when calling py2app, otherwise they don't get included in Resources
  2. I had to add a url route (which I called /quit) that called os._exit() to shut down the server (otherwise done using CTRL-C). just exit() and even SIGINT did not work. This is a bit extreme, but I think safe in this usage.
I was happy to see that the created app used the Resources directory as the working directory and the packager put all the files in this directory making things very transparent and worry free.


From the HTML standards.
Use POST when the operation has lasting changes on the world, like a database modification.

Friday, May 27, 2011

Sunday, May 15, 2011

3d wall decor

So, there are artists who paint patterns on walls that turn into a recognizable figure if you are standing at the right perspective (e.g. here).

One easy way to generate these patterns is to set up a projector with a slide of the pattern that you want projecting from the vantage point you want, and then paint according to the projected pattern.

Sunday, May 1, 2011

Nikon D40 (almost) 2 years on

The D40 is an awesome camera, especially at the price I bought it ($375 with kit lens - but refurbished). It's years out of production now and has been superseded, but it does fine for my casual needs. The only two gripes I have about it, in order of importance are:

  1. Auto white balance is sucky for incandescent lighting. So I always switch from auto to preset when I shoot indoors.
  2. Only one func button makes it awkward. I have to pick between select ISO and select WB for that button. I think select WB wins out, but it would be nice to have dedicated buttons for that.

Nikon 35mm f/1.8 AF-S One year on

This lens is very, very good. I had read on boards from people who would never take this lens off the camera. I find that most of my shooting is done with this lens. The kit lens that came with the D40 (The Nikon 18-55 mm) is very good, but the 35mm's 1.8 aperture is great.

Monday, April 18, 2011

Size of objects in Python

struct.calcsize(fmt) - only gives size of fixed data types given by fmt
sys.getsizeof(obj) - (added in Python 2.7) gives the size in bytes of builtin types, but not the actual used memory
pympler.asizeof(obj) - gives the actual memory the object uses

Monday, April 11, 2011

Multiple logins to the same site

Multifox is a great firefox add-on that allows you to switch between multiple accounts on the same site (Say you have two facebook accounts and you need to switch between them and are tired of logging out and then back in again). It's user interface is simple and effective.

Diff files in eclipse

From here:
  • Add the two (or more) files you want to diff to your workspace.
  • Select all the files you want to diff.
  • Right click on one of the selected files and chose "Compare with -> each other"

Tuesday, April 5, 2011


Whenever I submit a job to the computing cluster I have to smile and wonder if the folks at Platform Computing planned their command naming very carefully so that one of their commands would endup as "bjobs".

Thunderbird bug

If new mail comes in and goes through a filter and into a folder, if you are viewing that folder, the new mail does not appear until you view a new folder and then go back to the old folder.

Sunday, April 3, 2011

Writing to NTFS partitions from Mac OS X 10.5

Use NTFS-3G. It did something odd to my Finder, but after I restarted the computer as recommended things went back to normal, though there was a longer than usual start-up time.

Install multiprocessing module for Python 2.5

The great multiprocessing module was included in the Python 2.6 distribution. If you need to use it on a box with Python 2.5 with no root access then do this:

From here download the package. And then install in user space by doing

python install --home=~

Wednesday, March 30, 2011

Tuesday, March 29, 2011

Installing python in user space

cd Python-2.6.6/
./configure --prefix=/home/kg46/local
make install

git clone git:// numpy
~/local/bin/python install --prefix=/home/kg46/local/

Installing matplotlib in user space


Need to install matplotlib in user space, because I'm going to use a HPC cluster and their version of everything is just a little out of date.


svn co matplotlib
cd matplotlib
python install --home=~

Now add to .bash_login the lines

And python is picking up the svn version

Spotlight : disabling and reenabling


I completely screwed up spotlight by killing the mds process AND deleting some spotlight files


From here: The following commands can be used to properly enable and disable spotlight

sudo launchctl unload -w /System/Library/LaunchDaemons/
sudo launchctl unload -w /System/Library/LaunchAgents/

sudo launchctl load -w /System/Library/LaunchDaemons/
sudo launchctl load -w /System/Library/LaunchAgents/

Saturday, March 26, 2011

Important Bangla unicode rules

  1. Consonant conjucts are made using hashanta 0x09cd
  2. Hashanta is generated using hashanta + zero width non joiner 0x09cd + 0x200c
  3. The zwnj is only needed if the next letter would form a conjunct. Otherwise, 0x09cd is understood to be a hashanta
  4. All indic scripts use the devnagari danda and double danda 0x0964,0x0965

      Monday, March 21, 2011

      Input Method for Bengali (Bangla) on Mac OS X

      1. Wanted to have an IM for Bangla on the Mac
      2. Didn't find any online
      3. Found this tutorial from Apple on how to make an IM
      4. Only problem is that the system does not seem to support the generation of conjunctions of vowel modifiers.
      5. This means that each and every conjunction and consonant+vowel combination must be present in the file:
        i.e. if we want to write কৌ (kou) we have to set kou = কৌ in the file: we can't build a rule. So this file must have all the possible combinations, like কু  কূ etc. etc.
      6. This is tedious to do by hand. I'm working on a small script to do these mix-n-matches and print out a input method file based on the rules I developed for lekho (see I knew that effort would come in useful some day!)
      7. Jamil Ahmed has put up a bangla dictionary for Firefox.
      8. Input keystrokes are case insensitive.

      Google can block searches for you

      1. Do a search
      2. Click on the link to go to some hit (This must be done first)
      3. Hit back on the browser
      4. The relevant hit now has a "Block" link below it
      5. Click it if you want to block search hits from that site
      I was initially looking for the "Block" link and didn't find it. It is not so well advertized that you have for first visit the site, before you can block it.

      Friday, March 18, 2011

      Time Machine and fsck_hfs (Mac OS X 10.5.x)

      Problem:You start a backup and Time Machine sits for ever in the "preparing backup" phase. It seems that fsck_hfs is suddenly doing a lot of work.
      Solution: Click "Stop backing up" and then restart the backup

      Mac OS X: Sparse Bundles and backups

      1. Sparse bundles, like read/write disk images, can be encrypted
      2. Create a sparse bundle through the disk utility
      3. The sparse bundle's size on disk corresponds to its content size (which I like, but you may not like that it gives away some information about its insides)
      4. The change password function of Disk Utility (Images->Change Password) does not work through the GUI (the sparse bundle is greyed out and can not be selected). Use the command line instead:
        hdiutil chpass /path/to/Encrypted.sparsebundle/
      5. The sparse bundle is supposed to be easier on Time Machine and other backups, because it is not one monolithic structure.
      6. Resizing the sparse bundle needs to also be done through the command line (GUI greyed out) and is done using
        hdiutil resize -size 1.5g /path/to/bundle
        Where 1.5g is the new size. It will ask you for the password.

      Tuesday, March 8, 2011

      Rubyforge repository public key (from Ben)

      Ben writes:

       I don't know if you use rubyforge much anymore, but I found that the reason for this [he could not commit or update his code repository on rubyforge] was because they suddenly started requiring a public key and password authentication is now disabled.

      Thought I'd give you a heads up so you don't spend the time trying to troubleshoot on your end.  Maybe you can write a blog entry about it.  It sort of annoys me that they didn't at least notify us by email that they were changing their login setup.
      Blog entry plagiarized as requested.

      Also see

      Friday, March 4, 2011

      Sunday, February 13, 2011

      Genealogy software: GRAMPS

      1. GRAMPS is a nice, open source (GPLd) program written in python that allows you to build up a database of family relationships, from which you can make genealogy trees
      2. GRAMPS runs on mac os, and they say it should run on windows and linux too
      3. It is very intuitive and easy to use
      4. I miss the ability to see descendant family trees with siblings etc (like this), but perhaps there is a trick to do that.

      Thursday, February 3, 2011

      Mac OS X finder not displaying thumbnails properly

      Problem: Mac OS X Finder doesn't show the little thumbnails properly (where it used to before). For example, it will show thumbnails for .txt or .html files, but will only show a generic JPEG thumbnail for .jpg files
      Solution: (From here) Go to (your home) > Library > Preferences folder, delete the file and the file if it exists, and then restart Finder from the force quit menu.