Assorted Experience

Mac OS Lion, external display and sleep

2012-01-24T06:19:00.000-05:00

When I close the Macbook (Mac OS Lion) lid it usually goes to sleep. But with an external display attached and a usb mouse attached the computer just switches to using the external display, rather than sleeping when the lid is closed.

From here, we see that we should use the apple menu to explicitly send the computer to sleep.

View markdown on Mac OS X

2011-12-22T04:43:00.002-05:00

Use the Quicklook plugin from here.

Simpy drop the QLMarkdown.qlgenerator to ~/Library/QuickLook or /Library/QuickLook

Now whenever you select a markdown file (.md) it will format the file and show the formatted version to you through quicklook.

The "page flash" on the kindle

2011-12-19T15:21:00.001-05:00

The "page flash" on the Kindle 4 - the sudden turning of all the pixels black and then white, before the page is displayed - occurs every 6 page turns. It seems to be necessary because a certain amount of 'dirt' accumulates inbetween - bold letters (and also normal text) seem to leave traces pixels around the edges of the letters after they have been wiped after a page turn. The 'flash' wipes these pixels out and makes the page look crisp again.

I don't know if this is a flaw in the driver software, or a shortcoming in the e-ink display itself (that leaves the traces).

Quick state machine in Python

2011-12-15T13:21:00.000-05:00

SM = {
  'INTERTRIAL': {'TRIALSTART': 'TSTART0'},
  'TSTART0': {'TRIALSTART': 'TSTART1'},
  'TSTART1': {'TRIALSTART': 'INTRIAL'},
  'INTRIAL': {'TRIALEND': 'TEND0'},
  'TEND0': {'TRIALEND': 'TEND1'},
  'TEND1': {'TRIALEND': 'INTERTRIAL'}
}

nextstate = SM[currentstate].get(input,currentstate)

matplotlib installation

2011-12-14T16:31:00.000-05:00

freetype2
http://download.savannah.gnu.org/releases-redirect/freetype/freetype-2.4.8.tar.gz
./configure
make
sudo make install
libpng (http://www.libpng.org/pub/png/libpng.html)
./configure
make
sudo make install
sudo easy_install matplotlib

Bye bye eclipse, hello PyCharm and QT creator

2011-12-12T12:07:00.001-05:00

I ditched eclipse for C when I found QT Creator. But I still kept it around for my python code, but it got annoying. Then I found PyCharm. I like it. Bye bye eclipse.

Changing how numpy arrays are printed

2011-12-12T12:05:00.000-05:00

Use numpy.set_printoptions. One of the things I like most about pylab (and numpy) is this flexibility, especially the summarization feature, which, when the array is very large, prints ellipses (...) instead of clobbering you with lines and lines of output. I like to set the threshold low (default is 1000) and I do this with pylab.set_printoptions(threshold=10)

Mac OS Lion : Install ipython + readline

2011-12-09T12:51:00.001-05:00

sudo easy_install-2.7 readline ipython

Mac OS Lion : install matplotlib

2011-12-07T07:44:00.001-05:00

Download source
"Building mpl on Mac OS X is a nightmare" - John Hunter
Use his very effective make.osx:
sudo make -f make.osx PREFIX=/temp/build/dir PYVERSION=2.7 fetch deps mpl_install_std
Remove old ~/.matplotlib directory (some weird warnings)

Mac OS X install pytables and h5py

2011-12-06T17:40:00.001-05:00

Install tables - need NumPy version 1.6
Get NumPy from sourceforge and install - need Python 2.7
Install python 2.7 on Lion, open new terminal (or refresh path)
curl http://python-distribute.org/distribute_setup.py | python
curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python
sudo pip install ipython
sudo pip install tables
need numexpr > 1.4.1,
Download anc ompile numexpr -> wants to compile using gcc-4.2
sudo ln -s gcc gcc-4.2
sudo pip install cython
Get HDf5 from http://www.hdfgroup.org/ftp/HDF5/current/bin/mac-intel-x86_64/hdf5-1.8.8-mac-intel-x86_64-shared.tar.gz
/configure and compile
Copy the hdf5 folder whereever you want
python setup.py build --hdf5=/path/to/hdf5 (from the unzipped source of h5py)
sudo python setup.py install --hdf5=/usr/local/hdf5/ (in the unzipped dir of pytables)

In contrast, to get h5py working on Ubuntu:

sudo apt-get install libhdf5-serial-dev

sudo easy_install h5py

Dealing with strict key checking for SSH

2011-12-06T12:49:00.001-05:00

Problem:

While using sftp (or some other ssh related service) the following warning shows up and we can't use the service:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b0:a8:eb:30:ce:1a:0e:6a:4d:7a:6b:3a:0a:c6:27:60.
Please contact your system administrator.
Add correct host key in /Users/xxx/.ssh/known_hosts to get rid of this message.
Offending key in /Users/xxx/.ssh/known_hosts:28
RSA host key for web.sourceforge.net has changed and you have requested strict checking.
Host key verification failed.
Connection closed

Solution:

Remove the key by doing
sed -i "28 d" ~/.ssh/known_hosts

We know it is line 28, because that came up in the warning message.

github project page checkout for working

2011-12-06T12:38:00.001-05:00

git clone git@github.com:kghose/Chhobi -b gh-pages

Mac OS Lion

2011-12-06T10:52:00.001-05:00

Things I like

New tab in terminal opens in same folder
Preview can make pdfs by stringing msic documents together

I don't like

Preview is sooo sluggish
Preview messes up some transparency in pdfs
No save as (gotta duplicate etc etc)

Reading Plexon files into Python

2011-12-03T09:23:00.001-05:00

Sample C code is at plx2csv, rather striaghtofrward reading of the plexon binary file
Needed to load the data into Python, so looked at Boost.Python and pybindgen, but then
decided that pickling the data from the c code and loading it into python scripts would be fine and so used
The PicklingTools library.

++i OR i++ ?

2011-12-01T09:59:00.001-05:00

From discussions on stack overflow we have:

++i does not expect to return a value while i++ might need to return avalue.
A modern compiler, for simple data types (i.e. integer) will treat the twothe same if the context is right e.g. in a for loop
For an object (e.g. an interator), however, ++i will be faster than i++because a compiler cannot optimize away the creation of a temporary object fori++

LaTeX : pretty printing the difference between two versions

2011-11-29T12:09:00.001-05:00

Use latexdiff

e.g. latexdiff ../v01/Complete/main_v02.tex main_v02.tex > diff.tex

Then compile diff.tex (and don't forget to run bibtex, though new citations don't show up as changes)

MS word (Mac os X) Different footer/header text

2011-11-28T15:59:00.001-05:00

Insert->Break->Section break(continuous)
Open formatting palette
Uncheck "link to previous"
Enter the new header/footer

WARNING: terminal is not fully functional (After installing Xcode on Mac OS X)

2011-11-21T11:37:00.001-05:00

From the first answer here:

This is due to xcode install messing with /usr/share/terminfo
The default terminal is xterm-256color which is removed from /usr/share/terminfo/78
If you have a TimeMachine backup, use that to restore /usr/share/terminfo.
If you don't, use the file provided here.

How to get file creation time on Mac OS X (64 bit)

2011-11-20T15:24:00.001-05:00

QFileInfo.created() on POSIX systems (like Mac OS X) returns the last modified time.
This is annoying because on Mac OS X you know the system (e.g. finder) has access to the actual creation time.
Turns out that BSD systems (like Mac OS X) have an extension to sys/stat.h that contains the file creation time.

The recipe, then is (shown using the QT framework):

#include <QtCore>
#include <sys/stat.h>

int main(int argc, char *argv[])
{
    QString fname("/Users/kghose/Sandbox/ChhobiTest/2005/2005_07_10/MVI_0693.AVI");
    QDateTime thedatetime;
    struct stat64 the_time;
    stat64(fname.toStdString().c_str(), &the_time);
    thedatetime.setTime_t(the_time.st_birthtimespec.tv_sec);
    qDebug() << thedatetime;
}

MenuMeters (Replacement for iStat)

2011-11-07T11:34:00.001-05:00

MenuMeters is an opensource program that does, very colorfully, what iStat does and works on Lion. The appearance is not as slick as iStat, but it is very configurable.

Show ~/Library on Lion

2011-11-03T11:12:00.001-04:00

chflags nohidden ~/Library/

From here.

Income inequality

2011-10-27T14:45:00.000-04:00

I don't deal with political topics here. But there were some thoughts I had on this fad going around on income inequality.

If you take a look at this report by the economist the claim is that while the income disparity between 99% of the people in the US hasn't changed much that between the 99% and the 1% has changed a lot.

I can see in my mind's eye some person or persons with a definite political agenda sitting at their software and moving a slider control. They first split the population 50/50 and grunt with dissatisfaction. They move it to 25/75. Still not so good. Then they get all excited and move it all the way to 1/99. Yay! Look at THAT income disparity.

I have a few concerns about this graph. Perhaps someone can enlighten me?

If I took the top 1% of a distribution MOST measures I would make using that 1% would be biased in some extreme way. There is no way I could stop that, because I took an extreme part of the population to start with. I'm taking a few samples from a tail, taking that average and comparing it with the average of the rest. What ELSE could I expect from any distribution in the real world?
What kind of outcome are we looking here? A truncation of the tail? Some way to get rid of the super rich? Sure we could, but WHAT WOULD THAT BUY US? What would we be happy with. One super rich individual per country? 10? 100?

I really don't understand this sentiment and frankly I am worried by it. I am worried by a sentiment that people should not be much more financially successful than the average.

If the anger is against the bank bailouts or bank policies, take your money and put it in a different bank or a credit union! If you hate outsourcing, buy local products. If you are unhappy because you are unemployed, start a business!

Life isn't easy, and never was and never will be. Anybody who promises otherwise is lying! There may be some temporary steps the government can take to increase short term demand, but getting rid of financially successful individuals can NEVER improve conditions. That's just jealousy pure and simple.

Progressive tax rates is fine, it's an economic decision on the part of the govt. but just remember, the top 1% are usually mobile, and have international interests, so they just might move to somewhere else more economically suitable for them - and we are left with, well 99% of the people who don't take the financial risk that a lot of the 1% are taking (which got them to 1%).

If you are angry with the bonuses that the wall street guys are getting MOVE YOUR MONEY ELSEWHERE! They didn't steal the money from us, we gave it do them in expectation of higher returns. MOVE THE MONEY ELSEWHERE! It was a bad decision.

If they broke laws, shout for their prosecution. If the laws were lax, shout for better ones.

But don't hate on the rich: it's the surest way to stay poor and miserable.

We are not serfs paying rent to rich landowners anymore. We have choices. But we need initiative and courage to take them. Sure we can take less risk, but then we get less reward. If we complain about that - well that's entitlement and that just keeps us poor.

Take control, and don't hate.

Use preview.app to insert images as pages into a pdf

2011-10-26T00:54:00.001-04:00

Open image in Preview.app, "print" and save as pdf
Open pdf you want to add to
Open the pages pane
Drag and drop the image pdf you just created into the pages pane (in whatever sequence you want it)
Preview.app will show the pdf to be "edited"
Save the pdf

You can directly drag and drop the image file from the Finder into the pages pane. This will, however, add a white border round the image.

iTunes: option + maximize = real maximize

2011-10-24T23:49:00.001-04:00

iTunes is annoying because clicking "+" (maximize) moves it to mini-player mode. Holding down "option" when clicking "+" will make it work like mazimize

Thunderbird: "Shift" sends emails in html

2011-10-20T08:49:00.001-04:00

Holding down "shift" when you press "reply" or "forward" causes emails to be composed in HTML in thunderbird.

Probably the most useful tip I have seen.

Export display from Linux to Mac

2011-10-18T14:36:00.002-04:00

From here

On client Mac startup X11.app and check preferences->Security->"Allow connections from network clients"
Open an xterm from the X11.app menu
ssh into host machine with the X flag (from the xterm)
ssh y.y.y.y -X -luser
Everything works (even with firewall on), no need to set DISPLAY etc.

Installing ubuntu (11.10)

2011-10-17T13:27:00.001-04:00

New HDD was cut into two partitions and win xp was installed on one.
Ubuntu was installed off a USB stick "Install alongside Win XP" was chosen. This correctly used the empty (unformatted) partition for Linux, with appropriate swap space choice.
Ssh needs to be installed - can be installed through the package manager. I used a metapackage to install both client and server. The changes took effect immediately (no restart or manually starting server). I went from not being able to ssh into the machine from my mac to being able to.
Python 2.6 comes installed
sudo apt-get python-pip
sudo pip install ipython
sudo apt-get install python-matplotlib
The Unity Launcher cannot be configured to be at the bottom. Idiots.
sudo apt-get install aptitude
install kubuntu-desktop via aptitude (picked kdm as display manager)
Muon package manager has authentication problems (can't authenticate - Muon creator says it is a policykit problem). Idiots
Uninstall KDE and go back to Unity: follow the instructions here. (This is important)
(sudo apt-get autoremove because it told me to)

Get rid of MySQL from Mac OS X

2011-10-12T17:51:00.001-04:00

Follow the instructions here. For Lion /etc/hostconfig seems to be going away.

Migrating from a 10.5 macbook to a 10.7 macbook pro

2011-10-12T17:26:00.000-04:00

Most impressed by migration: just needed to hook up my time machine disk to the new machine in in 2 hours my WHOLE computer was transplanted onto the new mac, including applications and frameworks.
When I went to run eclipse (Helios) I was asked to install java runtime, which the OS installer found and installed by itself.
The fast login switching, not so useful for single user laptops and also takes up menubar space with your name can be removed from system preferences (here).
Spaces threw me: from here, we see that the way to add new spaces is to go into misson control (middle click) and then move the pointer to the top right hand corner to add a new desktop.
To use the existing time machine backup (so you keep your history etc), however, all the files seem to be copied over afresh - so you get a new snapshot which uses a lot of space - the first new backup is not incremental.
The screen looks different - the colors are brighter - but perhaps my old macbook's lcd was just fading.
Two new bangla input methods are available and built in, though I am yet to try them properly.

Deus Ex HR

2011-09-27T00:42:00.001-04:00

One of the few games I ever bought when it came out (thus at full price. That's how much I wanted it)
Tossing a cardboard box at armed NPCs now leads the bastards to gun you down mercilessly, rather than simply yell at and berate you like they used to do in DE
You can drag a body through the vents.
Immersive side-story items
Improved dialog system much like RPG
You can sell stuff you don't need for cash - so I sell all the lethal weapons, which I don't use.

A LOT of attention has been paid to detail and atmosphere, one of the few games I would recommend to get at full price to reward the creators. If you liked Deus Ex, you will definitely enjoy this one, which has update graphics and some enjoyable new augs.

Travelling on advance parole from India to the US through Frankfurt

2011-09-17T05:09:00.001-04:00

The person at the Lufthansa counter accepted the combo EAD/AP card
The immigration official at Indian customs examined the card closely, looked at my expired H1-B visa stamp, asked me if they would let me in with this card (I said yes) and then let me pass
At Frankfurt, I was asked if I have a visa, I said "Yes, a German visa" (I had a one year multiple entry German visa). She did not flip through the passport to actually find the visa, she just let me through after looking at the front page. Current regulations state you need a transit visa if you are on AP and are not from one of the visa waiver countries.
At US immigration I was asked to fill out the white I-94 card and sent to secondary inspection
At secondary inspection the lady asked a few questions about my specialization and then stamped the I-94 card as "DA/AOS" and let me through. She also "Cancelled Without Prejudice" my expired H1-B visa stamp
I did not have to show any documents other than my passport and EAD/AP card to anyone.

Extract a clip from a video with mencoder

2011-08-26T23:23:00.000-04:00

mencoder -ss 00:38:46 -endpos 00:05:40 -oac copy -ovc copy movie.avi -o excerpt.avi

00:38:46 is the start position and 00:05:40 is the duration

Latex: Pushing floats to the end of a document

2011-08-11T18:46:00.002-04:00

The endfloat package is kind of fancy but the problem I faced with it is that I was renumbering figures in the middle of the document and the latex referencing algorithms were handling it fine, but when the figures appeared at the end of the document their numbering was screwed up.

There is an easy way, however, to push the figures to the end of the document as I wanted. From some hints here about how LaTeX decides whether there is enough space on a page to put in a figure


\renewcommand{\textfraction}{1.0}
\renewcommand{\floatpagefraction}{0.9}

in the preamble does the trick, by fooling LaTeX into deciding there is NEVER enough space to put figures on a page until it is all out of text and has to just dump the figures as best as it can, one to a page.

Latex + Unicode = XeTeX

2011-08-09T17:03:00.000-04:00

Never bothered to figure out what this xetex thing was. Turns out it can handle unicode. For texmaker just replace pdflatex command with xelatex and use a unicode aware font (e.g. \usepackage{helvet}) and things should work

Updating Eclipse

2011-08-07T17:05:00.002-04:00

From here

Help->Install New Software->Add site-> http://download.eclipse.org/releases/indigo (or whatever release you are going to)

Help->Check for updates

And eclipse should update itself with no fuss.

migrating existing svn repo to git (and use with eclipse)

2011-08-01T14:19:00.001-04:00

Install git
sudo gem install svn2git --source http://gemcutter.org (here)
svnserve -d (start readonly svnserver on local machine)
mkdir new directory and cd into it
svn2git svn://localhost/path/to/local/svn/repo
Install EGit and restart.
(Eclipse) new pydev project -> browse to existing folder
Set up git locally (here)
set up to use git
team -> remote -> push -> (url from your project page)

Notes:

git svn clone file: did not work because of a file format problem (Expected FS format '2'; found format '4' at /usr/local/git/libexec/git-core/git-svn line 1535)

small issue with hist

2011-07-27T10:45:00.001-04:00

The left is the matplotlib plot - notice the slight rightward shift of the drawn histogram with respect to the solid black line.

The right is the pdf produced from the same plot, which is exactly where it should be.

Same deal, but with 'Agg' backend and printed to png

LyX + Mac + Math heavy document = keystone cops

2011-07-27T08:05:00.001-04:00

LyX has a really nice online math type setting system. I have a math heavy document that I wanted to type up using LyX. Pretty soon I had the classic problem of any feedback system with feedback delay - oscillations. I would type something, it would take about a quarter of a second to update, I would move my mouse to click on a formula, it would take a quarter of a second to change state and by that time I would have clicked twice, the document would have scrolled, and the formula would change state again, because I had clicked somewhere else.

Welp. Its back to good old latex and text editors for me. I don't mind the write-compile cycle.

LyX is good for text documents, but for math, at least on mac, it is so sluggish, it is more annoying than anything.

UPDATE:

I have found texmaker to be a very nice IDE on mac.

Word for Mac 2008 keyboard shortcuts

2011-07-24T19:00:00.000-04:00

http://grok.lsu.edu/article.aspx?articleid=7439

Tracking down backup space hogs

2011-07-21T19:49:00.000-04:00

Problem: Even though I haven't done many file changes the Time machine incremental backup is about 350 MB per flush (resulting in many GB per day for hourly backups). I don't generate THAT much work each day.

Using Time Machine and Time Tracker to track down the culprits.

Thunderbird:

global-messages-db.sqlite (170.5 MiB)

Firefox:

urlclassifier3.sqlite (50.6 MiB)
Cache 78.2 (MiB)

ipython + matplotlib crashing

2011-07-15T21:50:00.000-04:00

I've been having this problem ever since I changed to 1.0.1. It's not clear if it is ipython's fault or matplotlib's fault.

Today I got the first error message before segfault:

objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470

And Mac OS X error report:

Date/Time:       2011-07-15 21:47:54.500 -0400
OS Version:      Mac OS X 10.5.8 (9L30)
Report Version:  6
Anonymous UUID:  655916BF-D521-48C5-B8A3-FBC5E5FA6683

Exception Type:  EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Crashed Thread:  0

Application Specific Information:
objc[56139]: FREED(id): message removeFromSuperviewWithoutNeedingDisplay sent to freed object=0x19b37470

Thread 0 Crashed:
0   libobjc.A.dylib                0x90595bfa _objc_error + 116
1   libobjc.A.dylib                0x90595c30 __objc_error + 52
2   libobjc.A.dylib                0x90594637 _freedHandler + 58
3   com.apple.CoreFoundation       0x93faa676 CFArrayApplyFunction + 198
4   com.apple.AppKit               0x91a0aadc -[NSView _finalizeWithReferenceCounting] + 712
5   com.apple.AppKit               0x91a0a7e8 -[NSView dealloc] + 46
6   com.apple.Foundation           0x9524e68f NSPopAutoreleasePool + 1007
7   _macosx.so                     0x02932580 wait_for_stdin + 561 (_macosx.m:171)
8   readline.so                    0x0133bac0 call_readline + 432
9   org.python.python              0x001e04de PyOS_Readline + 254
10  org.python.python              0x0028dbfc builtin_raw_input + 412
11  org.python.python              0x00296165 PyEval_EvalFrameEx + 19429
12  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
13  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
14  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
15  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
16  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
17  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
18  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
19  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
20  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
21  org.python.python              0x0029634c PyEval_EvalFrameEx + 19916
22  org.python.python              0x002982dd PyEval_EvalCodeEx + 2109
23  org.python.python              0x002983f7 PyEval_EvalCode + 87
24  org.python.python              0x002bcf08 PyRun_FileExFlags + 168
25  org.python.python              0x002bddf3 PyRun_SimpleFileExFlags + 867
26  org.python.python              0x002cf902 Py_Main + 3122
27  org.python.python              0x00001f82 0x1000 + 3970
28  org.python.python              0x00001ea9 0x1000 + 3753

Bug in STIX fonts

2011-07-14T23:45:00.002-04:00

Grumble, grumble. MathML on Firefox on mac was going great guns, until I find that

\mathcal{N}

gives me garbage (but

\mathcal{M}

looks fine). I see from this bugreport, that this is an issue with the STIX fonts.

Making up a GET request in bottle

2011-07-14T12:48:00.000-04:00

Problem: I'm using bottle for a desktop browser based app. I needed to call a page view from an action. The page view normally gets some parameters encoded in a GET request.

Solution: The GET request can be faked by doing:

request.GET.append(key, value)

Since

request.GET

is just a dictionary.

subclipse, branching, merging etc.

2011-07-14T09:37:00.000-04:00

Sometimes merge/branch will refuse to work, saying your resource is out of synch. Do an 'update' and then a 'commit'.
After a rename or move + commit, subclipse will say resource is out of synch with the filesystem, just refresh,commit and continue
pydev explorer does not show the .* project files. These need to be added to svnignore (or committed if you wish). Use the eclipse navigator to see these file
switching a branch to be the trunk is annoying in svn: basically you have to move the folder over - so the current trunk has to be moved (possibly under branches with a new name) and the branch you want has to be moved over and renamed as trunk

Mac OS X MS Word and Equations

2011-07-13T16:05:00.000-04:00

I like to compose my equations in LaTeX. I find it a lot faster than the click interface of MS Word's own equation editor. I type up my notes in a LaTeX format so I already have the equations and its a terrible inefficient duplication of work to hunt-n-click them into ms word.

One solution is to print a pdf of all the equations, and then use the select tool in preview to copy the equation and then paste it into the word document.

The advantage in mac os x is that the copy is in vector format, which means that the inserted equation does not behave as an image - becoming pixellated at the worst moments, but rather scales gracefully.

That's gonna save me some time.

Removing the top and right lines in the axis box in matplotlib

2011-07-13T14:59:00.002-04:00

From here:

ax.axis["right"].set_visible(False)
ax.axis["top"].set_visible(False)

Where ax is an axis instance (you can get it using pylab.axis())

PS. Those lines are called spines, in case you haven't caught on

Pickling python classes

2011-07-07T19:19:00.001-04:00

Ok, this is the first thing in Python I have found to be annoying and nonintuitive. When you pickle a class object you need to make sure that the module for the class is explicitly in scope and imported.

I kept banging my head against this problem and only understood it after looking at this guy's blog.

When you pickle a class object included in the pickle file is a coded import statement telling the interpreter which module to import to look for the definition of the class. This leads to the following gotcha:

The following code will define a class, instantiate it and pickle it without problems

#File class_a.py
import cPickle

class A:
  def __init__(self):
    self.x = 22

if __name__ == "__main__":
  a = A()
  cPickle.dump(a, open('obja.pkl','wb'), protocol=-1)
  print cPickle.dumps(a, protocol=0)

Notice the stringification of the class : it begins with

i__main__

Now, we run the following code to load the object from the pickle:

import cPickle
m = cPickle.load(open('obja.pkl'))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/kghose/Research/2011/Papers/SpatialIntegration/Python/Sandbox/ in ()
----> 1 m = cPickle.load(open('obja.pkl'))

AttributeError: 'FakeModule' object has no attribute 'A'

Whaaa?

Now, say we change the original file by explicitly importing our module and instantiating the object as a class from that module (as would happen if our pickling code was in a different file from the class definition file):

#File class_a.py
import cPickle

class A:
  def __init__(self):
    self.x = 22

if __name__ == "__main__":
  import class_a
  a = class_a.A()
  cPickle.dump(a, open('obja.pkl','wb'), protocol=-1)
  print cPickle.dumps(a, protocol=0)

Now notice that the serialization begins with

iclass_a

which is our module name.

And when we try to load it, we don't get any errors.

Hide the dock icon of a program when it is running

2011-07-01T09:50:00.000-04:00

Go to the application's bundle (e.g. X11.app) and open the bundle contents. Navigate to Info.plist, open it with a text editor or plist editor. Add or change an item with the key "NSUIElement" and set its value to "1" (type = string). On relaunch of the application, its icon will not appear in the dock.

This is usefull for apps like X11 which are servers and should be running in the background.

Force FireFox to reload a short cut icon

2011-06-30T18:02:00.001-04:00

Type in the url of the icon file and load the image.

utf-8 encoded files in python

2011-06-30T14:08:00.000-04:00

From this PeP:

Adding a line that looks like:

# This Python file uses the following encoding: utf-8

Is sufficient to avoid the

no encoding declared; see http://www.python.org/peps/pep-0263.html for details

error

minidom.writexml handling unicode errors

2011-06-27T10:18:00.000-04:00

Problem:
Writing xml with unicode characters does not work, even with the encoding set to 'utf-8'. Always results in something like

UnicodeEncodeError('ascii', u'kr\xf6se1989', 2, 3, 'ordinal not in range(128)')

Solution (from here):
Use the codecs module and manually open the file in utf-8 mode for writing

import codecs
doc.writexml(codecs.open(fname,'wb','utf-8'), encoding='utf-8')

SQL query binding for a list using Python

2011-06-27T09:29:00.003-04:00

From here:

 l=[1,2,3,4] placeholder= '?' # Works For SQLite.  placeholders= ', '.join(placeholder for dummy in l) #puts in len(l) '?' marks, separated by ', ' query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders cursor.execute(query, l)

Firefox + Mac OS X + bangla fonts = disaster

2011-06-16T22:29:00.000-04:00

Please see this bugzilla issue.
In short:

Test any bangla font you install using Mac OS X's fontbook. Do not install any font that shows warnings or errors
The only Bangla font that I found to work with Firefox and not crash it on Mac OS X is at onkur.

Sources for Bangla unicode fonts

2011-06-16T21:25:00.000-04:00

slarc uchicago
Free bangla fonts (nongnu)

Strange inefficieny of pysqlite over sqlite (and apsw)

2011-06-14T15:07:00.000-04:00

I've been happily using pysqlite to access my sqlite databases from python. Suddenly, I find, however, that there is a query that runs much, much slower in pysqlite than in apsw and the sqlite shell. I've filed a bug report but it looks like I'll be switching to apsw until this thing gets resolved.

Another open source reference manager : Paperpile

2011-06-11T08:07:00.000-04:00

From a blog post here, I found out about Paperpile. I downloaded and ran it, but haven't used it, since I use my own, but Paperpile has a nice clean look. Though there is GPL code of Paperpile on github, the main website seems to indicate the author has hopes of commercializing (and closing the source) to Paperpile if things look successful.

Compress camera video for a windows machine

2011-06-08T10:11:00.001-04:00

mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=1:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o /dev/null MVI_0052.AVI
mencoder -ovc lavc -lavcopts vcodec=msmpeg4v2:vpass=2:vbitrate=375000:mbd=2:keyint=132:vqblur=1.0:cmp=2:subcmp=2:dia=2:mv0:last_pred=3 -oac copy -o output.avi MVI_0052.AVI

Python and word documents

2011-06-07T14:03:00.001-04:00

From here, using only standard python modules:

import zipfile, re

docx = zipfile.ZipFile('/path/to/file/mydocument.docx')
content = docx.read('word/document.xml')
cleaned = re.sub('<(.|\n)*?>','',content)
print cleaned

But, if you want to mess around in more detail in the document, then we can use the python-docx module.

discount bug

2011-06-06T12:28:00.000-04:00

import discount
a = discount.Markdown('')
a.get_html_content()
--> Yields TypeError
TypeError: 'NoneType' object is unsubscriptable
Should return a blank.

Found that markdown or markdown2 give comparable performance, so now no longer using discount.

Using adminer on Mac OS X

2011-06-03T13:42:00.000-04:00

adminer is a nice php based sqlite manager. I prefer the firefox plugin "sqlite manager" but it currently has a strange issue with FF5 that basically makes it unworkable, so I was looking for an alternative to tide me over.

I really don't want apache running all the time on my computer and don't want people browsing to my computer, so what I needed to do was:

Download the adminer php script into /Library/WebServer/Documents/
Change /etc/apache2/httpd.conf to allow running of php scripts (uncomment the line that begins: LoadModule php5_module
Start the apache server: sudo apachectl -k start
Operate the script by going to localhost
Stop the server: sudo apachectl -k stop

Bottlepy server exits during sleep (Mac OS X)

2011-06-03T10:32:00.000-04:00

Problem: Bottlepy server exits when computer comes out of sleep
Solution: Set "reloader" option to False.

Text search, sqlite bindings and Python

2011-05-31T12:28:00.001-04:00

c.execute("SELECT * FROM entries WHERE (title LIKE ? OR body LIKE ?) order by date desc", ("%%%s%%" %text, "%%%s%%" %text))

Notes

Putting %?% does not work
"%?%" does not work
Double percent sign (%%) due to python string substitution rules

Standalone browser based app

2011-05-28T16:50:00.001-04:00

Perhaps my long search for a convenient way to code database frontends that work locally ("desktop app") but leverage the power of a browser as a GUI has come to a successful point. This point involves:

Python (Naturally)
Bottle - a "microframework"
py2app - py2exe for windows

The only tricks making Bottle and py2app work well together were

You need to explicitly list the template files when calling py2app, otherwise they don't get included in Resources
I had to add a url route (which I called /quit) that called os._exit() to shut down the server (otherwise done using CTRL-C). just exit() and even SIGINT did not work. This is a bit extreme, but I think safe in this usage.

I was happy to see that the created app used the Resources directory as the working directory and the packager put all the files in this directory making things very transparent and worry free.

GET or POST?

2011-05-28T10:10:00.000-04:00

From the HTML standards.
Use POST when the operation has lasting changes on the world, like a database modification.

Setting up svn repo on local machine

2011-05-27T18:06:00.000-04:00

svnadmin create /path/to/repository

Point subclipse to that repo and make a new folder

3d wall decor

2011-05-15T21:12:00.000-04:00

So, there are artists who paint patterns on walls that turn into a recognizable figure if you are standing at the right perspective (e.g. here).

One easy way to generate these patterns is to set up a projector with a slide of the pattern that you want projecting from the vantage point you want, and then paint according to the projected pattern.

Nikon D40 (almost) 2 years on

2011-05-01T10:43:00.000-04:00

The D40 is an awesome camera, especially at the price I bought it ($375 with kit lens - but refurbished). It's years out of production now and has been superseded, but it does fine for my casual needs. The only two gripes I have about it, in order of importance are:

Auto white balance is sucky for incandescent lighting. So I always switch from auto to preset when I shoot indoors.
Only one func button makes it awkward. I have to pick between select ISO and select WB for that button. I think select WB wins out, but it would be nice to have dedicated buttons for that.

Nikon 35mm f/1.8 AF-S One year on

2011-05-01T10:39:00.000-04:00

This lens is very, very good. I had read on boards from people who would never take this lens off the camera. I find that most of my shooting is done with this lens. The kit lens that came with the D40 (The Nikon 18-55 mm) is very good, but the 35mm's 1.8 aperture is great.

Size of objects in Python

2011-04-18T22:36:00.001-04:00

struct.calcsize(fmt) - only gives size of fixed data types given by fmt
sys.getsizeof(obj) - (added in Python 2.7) gives the size in bytes of builtin types, but not the actual used memory
pympler.asizeof(obj) - gives the actual memory the object uses

Python + macbook sms (sudden motion sensor)

2011-04-12T13:26:00.000-04:00

Install PyAppleSMS. Use it as

import applesms (x,y,z) = applesms.coords()

Multiple logins to the same site

2011-04-11T12:45:00.001-04:00

Multifox is a great firefox add-on that allows you to switch between multiple accounts on the same site (Say you have two facebook accounts and you need to switch between them and are tired of logging out and then back in again). It's user interface is simple and effective.

Diff files in eclipse

2011-04-11T11:53:00.000-04:00

From here:

Add the two (or more) files you want to diff to your workspace.
Select all the files you want to diff.
Right click on one of the selected files and chose "Compare with -> each other"

bjobs

2011-04-05T18:27:00.000-04:00

Whenever I submit a job to the computing cluster I have to smile and wonder if the folks at Platform Computing planned their command naming very carefully so that one of their commands would endup as "bjobs".

Thunderbird bug

2011-04-05T17:47:00.000-04:00

If new mail comes in and goes through a filter and into a folder, if you are viewing that folder, the new mail does not appear until you view a new folder and then go back to the old folder.

Writing to NTFS partitions from Mac OS X 10.5

2011-04-03T14:42:00.001-04:00

Use NTFS-3G. It did something odd to my Finder, but after I restarted the computer as recommended things went back to normal, though there was a longer than usual start-up time.

Install multiprocessing module for Python 2.5

2011-04-03T14:32:00.000-04:00

The great multiprocessing module was included in the Python 2.6 distribution. If you need to use it on a box with Python 2.5 with no root access then do this:

From here download the package. And then install in user space by doing

python setup.py install --home=~

Medical costs in Malden/medford area

2011-03-30T12:35:00.001-04:00

chest x-ray - $33
routine mammogram - $100 ($10 for x-ray, rest for diagnosis)

Installing python in user space

2011-03-29T16:25:00.000-04:00

wget http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tar.bz2
cd Python-2.6.6/
./configure --prefix=/home/kg46/local
make
make install

git clone git://github.com/numpy/numpy.git numpy
~/local/bin/python setup.py install --prefix=/home/kg46/local/

Installing matplotlib in user space

2011-03-29T13:59:00.000-04:00

Problem:

Need to install matplotlib in user space, because I'm going to use a HPC cluster and their version of everything is just a little out of date.

Solution:

svn co https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/trunk/matplotlib matplotlib cd matplotlib python setup.py install --home=~

Now add to .bash_login the lines
PYTHONPATH='/home/kg46/lib/python/' export PYTHONPATH

And python is picking up the svn version

Spotlight : disabling and reenabling

2011-03-29T11:30:00.001-04:00

Problem:

I completely screwed up spotlight by killing the mds process AND deleting some spotlight files

Solution:

From here: The following commands can be used to properly enable and disable spotlight

Disable:
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist sudo launchctl unload -w /System/Library/LaunchAgents/com.apple.Spotlight.plist

Enable:
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist sudo launchctl load -w /System/Library/LaunchAgents/com.apple.Spotlight.plist

Important Bangla unicode rules

2011-03-26T12:42:00.002-04:00

Consonant conjucts are made using hashanta 0x09cd
Hashanta is generated using hashanta + zero width non joiner 0x09cd + 0x200c
The zwnj is only needed if the next letter would form a conjunct. Otherwise, 0x09cd is understood to be a hashanta
All indic scripts use the devnagari danda and double danda 0x0964,0x0965

Battlefield Bad Company 2 resources

2011-03-24T23:36:00.001-04:00

Input Method for Bengali (Bangla) on Mac OS X

2011-03-21T12:00:00.002-04:00

Wanted to have an IM for Bangla on the Mac
Didn't find any online
Found this tutorial from Apple on how to make an IM
Only problem is that the system does not seem to support the generation of conjunctions of vowel modifiers.
This means that each and every conjunction and consonant+vowel combination must be present in the file:
i.e. if we want to write কৌ (kou) we have to set kou = কৌ in the file: we can't build a rule. So this file must have all the possible combinations, like কু কূ etc. etc.
This is tedious to do by hand. I'm working on a small script to do these mix-n-matches and print out a input method file based on the rules I developed for lekho (see I knew that effort would come in useful some day!)
Jamil Ahmed has put up a bangla dictionary for Firefox.
Input keystrokes are case insensitive.

Google can block searches for you

2011-03-21T11:32:00.000-04:00

Do a search
Click on the link to go to some hit (This must be done first)
Hit back on the browser
The relevant hit now has a "Block" link below it
Click it if you want to block search hits from that site

I was initially looking for the "Block" link and didn't find it. It is not so well advertized that you have for first visit the site, before you can block it.

Time Machine and fsck_hfs (Mac OS X 10.5.x)

2011-03-18T13:12:00.000-04:00

Problem:You start a backup and Time Machine sits for ever in the "preparing backup" phase. It seems that fsck_hfs is suddenly doing a lot of work.
Solution: Click "Stop backing up" and then restart the backup

Mac OS X: Sparse Bundles and backups

2011-03-18T11:48:00.001-04:00

Sparse bundles, like read/write disk images, can be encrypted
Create a sparse bundle through the disk utility
The sparse bundle's size on disk corresponds to its content size (which I like, but you may not like that it gives away some information about its insides)
The change password function of Disk Utility (Images->Change Password) does not work through the GUI (the sparse bundle is greyed out and can not be selected). Use the command line instead:
hdiutil chpass /path/to/Encrypted.sparsebundle/
The sparse bundle is supposed to be easier on Time Machine and other backups, because it is not one monolithic structure.
Resizing the sparse bundle needs to also be done through the command line (GUI greyed out) and is done using
hdiutil resize -size 1.5g /path/to/bundle
Where 1.5g is the new size. It will ask you for the password.

Rubyforge repository public key (from Ben)

2011-03-08T11:28:00.002-05:00

Ben writes:

I don't know if you use rubyforge much anymore, but I found that the reason for this [he could not commit or update his code repository on rubyforge] was because they suddenly started requiring a public key and password authentication is now disabled.

http://www.ruby-forum.com/topic/1205279

Thought I'd give you a heads up so you don't spend the time trying to troubleshoot on your end. Maybe you can write a blog entry about it. It sort of annoys me that they didn't at least notify us by email that they were changing their login setup.

Blog entry plagiarized as requested.

Also see http://rubyforge.org/docman/view.php/5/460/faq.html#scm

Win XP: multiple time zones in system tray

2011-03-04T07:17:00.000-05:00

Use microsoft time zones application.
Install the .NET framework v1.1 (regardless of what other version you already have installed)

From here.

Genealogy software: GRAMPS

2011-02-13T10:52:00.000-05:00

GRAMPS is a nice, open source (GPLd) program written in python that allows you to build up a database of family relationships, from which you can make genealogy trees
GRAMPS runs on mac os, and they say it should run on windows and linux too
It is very intuitive and easy to use
I miss the ability to see descendant family trees with siblings etc (like this), but perhaps there is a trick to do that.

Mac OS X finder not displaying thumbnails properly

2011-02-03T11:49:00.000-05:00

Problem: Mac OS X Finder doesn't show the little thumbnails properly (where it used to before). For example, it will show thumbnails for .txt or .html files, but will only show a generic JPEG thumbnail for .jpg files
Solution: (From here) Go to (your home) > Library > Preferences folder, delete the com.apple.finder.plist file and the com.apple.sidebarlists.plist file if it exists, and then restart Finder from the force quit menu.

Cleaning a fancy, metal candle holder

2010-12-11T17:05:00.000-05:00

Candle wax softens and melts below the boiling point of water
Heat up water in a kettle, heat a cup of water in a cup, or if your kitchen water is properly hot, use the kitchen faucet
Using gloves to manipulate the holder flush the holder with boiling water
Use a spoon (preferably plastic) to scrape away the softening wax
Run the faucet into hard to get at nooks and crannies
This will do less damage to the holder than scraping it with a hard object like a knife or scissor

Using a web browser as a GUI for Python programs

2010-10-26T14:16:00.003-04:00

Resources:

References:

Eddie

2010-10-14T14:00:00.000-04:00

"Eddie is a programmers editor for MacOSX and Gnome Linux. Inspired by the Macintosh Programmers Workshop, Eddie brings some of it's powerful features into a modern lightweight and capable programmer's editor. Eddie is ideal for C/C++ development using makefiles and shell tools but can also be used with XCode projects. The Worksheet in Eddie is a full-featured shell that combines the power of bash and the ease of editing in a normal text window-like mode. Eddie is equally capable at editing HTML, JavaScript, Python, Ruby, Lua and many other languages.
Originally written for BeOS, it is currently actively developed for MacOSX and Gnome Linux."

Homepage

Split screen in word

2010-10-01T09:48:00.001-04:00

Use the splitter tool located on the right, just above the scroll bar

Drag the "splitter" down to where you want it

Remove horizontal line from Word

2010-09-29T22:42:00.000-04:00

Problem: There are horizontal lines in the word document that can't be selected, only moved.
Solution: Set the cursor above the line, go to Format->Borders and Shading and then remove the border.

Multidimensional find in pylab/numpy/matplotlib

2010-09-09T16:25:00.000-04:00

A = pylab.random((3,3))
print A
idx = pylab.find(A < .5) #Returns an unravelled index
A.flat[idx] = .5 #allows us to use this flat index directly
print(A) #and it works just fine

Parallel processing in a bash script - wait

2010-08-28T00:46:00.000-04:00

Use the 'wait' command.

some command &
some other command &

wait
command that needs the results of the previous two

Strange crash with pyglet, wxwindows and multiprocessing

2010-08-16T14:44:00.001-04:00

The following code will crash. Commenting out the wx import (note that wx is never used) will work as expected.

import multiprocessing as mp
import wx

def run():
import pyglet
w = pyglet.window.Window(resizable=True)
while not w.has_exit:
w.dispatch_events()
w.close()


p = mp.Process(target=run)
p.start()
p.join()

This may be related to the shadow window bug, but setting shadow window to False does not help.

Convert flv to mov (or other format) on a Mac

2010-08-11T10:30:00.000-04:00

Don't bother with ffmpeg or ffmpegX.

Svn revision numbers

2010-08-06T14:55:00.000-04:00

From here:

"Subversion's revision numbers apply to entire trees, not individual files. Each revision number selects an entire tree, a particular state of the repository after some committed change. Another way to think about it is that revision N represents the state of the repository filesystem after the Nth commit. When a Subversion user talks about “revision 5 of foo.c”, they really mean “foo.c as it appears in revision 5.” Notice that in general, revisions N and M of a file do not necessarily differ!"