Sunday, October 27, 2013

Posting a link to LinkedIn

Sometimes LinkedIn will not pull the metadata/images from a page you link to in a status update. In my case I was trying to link to a page on latimes. I found that if you get a tinyurl to the page, that works. I suspect that the url parser LinkedIn uses can not handle 'weird' characters in an url, like commas (this url had a comma) or else, can't handle urls beyond a certain length.

Saturday, October 26, 2013

Plotting state boundary data from shapefiles using Python

The great folks at have put up some of the data they collect so we can download and use it. On this page they have data relating to state boundaries. The files are available as zipped directories containing a shapefile and other metadata information. If you want to plot state boundaries and some state metadata (like zip code, state name) the .shp shapefile is sufficient. Assuming that the shape file is 'tl_2010_us_state10/tl_2010_us_state10.shp', some sample code using the pyshp package is:
import shapefile as sf, pylab

map_f = sf.Reader('tl_2010_us_state10/tl_2010_us_state10.shp')
state_metadata = map_f.records()
state_shapes = map_f.shapes()

for n in range(len(state_metadata)):
  pylab.plot([px[0] if px[0] <0 else px[0]-360 for px in state_shapes[n].points],[px[1] for px in state_shapes[n].points],'k.',ms=2)

for n in range(len(state_metadata)):
The pyshp package makes things so easy! Note that you can plot continuous lines instead of dots for the state boundaries, however, for some states like Alaska and Florida with islands, where the boundaries are not contiguous, you get nasty disjoint lines. Removing this requires much more processing (unless you do it by hand and break down states into "sub-states".

Friday, October 25, 2013

Pandas: the frame_table disk space overhead

When a Pandas DataFrame is saved (via PyTables) to hdf5 as a frame_table there is a varying amount of disk space overhead depending on how many columns are declared as data_columns (i.e. columns you can use to select rows by). This overhead can be rather high.

import pandas as pd, numpy

df = pd.DataFrame(numpy.random.randn(1000000,3),columns=['a','b','c'])
df.to_hdf('data_table_nocomp.h5','data') #-> 32 MB
df.to_hdf('data_normal.h5','data',complevel=9,complib='bzip2') #-> 21.9 MB
df.to_hdf('data_table.h5','data',complevel=9,complib='bzip2',table=True) #-> 22.5 MB
df.to_hdf('data_table_columns1.h5','data',complevel=9,complib='bzip2',table=True,data_columns=['a']) #-> 29.1 MB
df.to_hdf('data_table_columns2.h5','data',complevel=9,complib='bzip2',table=True,data_columns=['a','b']) #-> 35.8 MB
df.to_hdf('data_table_columns3.h5','data',complevel=9,complib='bzip2',table=True,data_columns=['a','b','c']) #-> 42.4 MB
df.to_hdf('data_table_columns3_nocomp.h5','data',table=True,data_columns=['a','b','c']) #-> 52.4 MB

Monday, October 14, 2013

The one thing I would have changed in 'Gravity'

Gravity is a great movie on many levels. It can't quite beat 2001 for solitude, desolation and a tiny cast, but its good. The three actors, Clooney, Bullock and Sir Newton do a great job and work well together, though there is not much by way of character development.

There is one raging issue that I have though. It only lasts 20 seconds in the movie and I don't quite know why its there. 

So here are Clooney and Bullock drifting towards the ISS. They get entangled in the parachute cords which stops their momentum relative to the ISS. Then, for some inexplicably reason, for 20 seconds Sir Isaac Newton goes on coffee break but the crew keep filming! Clooney is pulled by some mysterious phantom force that affects only him and Bullock but not the ISS. Clooney cuts himself loose and slingshots outward. Bullock kind of drifts back, so you know Sir Newton is slowly waking up from the coffee, but not quite, so it's not really clear what's going on.

Here's a tweak I would make to those 20 seconds that would keep Ike in the cast and keep everything else intact:
Clooney and Bullock are drifting towards the ISS. Clooney's jet pack runs out of gas, but their vector isn't close enough. They are going to miss the ISS by 20m or so and they will both die. Clooney unhooks himself from a semi-conscious Bullock, hurls her in the right vector for an ISS intercept and directs her into the airlock as he goes spinning into space.

Personally, as Bullock, I would hurry as fast as I could to get into the Soyuz to rescue Clooney, but whatever, fire in space is cool and I would have stopped and stared too.

I really can't understand why this discrepancy is there in movie where the folks seem to have worked somewhat hard to bring us the wonderful world of Newton's laws with no net external force.

Sunday, October 13, 2013

Reversed 28mm on a D5100 is 2:1 macro

Wooden ruler, mm grading
D5100 sensor size 23.6 mm × 15.6 mm
8mm of wooden ruler spans height of sensor
Magnification is 15.6mm : 8mm

About 2:1 which is great, but the focus distance is insanely close.

Saturday, October 12, 2013

Wednesday, October 9, 2013

Adventures with a Nikkor H 28mm f3.5

I wanted to find out what all this hoopla about old manual lenses was about, so I went looking for one. Apparently old manual lenses aren't THAT cheap, or else my definition of cheap is about a standard deviation below the mean. However, I did find a manual lens that fit my budget. Apparently the Nikon 28mm f3.5 isn't as hot an item as some other lenses.

The lens I got is older than I am, but in better condition. Nikon ended the run of this version of the lens in 1971. It's a non-Ai lens with the metering prong taken off (which makes it worthless for a collector, I guess). This suited me for two reasons: it made it cheap and it meant I could fit it on my D5100 (I read that you can fit the lens on the camera even with the prongs, but I don't believe it - the flash housing juts over the camera mount pretty closely, and I suspect the prongs would foul verified- the flash housing is JUST high enough that the prongs don't foul.). I inspected the lens for fungus and dust by shining a flashlight through the front and rear elements, and found the lens to be remarkably clean. There were no streaks on the front element.

The D5100 is a 'entry level camera' so it doesn't have a body motor. This turns a range of AF-D lenses into badly designed manual focus lenses. However, it also does not have the Ai-tab that allows higher end Nikons, like the D7100, to meter with Ai lenses. This allows the D5100 to accept non-Ai (or pre-Ai) lenses.

Now I understand what those coots mean when they rave about old manual lenses. AF-S and AF-D lenses are designed around an internal or body motor. You want the lens focus to zip from one end to the other as fast as possible. So you design the focus system to go from infinity to closest focal point with as little twist of the barrel as possible (called throw). Also, you want the motor to have as little resistance as possible, so you make the focusing ring as loose and light as possible. This sucks for manual focus.

Firstly, you have to be very delicate when you turn the focusing ring because you zip past the focus point really quickly. Secondly, if you breathe at the wrong moment, you lose your focus, because the ring is so loose (I'm looking at you 50mm f/1.8 AF-D). I use the red dot to confirm focus and I have to keep twiddling the ring to settle on the correct focus point, as if I'm some kind of CDAF algorithm.

(As a side note, I'm annoyed that the D5100 doesn't show the rangefinder when in full manual mode. You can't shoot a non-Ai lens in S or A mode, fair enough. But you get the rangefinder in those modes! I just can't understand this design decision).

A true manual focus lens, like this one, is very pleasant. The focus ring is damped. You have to push a little to move it, but it feels good and solid. Next, the throw is huge: a full 180 degrees. I was focusing and brought a subject into focus, and I tried to move beyond the focus point as I was used to, but I kept turning and turning and the subject did not go out of focus for what seemed like an eternity.

I kind of knew about these two quantities but I did not realize how big a deal they would be for shooting and getting in-focus shots. I've had disheartening experiences with the 50/1.8 AF-D and my experience with this 28mm is very different, where I always nailed the focus and got really bright, contrasty shots.

However, though I began to rave like the other coots, technically speaking, the better results I got from the manual focus lens is due to a complex mixture of factors and the comparison is not exactly fair. The main thing is that, once you have f1.8 you want to use it ALL THE TIME. So naturally, with the 50mm 1.8 wide open and trying to manual focus, I get a bunch of out of focus shots. I have much better images when I stop down to 3.5 or so. The 28mm lens starts out at 3.4 and is much wider, so that itself gives me a large DoF that masks any slop in focusing.

I was wowed by how sharp and contrasty the images from this lens were wide open. I'm used to hearing that one needs to stop down the lens a bit, and the largest aperture is really just there for low-light situations. However, a factor to consider here is that this is a 'FX' lens (from a time when everything was FX) used on a DX body. The engineers worked hard to get the center of the lens sharp and contrasty, the edges are probably more flawed, but I don't get to see that. Another score for the 'entry-level' camera...

I guess a real test would be for me to find a manual focus 50mm f1.8 or 1.4 and then compare my 50mm AF-D with that. Hmm, where is that piggy-bank?

Friday, October 4, 2013

Some notes from an ebay newbie

I was always suspicious of eBay (mostly because of Paypal). But I decided to jump in (like, what, about a few decades behind the curve) and try it out. I have fairly specific things I look for on eBay: photo stuff, my idea is that eBay is the giant garage sale in the ether and sometimes you can find exactly what you want for exactly what you want to pay for.

I don't have any deep observations, but I think one simple rule is important. I saw a lot of advice about sniping (bidding at the last second) and I think eBay does a very fair thing, in the true spirit of trying to find the appropriate price for an item.

Ebay allows you to set a maximum bid and will automatically bid just enough to keep you ahead up to the maximum. If someone comes in after you with a series of bids your bid always has precedence until your limit. I think this, if you are using eBay as a garage sale to find cheap items for a hobby, is the proper way to go.

When you first see an item, decide how much at most you want to pay for it, set that as your maximum bid AND THEN FORGET ABOUT IT.

EDIT: Now, however, I think this not literally how you should do it. You should write your maximum bid out, to remind yourself of what the real valuation of the item is. However, if everyone entered their max bid into eBay right away, all the prices would instantly saturate out, leaving the price at some small increment above the second highest bid. I think I understand "sniping" a little better - you should slowly work your way up to your max bid in relation to other bids, and as late as possible. However, the crucial key here is to remember your own maximum valuation and not budge from that, otherwise you will overpay.

eBay, of course, makes money off commissions, so it has an interest in you bidding on items and keeping increasing your bid. It's site is tuned to competitiveness in bidding, and I can see how people with a weakness for gambling or a misplaced competitive streak would get into a bidding war forgetting what their original base valuation for an item was. eBay is not really at fault here.

In general, item prices on open auction (where the lowest bid is very low) tend to be exponential, with prices racing up in the last 15min or so. I can see how, for expensive items, you can occasionally get a deal - there is probably a lot of noise in the price curve and since the growth is exponential the last few minutes/seconds of a bidding war can lead to large fluctuations in final price. The danger is that even with fluctuations, if you forget your original valuation, you are ending up paying more than the item is worth to you if you were not in a 'war': the only difference is in how much more you are paying than what the item was really worth to you.

One set of items I had been looking at were old 85mm primes. I was surprised at how much money people were paying for these old lenses. I can understand that a lens that has never been used, with its original packing and caps etc, would have a collector value. I could not understand how a lens with bits and pieces missing could sell for nearly $300. In all of these auctions I could see this exponential bid growth in the last hour. I guess there is a craze for Nikon 85mm f/1.8 D lenses well beyond their actual value as a photographic instrument!

Any how, it is possible to get deals from eBay, but it can't be something that there is a craze for, otherwise the item becomes over priced.

Wednesday, October 2, 2013

A simple exchange on eBay

I bought a 52mm-52mm coupler from a HK supplier (goes by the name of william-s-home). After I paid for the item, I noticed that the seller had a warning that the shipping could take 20-30 days and to email them if I wanted to cancel because I was just reading this note.

I emailed him and requested a cancellation. The seller was SO polite. We had a few exchanges and he/she was always extremely respectful.

I now have this image in my head of a venerable old Chinese trader who takes his business and reputation very seriously. For him, this is not just a way to earn money. It is a way of life, a principle, and things must be done correctly. The item cost $4.00 with shipping. It probably cost more than that for both of us in terms of the time spent emailing and completing the formalities for cancelling the transaction.

It was all very civilized and suddenly made me want to be a global trader, exchanging emails with people from far flung places in the globe, because life is too short and the world is too big and there are too many good people out there to not get to know them.