Skip to main content

h5py and pandas for large array storage

I've actually gone back to pure hdf5 (via the h5py interface) for storing and accessing numerical data. Pandas via PyTables started to get too complicated and started to get in the way of my analysis (I was spending too much time on the docs, and testing out cases etc.).

My application is simple. There is a rather large array of numbers that I would like to store on disk and load subsets of to perform operations on cells/subsets. For this I found pandas to be a bad compromise. Either I had to load all the data all at once into memory, or I had to go through a really slow disk interface (which probably WAS loading everything into memory at the same time). I just don't have the luxury to fight with it so long.

I'm seeing that pandas has a (kind of) proper way of doing what I'm doing, but in h5py it just seems more natural and less encumbering :(

UPDATE: So, as previously mentioned, Pandas shines as a database substitute, where you want to select subsets of data based on some criterion. Pandas has a method (to_hdf) that will save a dataframe as a PyTables table that DOES allow you to do efficient sub-sampling without loading everything onto disk using the 'select' method, but even that is pretty slow compared to directly pulling things using h5py (and cumbersome). But it works really nicely for actual conditional select statements. Code updated to reflect this.

Timing information for randomly accessing 1000 individual cells from a 1000x1000 array of floats
h5py                - 0.295 s
pandas frame        - 14.8 s   (reloaded table on each lookup probably)
pandas frame_table  - 3.943 s  
python | grep 'function calls'
         95023 function calls in 0.295 seconds
         711312 function calls (707157 primitive calls) in 14.808 seconds
         1331709 function calls (1269472 primitive calls) in 3.943 seconds

import h5py, pandas as pd, numpy, cProfile

def create_data_files():
  r = numpy.empty((1000,1000),dtype=float)
  df = pd.DataFrame(r)

  with pd.get_store('pandas.h5','w') as f:
    f.append('data', df)

  with h5py.File('h5py.h5','w') as f:
    f.create_dataset('data', data=r)

def access_h5py(idx):
  with h5py.File('h5py.h5') as f:
    for n in range(idx.shape[0]):

def access_pandas(idx):
  with pd.get_store('pandas.h5') as f:
    for n in range(idx.shape[0]):

def slice_pandas(idx):
  with pd.get_store('pandas.h5') as f:
    for n in range(idx.shape[0]):'data', [('index', '=', idx[n][0]), ('columns', '=', idx[n][1])])

idx = numpy.random.randint(1000,size=(1000,2))'access_h5py(idx)')'access_pandas(idx)')'slice_pandas(idx)')


Post a Comment

Popular posts from this blog

Flowing text in inkscape (Poster making)

You can flow text into arbitrary shapes in inkscape. (From a hint here).

You simply create a text box, type your text into it, create a frame with some drawing tool, select both the text box and the frame (click and shift) and then go to text->flow into frame.


The omnipresent anonymous asked:
Trying to enter sentence so that text forms the number three...any ideas?
The solution:
Type '3' using the text toolConvert to path using object->pathSize as necessaryRemove fillUngroupType in actual text in new text boxSelect the text and the '3' pathFlow the text

Pandas panel = collection of tables/data frames aligned by index and column

Pandas panel provides a nice way to collect related data frames together while maintaining correspondence between the index and column values:

import pandas as pd, pylab #Full dimensions of a slice of our panel index = ['1','2','3','4'] #major_index columns = ['a','b','c'] #minor_index df = pd.DataFrame(pylab.randn(4,3),columns=columns,index=index) #A full slice of the panel df2 = pd.DataFrame(pylab.randn(3,2),columns=['a','c'],index=['1','3','4']) #A partial slice df3 = pd.DataFrame(pylab.randn(2,2),columns=['a','b'],index=['2','4']) #Another partial slice df4 = pd.DataFrame(pylab.randn(2,2),columns=['d','e'],index=['5','6']) #Partial slice with a new column and index pn = pd.Panel({'A': df}) pn['B'] = df2 pn['C'] = df3 pn['D'] = df4 for key in pn.items: print pn[key] -> output …

Drawing circles using matplotlib

Use the pylab.Circle command

import pylab #Imports matplotlib and a host of other useful modules cir1 = pylab.Circle((0,0), radius=0.75, fc='y') #Creates a patch that looks like a circle (fc= face color) cir2 = pylab.Circle((.5,.5), radius=0.25, alpha =.2, fc='b') #Repeat (alpha=.2 means make it very translucent) ax = pylab.axes(aspect=1) #Creates empty axes (aspect=1 means scale things so that circles look like circles) ax.add_patch(cir1) #Grab the current axes, add the patch to it ax.add_patch(cir2) #Repeat