Skip to main content

h5py and pandas for large array storage

I've actually gone back to pure hdf5 (via the h5py interface) for storing and accessing numerical data. Pandas via PyTables started to get too complicated and started to get in the way of my analysis (I was spending too much time on the docs, and testing out cases etc.).

My application is simple. There is a rather large array of numbers that I would like to store on disk and load subsets of to perform operations on cells/subsets. For this I found pandas to be a bad compromise. Either I had to load all the data all at once into memory, or I had to go through a really slow disk interface (which probably WAS loading everything into memory at the same time). I just don't have the luxury to fight with it so long.

I'm seeing that pandas has a (kind of) proper way of doing what I'm doing, but in h5py it just seems more natural and less encumbering :(

UPDATE: So, as previously mentioned, Pandas shines as a database substitute, where you want to select subsets of data based on some criterion. Pandas has a method (to_hdf) that will save a dataframe as a PyTables table that DOES allow you to do efficient sub-sampling without loading everything onto disk using the 'select' method, but even that is pretty slow compared to directly pulling things using h5py (and cumbersome). But it works really nicely for actual conditional select statements. Code updated to reflect this.

Timing information for randomly accessing 1000 individual cells from a 1000x1000 array of floats
h5py                - 0.295 s
pandas frame        - 14.8 s   (reloaded table on each lookup probably)
pandas frame_table  - 3.943 s  
python test.py | grep 'function calls'
         95023 function calls in 0.295 seconds
         711312 function calls (707157 primitive calls) in 14.808 seconds
         1331709 function calls (1269472 primitive calls) in 3.943 seconds

import h5py, pandas as pd, numpy, cProfile

def create_data_files():
  r = numpy.empty((1000,1000),dtype=float)
  df = pd.DataFrame(r)

  with pd.get_store('pandas.h5','w') as f:
    f.append('data', df)

  with h5py.File('h5py.h5','w') as f:
    f.create_dataset('data', data=r)

def access_h5py(idx):
  with h5py.File('h5py.h5') as f:
    for n in range(idx.shape[0]):
      f['/data'][idx[n][0],idx[n][1]]

def access_pandas(idx):
  with pd.get_store('pandas.h5') as f:
    for n in range(idx.shape[0]):
      f['data'].iloc[idx[n][0],idx[n][1]]

def slice_pandas(idx):
  with pd.get_store('pandas.h5') as f:
    for n in range(idx.shape[0]):
      f.select('data', [('index', '=', idx[n][0]), ('columns', '=', idx[n][1])])

#create_data_files()
idx = numpy.random.randint(1000,size=(1000,2))
cProfile.run('access_h5py(idx)')
cProfile.run('access_pandas(idx)')
cProfile.run('slice_pandas(idx)')

Comments

Post a Comment

Popular posts from this blog

Flowing text in inkscape (Poster making)

You can flow text into arbitrary shapes in inkscape. (From a hint here).

You simply create a text box, type your text into it, create a frame with some drawing tool, select both the text box and the frame (click and shift) and then go to text->flow into frame.

UPDATE:

The omnipresent anonymous asked:
Trying to enter sentence so that text forms the number three...any ideas?
The solution:
Type '3' using the text toolConvert to path using object->pathSize as necessaryRemove fillUngroupType in actual text in new text boxSelect the text and the '3' pathFlow the text

Drawing circles using matplotlib

Use the pylab.Circle command

import pylab #Imports matplotlib and a host of other useful modules cir1 = pylab.Circle((0,0), radius=0.75, fc='y') #Creates a patch that looks like a circle (fc= face color) cir2 = pylab.Circle((.5,.5), radius=0.25, alpha =.2, fc='b') #Repeat (alpha=.2 means make it very translucent) ax = pylab.axes(aspect=1) #Creates empty axes (aspect=1 means scale things so that circles look like circles) ax.add_patch(cir1) #Grab the current axes, add the patch to it ax.add_patch(cir2) #Repeat pylab.show()

Running a task in a separate thread in a Tkinter app.

Use Queues to communicate between main thread and sub-threadUse wm_protocol/protocol to handle quit eventUse Event to pass a message to sub-threadimport Tkinter as tki, threading, Queue, time def thread(q, stop_event): """q is a Queue object, stop_event is an Event. stop_event from http://stackoverflow.com/questions/6524459/stopping-a-thread-python """ while(not stop_event.is_set()): if q.empty(): q.put(time.strftime('%H:%M:%S')) class App(object): def __init__(self): self.root = tki.Tk() self.win = tki.Text(self.root, undo=True, width=10, height=1) self.win.pack(side='left') self.queue = Queue.Queue(maxsize=1) self.poll_thread_stop_event = threading.Event() self.poll_thread = threading.Thread(target=thread, name='Thread', args=(self.queue,self.poll_thread_stop_event)) self.poll_thread.start() self.poll_interval = 250 self.poll() self.root.wm_protocol("WM_DELETE…