Skip to main content

h5py in memory files

The h5py library is a very nice wrapper round the HDF5 data storage format/library. The authors of h5py have done a super job of aligning HDF5 data types with numpy data types including structured arrays, which means you can store variable lengths strings and jagged arrays. One of the advantages of HDF5 for large datasets is that you can load slices of the data into memory very easily and transparently - h5py and HDF5 take care of everything - compression, chunking, buffering - for you.

As I was playing around with h5py one thing tripped me up. h5py has an "in memory" mode where you can create HDF5 files in memory (driver='core') option which is great when prototyping or writing tests, since you don't have to clean up files after you are done.

In the documentation it says, even if you have an in-memory file, you need to give it a name. I found this requirement funny, because I assumed that the fake file was being created in a throw away memory buffer attached to the HDF5 object and would disappear once the object was closed or went out of scope.

So I did things like this:

import h5py
fp = h5py.File(name='f1', driver='core') # driver='core' is the incantation for creating an in memory HDF5 file
dset = fp.create_group('/grp1')
fp.keys() -> [u'grp1']

Great, things work as expected

fp1 = h5py.File(name='f1', driver='core') # driver='core' is the incantation for creating an in memory HDF5 file
fp1.keys() -> [u'grp1']


Closing the file didn't get rid of it! I have the data still!

del fp
fp2 = h5py.File(name='f1', driver='core') # driver='core' is the incantation for creating an in memory HDF5 file
fp2.keys() -> [u'grp1']

Whoah! Deleting the parent object doesn't get rid of it either!!!

fp3 = h5py.File(name='f2', driver='core') # driver='core' is the incantation for creating an in memory HDF5 file
fp3.keys() -> []

This surprised me a great deal. I had assumed the name was a dummy item, perhaps in order to keep some of their internal code consistent, but I did not ever expect a persistent memory store.

Welp, it turns out this is a *memory mapped* file and there is an actual file called f1 and f2 on the file system now. In order to make a file truly stored in memory, you have to use an additional option backing_store

fp = h5py.File(name='f1', driver='core', backing_store=False)
dset = fp.create_group('/grp1')
fp.keys() -> [u'grp1']

fp = h5py.File(name='f1', driver='core', backing_store=False)
fp.keys() -> []


Popular posts from this blog

Flowing text in inkscape (Poster making)

You can flow text into arbitrary shapes in inkscape. (From a hint here).

You simply create a text box, type your text into it, create a frame with some drawing tool, select both the text box and the frame (click and shift) and then go to text->flow into frame.


The omnipresent anonymous asked:
Trying to enter sentence so that text forms the number three...any ideas?
The solution:
Type '3' using the text toolConvert to path using object->pathSize as necessaryRemove fillUngroupType in actual text in new text boxSelect the text and the '3' pathFlow the text

Pandas panel = collection of tables/data frames aligned by index and column

Pandas panel provides a nice way to collect related data frames together while maintaining correspondence between the index and column values:

import pandas as pd, pylab #Full dimensions of a slice of our panel index = ['1','2','3','4'] #major_index columns = ['a','b','c'] #minor_index df = pd.DataFrame(pylab.randn(4,3),columns=columns,index=index) #A full slice of the panel df2 = pd.DataFrame(pylab.randn(3,2),columns=['a','c'],index=['1','3','4']) #A partial slice df3 = pd.DataFrame(pylab.randn(2,2),columns=['a','b'],index=['2','4']) #Another partial slice df4 = pd.DataFrame(pylab.randn(2,2),columns=['d','e'],index=['5','6']) #Partial slice with a new column and index pn = pd.Panel({'A': df}) pn['B'] = df2 pn['C'] = df3 pn['D'] = df4 for key in pn.items: print pn[key] -> output …

Python: Multiprocessing: passing multiple arguments to a function

Write a wrapper function to unpack the arguments before calling the real function. Lambda won't work, for some strange un-Pythonic reason.

import multiprocessing as mp def myfun(a,b): print a + b def mf_wrap(args): return myfun(*args) p = mp.Pool(4) fl = [(a,b) for a in range(3) for b in range(2)] #mf_wrap = lambda args: myfun(*args) -> this sucker, though more pythonic and compact, won't work, fl)