Skip to main content

Reading spreadsheet data in Python: The lack of a good ODS reader

I try and keep long term data in as simple a format as possible, which means text where ever possible.

In earlier times I would enter data in excel spreadsheets and then read them from my Python programs using the xlrd package which is excellent. This works well, but in the back of my mind is the thought that someday Microsoft might do something funny with their business model making office software more janky to use and all my fears about keeping data in proprietary formats would come true. Oh, look, that day is today.

So, I'm completely abandoning the MS Office suite and going back to basic text files.

However, there is a tension between keeping tabulated data in a simple form, such as csv, and entering it in a convenient manner.

Excel, of course, nags you everytime you edit a csv file and save it. Libreoffice is excellent: it handles loading and saving in a very streamlined fashion. However, every time you open up the csv file you need to tell Calc what widths you want for the cells and how to wrap the cells (for textual data) till you get it how you like it so its comfortable to enter data.

The .ods format, because it is a standard, stands some chance of being read in the future if it is widely adopted, but really nothing beats text.

So, my compromise is to tabulate the data using LO and then, at the end, archive it as a csv file.

What I would really like is a python package that mirrors the abilities of xlrd and allows me to read my .ods file from Python. I could export to csv every time and use , but I will forget to do that once in a while, and it will usually be at an inopportune time and it will take me hours to chase down why I can't see the updated data. So, I would rather read the data directly from the .ods file.

The solutions for that, while not as polished as xlrd, do exist:
  1. odfpy - not really directly usable, more like an API
  2. simple-odspy - decently user friendly, uses odfpy
  3. odsreader - very easy to use, but loads whole spreadsheet into memory at a time.
UPDATE: I'm finding the use of odfpy and other modules slightly buggy for my spreadsheets. I've reverted to using Libre Office to generate .xlsx and using the tried and trusted xlrd module to read the files
Labels:

Comments

Post a Comment

Popular posts from this blog

A note on Python's __exit__() and errors

Python's context managers are a very neat way of handling code that needs a teardown once you are done. Python objects have do have a destructor method ( __del__ ) called right before the last instance of the object is about to be destroyed. You can do a teardown there. However there is a lot of fine print to the __del__ method. A cleaner way of doing tear-downs is through Python's context manager , manifested as the with keyword. class CrushMe: def __init__(self): self.f = open('test.txt', 'w') def foo(self, a, b): self.f.write(str(a - b)) def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): self.f.close() return True with CrushMe() as c: c.foo(2, 3) One thing that is important, and that got me just now, is error handling. I made the mistake of ignoring all those 'junk' arguments ( exc_type, exc_val, exc_tb ). I just skimmed the docs and what popped out is that you need to return True or...
Post a Comment
Read more

Store numpy arrays in sqlite

Use numpy.getbuffer (or sqlite3.Binary ) in combination with numpy.frombuffer to lug numpy data in and out of the sqlite3 database: import sqlite3, numpy r1d = numpy.random.randn(10) con = sqlite3.connect(':memory:') con.execute("CREATE TABLE eye(id INTEGER PRIMARY KEY, desc TEXT, data BLOB)") con.execute("INSERT INTO eye(desc,data) VALUES(?,?)", ("1d", sqlite3.Binary(r1d))) con.execute("INSERT INTO eye(desc,data) VALUES(?,?)", ("1d", numpy.getbuffer(r1d))) res = con.execute("SELECT * FROM eye").fetchall() con.close() #res -> #[(1, u'1d', <read-write buffer ptr 0x10371b220, size 80 at 0x10371b1e0>), # (2, u'1d', <read-write buffer ptr 0x10371b190, size 80 at 0x10371b150>)] print r1d - numpy.frombuffer(res[0][2]) #->[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] print r1d - numpy.frombuffer(res[1][2]) #->[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] Note that for work where data ty...
Post a Comment
Read more