Skip to main content

Posts

Showing posts from January, 2016

Python: Getting position in a compressed file

Problem: Python's gzip module is awesome, but poses a problem when we wish to report reading progress along a compressed file: .tell() returns the total bytes read but we usually only have the compressed size of the total file available, e.g. via os.path.getsize.

Solution: One trick is probably to extract the original size of the gzipped file from the last 4 bytes in the file, but, there are interesting caveats to this. Another method, slightly more principled, is to use .raw.fileobj.tell which, interestingly, reports the position on the compressed file, rather than the uncompressed stream.

git config core.fileMode false

Problem: After moving data from a Time machine backup to a new Mac (EDIT) Actually, I copied the data to a NTFS drive and then back to a Mac, so the blame goes to NTFS, all files in a repository show up as changed files under git, even though a diff proves that their contents are unchanged.

Solution: From a tip here, it turns out that it's related to file mode (and other metadata) changes which may occur during such a restore operation (for example the UID of the user may change, I guess) the solution is to tell git to ignore the file mode


git config core.fileMode false