Monday, November 24, 2014

A small caution when using virtualenvs

Virtualenvs are a great way to test and develop code in insulated containers. My main use case is to have an insulated environment where I can mock install a package I'm developing and ensure that I've taken care of all the dependencies so that a stranger can use the package as is. The virtualenvwrapper package is a great utility that simplifies managing multiple virtual environments.

One caution I have to observe is that packages installed out side the virtual environment can interfere in ways that make behaviors inside the virtual env very mysterious. For example, before I started using virtual environments seriously I had installed the nose and tox modules in my base python install. A month or so afterwards I had created a new test environment and was doing pip install -e . to test whether a package I was writing would install correctly on a fresh environment.

Everything installed fine, including an external package A my code needed. But, when I went to run nosetests or tox my code errored out, claiming package A had not been installed. BUT I CAN $%#@ SEE IT. LOOK, I TYPE pip list AND THERE IT IS!

What was happening, is that I was picking up both nose and tox from the base environment, which did not have A installed. I lost some more hair over this. People who know me will confirm that I can't afford such hair loss.

As a rule, if you want to use virtual envs, it is best to have a bare install of Python and then not install anything on top of that outside of a virtual environment.

git merge

I'm so used to git merge doing the right thing re: merging files - and possibly used to working mostly by myself - that when git merge fails I always expect something messy has happened. However, just now, I got a merge complain that amounted to this:

from setuptools import setup, find_packages

setup(
      ... some things ...

<<<<<<< HEAD
    ext_modules=cythonize('lib/*.pyx'),
    entry_points={
      # Register the built in plugins
              ....
      # Command line scripts
              ....
    }
)
=======
    install_requires=
             ....
    cython_ext='lib/*.pyx'
)
>>>>>>> origin/fix_setup

I was surprised since the changes I made and my colleague made were simple and non-overlapping. They just need to come in sequence.

I took a look at the git merge documentation which said

When both sides made changes to the same area, however, Git cannot randomly pick one side over the other, and asks you to resolve it by leaving what both sides did to that area.

What happened here was that there were no intervening lines. We both appended our changes below "... some things ..." and above ")".

git merge's algorithm can't assume (rightly so) that these are tandem changes. Which should come first? Should they both be kept? And so on. It's not a mess, but it still takes humans - actually people working on the code - to figure out that, in this case, both should be kept, and it does not matter which comes first.


Wednesday, November 12, 2014

cdef vs cpdef

In a .pyx file if you want a function that you declared as cdef (e.g. because you want inlining) to be visible to the Python interpreter, you need to use cpdef instead. However, it seems the cpdef version is a lot less efficient than cdef, and it might be better to have a wrapper of some sort instead.