IPython Magic

During my work I often make use of a great piece of software called Ipython, which makes it a great deal easier to work with Python interactively. Despite its appeal, there is one thing about IPython which I noticed was starting to hinder me, especially when working on notebooks where I am doing a set of related, but not identical, tasks. Luckily, due to Ipython's modular design I was able to write a really simple (4 lines!) "magic" function which solves my problem.

The Workflow

My general workflow is that I have a notebook with several sections, each dedicated to doing a particular calculation; usually each section has some (small) differences with respect to the others (e.g. changing parameters). It is often not possible or time-efficient to be constantly refactoring the sections when it becomes clear that a certain operation is common between sections. In addition each section necessarily takes more than 1 cell, as I often have some setup, an intense calculation, and some post-treatment of the data; I don't necessarily want to re-run the calculation every time! The notebook in my workflow can thus be seen as a set of groups of related cells, and the groups should not have any effect (unless explicitly stated) on one another.

The Problem

While the above view of the organisation and interactions within an Ipython notebook is useful for my workflow it is not (unfortunately) how the notebook actually works; all the cells within the notebook run in a single namesepace. This quickly leads to a situation where names get reassigned and working out which names refer to which objects becomes rather difficult. This has already caught me out several times (especially loop variable names, due to Python's scoping rules).


It is entirely possible that my whole workflow is bogus, and that I should keep my notebooks shorter and have more of them, thus mitigating the namespace clashes to some degree. While this could work to a certain extent, if ever I wanted interaction between the notebooks I would have to add some new layer to handle this (e.g. using the filesystem).

Another possibility would be to try and make my names even more descriptive: result becomes simulation1_result and the like. We can quickly see that this would become very cumbersome and would also not make it easy to group related objects together. The obvious extension would be to create a dictionary called simulation1 and then have the objects stored in this dictionary like simulation1['result']. While we could alleviate the syntactic noise somewhat by creating a dictionary class which allows elements to be accessed like attributes, simulation1['result'] becomes simulation1.result, this is still quite bulky given that most of the time manipulations within a particular cell will be on objects related to a particular context (simulation1 in this example). This last remark leads us to the idea that we'd like to have some way to be able to group all names defined in a cell so that they all belong to a particular namespace. Luckily IPython provides a mechanism for acting on whole cells at a time, namely via cell magics! The sort of interface that we'd like is the following:

%%namespace simulation1
def big_sim(...):
result = big_sim(...)

%%namespace simulation2
def big_sim(...):
result = big_sim(...)

So that big_sim and result defined in the two cells do not clobber one another. This is actually very easy to do; my solution consists essentially of only 4 lines!

class Namespace(Magics):

    _namespaces = defaultdict(dict)

    def namespace(self, line, cell):
        line = line.strip()
        code = compile(cell, line, 'exec')
        local_ns = self._namespaces[line] if line else {}
        exec(code, get_ipython().user_global_ns, local_ns)

The Namespace class is just to provide a framing namespace for _namespaces, which is a map from names (of namespaces) to namespaces. A namespace itself is, of course, just a map from names (of objects) to objects. The key line is the last one, where we execute the cell within the global namespace of the IPython kernel, but the local namespace given by one of the stored namespaces in _namespaces. In addition, if no name is given to the namespace when invoking the namespace, a temporary namespace (dict) is used.