PythonTeX: reproducible documents with LaTeX, Python, and more

A number of Python-based tools exist for creating reproducible documents with embedded code.

• Sphinx [17] was designed for creating Python documentation. Depending on the extensions employed, it can execute embedded Python code to provide at least basic reproducibility.
• Pweave [18] is essentially a Python version of Sweave. It allows Python to be embedded in reST, Sphinx, LaTeX, and markdown documents. Like Sweave and knitr, Pweave works by extracting all code from a document, executing the code, and then creating a copy of the original document in which code is replaced by its output. This makes compatibility with a range of document formats relatively simple, but typically does not provide close integration with any one document format.
• The IPython [19] notebook provides a browser-based, interactive interface in which code, output, and text may be combined. The notebook works with a single primary programming language, which is usually Python (although Ruby, Julia, and Haskell variants are in development). The Python-based notebook has the ability to execute independent snippets of code in several additional languages. The recently added nbconvert utility can convert the notebook into static formats including HTML, LaTeX, markdown, and reStructuredText.
• The Python package [20] for LaTeX provides a very minimal system for executing independent chunks Python code and including the output. It was the first LaTeX package for running Python code (2007).
• The SageTeX package [21] for LaTeX, released in 2008, provides a full-featured system for reproducible documents using the Sage [22] mathematics software, which is largely built on Python. The SympyTeX package [23] (2009) is largely derivative and works with Python rather than Sage. SageTeX provided inspiration for PythonTeX when I started development in the spring of 2011.

Only Pweave, IPython, and SageTeX have a relatively complete built-in feature set for reproducible documents with embedded code. PythonTeX was designed to provide features that are missing in some or all of these programs, with a focus on ease of use, performance, and superior LaTeX integration.

• Embedded code may be divided into independent user-defined sessions. When code is executed, sessions run in parallel. Different sessions may use different languages. Pweave, IPython, and SageTeX all work with a single primary session (and thus a single primary language), which can be inefficient for performing independent calculations.
• Built-in utilities are provided for tracking dependencies such as data. Code is re-executed when dependencies change. Utilities are also included for tracking created files such as figures. When file names are changed or files are no longer used, old versions are automatically cleaned up.
• Errors and warnings are synchronized with the LaTeX document, so that the user knows not just where in the code things went wrong, but also where that code is located in the document. Synchronization is less of an issue in IPython due to its interactive nature. SageTeX includes synchronization, but its system is based on Python's try/except exception handling and thus fails whenever errors prevent execution (such as syntax errors). Pweave lacks synchronization.
• As a LaTeX package, PythonTeX allows users to write valid LaTeX documents in which the full power of LaTeX is immediately accessible, rather than hybrid documents that contain LaTeX markup. Unlike in IPython and Pweave, it is simple and straightforward to pass information such as page dimensions from LaTeX into Python. It is even possible to create LaTeX macros that mix LaTeX with other languages.

Over the last few years the IPython notebook has arguably become the dominant tool for reproducible documents with Python. PythonTeX lacks the notebook's interactivity and simple compatibility with multiple output formats, but it does have some significant advantages. The notebook does not have built-in support for hiding code or for treating code output as part of the normal text of the document. It also does not allow variable values to be accessed within text cells. PythonTeX allows code, code output, or both to be displayed. Code output is interpreted as LaTeX by default (with built-in options for showing output verbatim), so it is easy to generate parts of the document text using code. Variable values may be accessed within normal text; inline executable code is also allowed. Writing a paper with IPython to conform to a journal's LaTeX standards may require editing the default templates and ensuring that the converted LaTeX output is acceptable. With PythonTeX, this is very simple since everything is written in LaTeX directly. Also, since the document is written entirely as valid LaTeX, any LaTeX editor may be used. This can significantly aid the writing process, since many editors provide forward and inverse search between the document source and the PDF output, autocompletion for references and citations, citation information on mouse hover, and similar features. Such tools are not available when writing in the IPython notebook.

I will begin with several basic examples of PythonTeX usage. The LaTeX source for most of the examples in this paper is available as supplementary materials. First, consider a 'Hello, world!' document, hello.tex.

Using PythonTeX simply involves adding the ⧹usepackage{pythontex} command in the preamble. (PythonTeX is part of TeX Live [24], version 2013 and later; the package also includes an installation script for other TeX distributions.) All code in the pycode environment will be executed, and anything that is printed will be brought into the document automatically. Compiling the document requires a three-step process:

• (i)  
  Run LaTeX. For example, pdflatex hello.tex. PythonTeX is compatible with the pdfTeX, XeTeX, and LuaTeX engines, so the document may be compiled with the latex, pdflatex, xelatex, or lualatex commands. In this step, the PythonTeX package saves all code, along with information about where it was located in the document, to a temporary file.
• (ii)  
  Run the PythonTeX script. In a typical installation, this may be done with the command pythontex hello.tex. In this step, code is loaded from the temporary file and executed, and its output is saved in a form accessible to LaTeX.
• (iii)  
  Run LaTeX again. All code output is brought in to create the final document.

In this case, the resulting PDF document contains the following text: 'Hello, world!'

Notice that the printed content is interpreted as LaTeX code; the ⧹emph command does not appear as literal text but rather acts to emphasize the word 'Hello'. Also be aware that the three-step compile process is only necessary when code needs to be executed. For example, if I were to add some normal LaTeX text to the document, running LaTeX once would be sufficient to update the PDF.

Using the pycode environment is sufficient if I simply want to execute code. In some situations, I might want to show the code in the document as well. In this case, the pyblock environment may be used. Its contents are executed and the code is typeset in the document verbatim (with optional syntax highlighting via Pygments [25]). Any printed content is not automatically included, however, since it might be desirable to insert additional text or LaTeX commands between the typeset code and its output. Instead, printed content may be accessed via the ⧹printpythontex command. Thus, the following code

produces

Copy-and-pasting numerical values is a significant source of potential error when writing a document. To prevent this, PythonTeX includes commands that may be used within normal text to access variable values. Suppose the following code is used to find the length of the hypotenuse of a right triangle.

If I later wish to refer to the values of the variables, I may use the ⧹py command, which returns a string representation of whatever it is given. The LaTeX code

When $ a =⧹py{a}$ and $ b = ⧹py{b}$ , then $ c = ⧹py{c}$ .

results in

When a = 3.0 and b = 4.0, then c = 5.0.

It is also possible to use ⧹py to perform simple calculations and return the result inline. For example, ⧹py{2**16} yields 65536.

Command versions of the three environments are provided as well. The ⧹pyc command executes code. Anything that is printed will be included automatically by default. The ⧹pyb command executes code and also typesets it, with optional syntax highlighting. Anything that is printed may be accessed via ⧹printpythontex. Finally, the ⧹pyv command only typesets code, with optional highlighting; nothing is executed. All of these commands (including ⧹py) take an argument delimited by a pair of curly braces or by a pair of matched symbols (like LaTeX's ⧹verb), so that character escaping may be avoided. Thus, ⧹py{x} and ⧹py#x# would both be valid.

Accessing variable values may be less straightforward when complex calculations are involved. The code required may be long enough that it is not practical or efficient to include it within the LaTeX document. In such cases, the code may developed in an external file (perhaps as a module) and then imported. This can keep the code that is actually in the document at a manageable length. External code could also be executed using the subprocess module, though that will typically be less desirable. Any external code can still be typeset in the document if desired, using the ⧹VerbatimInput command from fancyvrb [26], the ⧹inputpygments command provided by PythonTeX, or similar commands from other packages for typesetting source code.

Another potential source of error when writing a document is interactive console sessions. If an author fails to copy the complete session, code will be missing. If a library changes, necessitating new syntax, console examples will become outdated, and updating them may be tedious.

To prevent this, PythonTeX provides a pyconsole environment. All Python commands entered within the environment are treated as the input to an interactive console session, via Python's code module. Since the code is actually executed, and consists only of commands, it is easy to make modifications to keep examples current and correct errors.

The following commands could be used to recalculate the hypotenuse of the right triangle.

This is the result:

As before, it is possible to access variable values via a command; in this case, ⧹pycon must be used. Additional special-purpose commands and environments for working with console code are also provided.

PythonTeX can simplify figures by allowing plots to be adjacent to the code that created them. And data analysis can be automated. Suppose I have a data file, rand_walk.csv, that gives the trajectory of a random walk:

Zoom In Zoom Out Reset image size

The advantage of this approach is that it is very convenient to edit the plot. For instance, if a dimension needs to be adjusted or a color changed, there is no need to search for an external file of code that created the plot, open that file, edit the code, and then execute it. Instead, the plot code may be edited in the document, and then the document may be recompiled with PythonTeX.

This example also illustrates how documents can be programmed to adapt automatically to whatever external data is provided. The figure caption gives the minimum and maximum positions reached by the random walker on the x-axis, as well as the number of times that the walker crossed x = 0. These values are calculated from the data, so they would automatically adjust if data for a different random walk were used. Such automation would be particularly beneficial in cases when reports must be generated for several similar data sets.

Another application is the automatic creation of mathematical tables or derivations. The following example uses SymPy [13], a Python library for symbolic mathematics, to perform symbolic differentiation. SymPy's Derivative class creates an unevaluated derivative; the doit() method actually evaluates the derivative. The latex() function returns a LaTeX version of its argument. Since SymPy uses an italic d for differentials, but IOP journals use a Roman d, all d's are replaced by the ⧹rmd macro.

$$\begin{eqnarray*}\displaystyle \frac{{\rm{d}}}{{\rm{d}}x}\mathrm{sin}(x) & = & \mathrm{cos}(x)\\ \displaystyle \frac{{\rm{d}}}{{\rm{d}}x}\mathrm{sinh}(x) & = & \mathrm{cosh}(x)\\ \displaystyle \frac{{\rm{d}}}{{\rm{d}}x}\mathrm{csc}(x) & = & -\mathrm{cot}(x)\mathrm{csc}(x)\end{eqnarray*}$$

It also possible to use SymPy to perform step-by-step solutions. In the case below, a double integral is evaluated in two steps. SymPy's Integral class creates an unevaluated integral, while the doit() method evaluates the integral.

$$\begin{eqnarray*}\iint {x}^{2}{\mathrm{sin}}^{2}(y)\;{\rm{d}}x\;{\rm{d}}y & = & \displaystyle \int {x}^{2}\left(\displaystyle \frac{y}{2}-\displaystyle \frac{1}{2}\mathrm{sin}(y)\mathrm{cos}(y)\right){\rm{d}}x\\ & = & \displaystyle \frac{{x}^{3}}{3}\left(\displaystyle \frac{y}{2}-\displaystyle \frac{1}{2}\mathrm{sin}(y)\mathrm{cos}(y)\right).\end{eqnarray*}$$

PythonTeX can include error messages next to the code that created them. This can be particularly useful for documentation or instructional purposes. When PythonTeX is loaded with the makestderr option (⧹usepackage[makestderr]{pythontex}), versions of error messages appropriate for inclusion in the document are saved. (This involves the system described in section 6.) These may then be accessed via the ⧹stderrpythontex command. Consider this example.

The output is shown below.

Code:

a = 1

b = a + 2 +

Error:

File " ${\mathtt{\lt }}{\mathtt{𝚏𝚒𝚕𝚎}}{\mathtt{\gt }}$ ", line 2

b = a + 2 +

o

SyntaxError: invalid syntax

PythonTeX provides a stderrfilename package option that allows the form of the file name that appears in error messages to be customized.

It is possible to program LaTeX macros that mix LaTeX with other languages. This is one area where PythonTeX's close LaTeX integration is particularly evident. Mixing languages can be vastly simpler than programming an equivalent with pure LaTeX.

LaTeX commands are created with code in the form

⧹newcommand{⧹${\mathtt{\lt }}$ cmd name ${\mathtt{\gt }}$}[ ${\mathtt{\lt }}$ no. of args ${\mathtt{\gt }}$ ][ ${\mathtt{\lt }}$ default 1st arg ${\mathtt{\gt }}$ ]{${\mathtt{\lt }}$ body ${\mathtt{\gt }}$}

In the ${\mathtt{\lt }}$ body ${\mathtt{\gt }}$ definition, arguments are referred to as #1, #2, etc. The easiest way to use Python in LaTeX macros is to use the ⧹py command. For example, the following code creates a ⧹reverse command that reverses the order of whatever text it is given, using Python's string indexing.

⧹newcommand{⧹reverse}[1]{⧹py{"#1"[::-1]}}

Using ⧹reverse{1234567890} yields 0987654321.

Notice that the #1 is wrapped in quotation marks in the command definition, so that Python will treat it as a string; otherwise, it would be interpreted as a variable or a number. Also keep in mind that the #1 argument could have been passed to a function (perhaps defined in a previous pycode environment).

This approach to macro programming with PythonTeX will work in most cases except when literal percent and hash characters are involved, since these have a special meaning in LaTeX. Working around those cases can require LaTeX programming techniques involving verbatim arguments, catcodes, and tokenization that are beyond the scope of this paper.

By default, all code for a given language is executed in a single session. This is what allows a command like ⧹py to access variables defined in a separate environment like pycode. As a result, whenever any code is modified, everything must be re-executed. That can quickly degrade performance when several independent, computationally intensive operations must be performed.

To address this, PythonTeX allows code to be divided into user-defined sessions. All commands and environments take an optional argument that specifies the session in which the code is executed. A slow calculation might be isolated in its own session like this:

⧹begin{pycode}[slow]

${\mathtt{\lt }}$ code ${\mathtt{\gt }}$

⧹end{pycode}

${\mathtt{\lt }}$

${\mathtt{\gt }}$

Separating independent calculations into their own sessions not only prevents code from being executed unnecessarily. It also allows additional performance gains. Whenever two or more sessions need to be executed, they automatically run in parallel on systems with multiple processors, using Python's multiprocessing module.

One disadvantage of dividing independent calculations into separate sessions is that any user code created for interacting with LaTeX (perhaps using the tools in section 4) is no longer shared. Such code could be copied into each session, but that would just create one more thing to get out of sync. For such cases, PythonTeX provides a pythontexcustomcode environment. This allows code to be added to all sessions of a given type. For example,

⧹begin{pythontexcustomcode}{py}

${\mathtt{\lt }}$ code ${\mathtt{\gt }}$

⧹end{pythontexcustomcode}

would add ${\mathtt{\lt }}$ code ${\mathtt{\gt }}$ to all sessions that use the commands and environments whose names begin with py (pycode, ⧹py, etc).

The embedded-code approach to reproducible documents can make it easy to update, say, a figure. The code for creating the figure can be right next to where the figure is included in the document. But what happens if the code for creating the figure relies on data in an external file, and the data is updated? More generally, how can external dependencies be tracked? It would be possible to re-execute all code to make sure that everything is up-to-date (using the methods described in section 5.3), but that could be slow and inefficient. This is a case where the makefile-style approach to reproducible documents might seem more attractive.

PythonTeX includes features for dependency tracking, so that at least in many situations, the use of an external makefile or similar system can be avoided. The PythonTeX utilities class (section 4) includes an add_dependencies() method that allows dependencies to be specified. For example, in Python dependencies may be specified as follows:

pytex.add_dependencies('data1.txt', 'data2.txt', 'data3.txt')

Once a dependency has been specified, PythonTeX will check it for modification on all subsequent runs. By default, modification is checked via file modification time (Python's os.path.getmtime()), since this is fast even for large files. An SHA-1 hash may be used instead via the hashdependencies package option.

Tracking dependencies addresses external files that are read. There is a related issue with external files that are written. Suppose a plot is created. Later, the plot's file name is changed, leaving the old file with the old name behind. Many unused files could easily accumulate in such a fashion. To avoid this, the utilities class provides an add_created() method. Whenever code is re-executed, all tracked files that it created on previous runs are deleted.

It is relatively straightforward to automate the tracking of dependencies and created files. For example, the PythonTeX utilities class includes a custom open() method. If a file is opened for reading, it is added as a dependency and then opened; if it is opened for writing, it is added as a created file and then opened. When such custom functions are used for working with files, add_dependencies() and add_created() need not be used explicitly.

PythonTeX automatically caches all code output. This means that when no code needs to be executed, running PythonTeX (followed by LaTeX) is unnecessary; if PythonTeX is run under such conditions, it determines that there is nothing to do and exits immediately. Thus, as long as no code needs to be executed, PythonTeX adds essentially no overhead to compiling a document.

By default, PythonTeX executes code whenever it is new or modified, or if it produced errors when it was last executed. While that approach is often desirable, there may be times when it is inconvenient. For instance, if one session is producing errors, but the user is currently working on code in another session, it might be best only to execute modified code. This may be configured using the package option rerun (⧹usepackage[rerun= ${\mathtt{\lt }}$ choice ${\mathtt{\gt }}$ ]{pythontex}). Five different thresholds for execution may be specified:

• never: never execute code. PythonTeX will issue a warning if there is modified code.
• modified: only execute code that has been modified (including code with modified dependencies; see section 5.2).
• errors: execute all modified code and all code that produced errors on the last run.
• warnings: execute all modified code and all code that produced errors or warnings on the last run.
• always: always execute code.