Packages, Modules, Libraries

Video

Modules

A long war in programming has been fought between the Modular and the Monolithic, and by and large, the Modular is winning.

Module - as a term - should invoke a particular set of principles:

• Self-Contained: Only interacts with other objects when you tell it to
• Assemblable: Something you are able to construct larger wholes out of.
• Reusable: Something with functionality general enough to drop in to multiple projects.

Modules, as a concept, are fundamental to all successful large multi-developer projects. Similarly, they can particularly help with development - and debugging - of your own programs.

Over the coming weeks, we are going to be working with several modules, from several sources.

Importing Modules

Fundamentally, modules contain code you can pull in and use.

Python - like cmd, bash, and latex - has its own path that it searches to import modules. To find out what path it follows, we can use the sys python module - designed for System information.

Run the following:

import sys
sys.path

And Python will tell you its full search path, in the order it searches. Note that (unlike the previously described software) it is explicitly telling us it checks the current directory first.

Imports can be anywhere in a document - even within functions - but by convention, for readability and performance, you should put them at the top of your document right after your shebang (except in special cases).

We have just loaded the file sys.py. Everything within that file that we loaded is accessible with a "dot" notation - the variable path in that file, which is a list, is accessed by sys.path. Modules are objects!

The most notable consequence of this is that we can use our exploratory tools - help and dir - to explore modules. For example, we can use:

import math
print(help(math))
print(dir(math))

To get the help for, and a list of objects within, the module math.

Often, you want just one portion of a larger object. For that, there are some additonal syntaxes. You may have seen:

from math import sqrt
sqrt(2)

There may be situations where we want to explicitly name the imported object - for example, we may want to use the symbolic sqrt from sympy too. For that, we can use:

from math import sqrt as core_sqrt
from sympy import sqrt as symbolic_sqrt
core_sqrt(2) != symbolic_sqrt(2)

to create an alias by which to refer to an imported module.

This is also used to create shorthands - for example, it is common to see:

import numpy as np

throughout not just the scientific world but also within Numpy itself.

Creating Modules

Creating modules in Python is - mechanically - very easy: simply create a .py file with a function: if we save example.py containing:

#!/usr/bin/env python3

def module_function():
    """ An Example Function for Import."""
    return True

And then, from another Python file - or shell- in the same directory, run:

from example import module_function
module_function()

to import our function and use it.

However, you must be careful doing this - any code in example.py will be executed every time it is imported, so try to keep it to simple definitions of functions and other objects.

If you want to have a file that you can run as well as import, there is a standard Python syntax for that. Python defines a magic variable __name__, the "Name" by which the running Python knows the code file by. For modules, it is the module name - but for the specific .py file being executed, its value is "__main__". Therefore, to run a script when executing just that file, we can do:

#!/usr/bin/env python3

def module_function():
    """ An Example Function for Import."""
    return True

# Runs our example function when Python is called on file.
if __name__ == "__main__":
    module_function()

Testing

Specification

A good module defines a specification - a set of behaviors it can hopefully be relied upon to perform. We have already seen the main way to communicate the specification - docstrings.

However, docstrings are often empty words - a promise, sometimes a wishlist or an explanation, but nothing more substantial than that.

We can turn promises into guarantees by running our code before sharing it, and checking certain values.

At this point, you have already done that by hand - and seen the Example section of the docstring to get inspiration.

Assert

The simplest tests you can write are declaring what should result from a function call, and throwing an error if it is false. In Python, it is easy to do that explicitly:

if(1+1 == 2):
  raise AssertionError("Expected value of 1+1 is 2")

We can do that with the short statement assert:

assert 1+1 == 2

And even still pass a message:

assert 1+1 == 2, "Expected value of 1+1 is 2"

This is also a quick way to avoid doing computations on bad data or input. We will cover better ways for when your code will be used more often, but having something quick, readable, and functional for simple scripts is well worth it.

In python, it is common to put your tests within your module's .py file itself, in the if __name__ == "__main__" block:

#!/usr/bin/env python3

def module_function():
    """ An Example Function for Import."""
    return True

# Runs our example function when Python is called on file.
if __name__ == "__main__":
    assert module_function() == True, "module_function should return True"

Conforming to Your Specification

We have already seen some docstrings with examples. It is a great idea to come up with one or more examples for every common use case of your function - when designing, and to communicate.

It also helps to have a broad array of examples for testing, and to make sure that when users run your examples, they get the right result.

Enter the doctest module. It pulls tests from your docstring, and automatically checks them - writing helpful messages about what went wrong, when.

#!/usr/bin/env python3

def module_function():
    """ An Example Function for Import.

    >>> module_function()
    True
    """
    return True

# Runs our example function when Python is called on file.
if __name__ == "__main__":
    import doctest
    doctest.testmod()

Note that this is one of the special circumstances, mentioned earlier, with an import statement somewhere other than the top. Inside your testing block is the second most common location for import statements - calls to it are almost always deliberate, and sometimes your tests rely on objects from other modules.

doctest can even test for specific errors and messages:

#!/usr/bin/env python3

def module_function():
    """ An Example Function for Import.

    >>> module_function()
    Traceback (most recent call last):
        ...
    ValueError: Message
    """
    raise ValueError("Message")

# Runs our example function when Python is called on file.
if __name__ == "__main__":
    import doctest
    doctest.testmod()

Packages

Sometimes you will notice that you are importing deeper trees. These are called packages.

For example, the os package - which you can import with import os - contains a variety of useful functions for interacting with the operating system. We will talk about it more later.

But for specific path manipulations, it contains a "submodule" - os.path - which can be imported, imported from, and used like any other module. This submodule specifically focusses on path manipulations - for example,

from os.path import join
from os import getcwd

join(getcwd(),"Deeper")

uses functions from both to construct a string path to a directory within the current directory.

example/
    __init__.py

os/
    __init__.py
    path.py

os/
    __init__.py
    path/
        __init__.py

Libraries

Libraries are a general term you will often hear used interchangeably with modules or packages - or even larger collections of modules and packages.

The most common library to hear about is stdlib - the collection of Python modules and packages that are included in (almost) every distribution. The scientific python collection is often also referred to as a "library" - as are truly huge and versatile packages like numpy.

Worksheet

Today's worksheet will take you through creating a module which does some mathematical tasks.