How to utilize Python Packages


Introduction

`setuptools`, `pip`, and related software allow for the utilization and distribution of python package ecosystems, complete with dependencies.  This guide intends to provide a brief guide to how to utilize the python packaging index (PyPI) in the control plane and iv-core plane environments.  It does not intend to be a thorough guide to python packaging, as vast literature exists on the web to said purpose.  Links to some good resources will be provided in the final section of this document.  Some good places to start are https://packaging.python.org/ , https://pip.pypa.io/ , and https://pypi.python.org/pypi (amongst others; search engines are also your friend).


Terminology

A `.py` file is a python module.  More correctly, if you have (for example) the `foo.py` file, this is the `foo` module and it may be imported if specified on `sys.path` with `import foo`.

A directory of modules toegether with an `__init__.py` file is a python package.  Python packages are distributed with a `setup.py` file that specifies the package name, the files to be distributed, dependencies on other python packages (including versions), and other metadata.

A python package index (PyPI) is a website that hosts a collection of packages.  The official PyPI is at https://pypi.python.org/pypi . While there are codified standards for how a python package index should work (see e.g. https://www.python.org/dev/peps/pep-0301/ ), for purposes of package installation a PyPI may be implemented by a static file server (e.g. nginx).

`distutils` is the original python package distribution library. `setuptools` followed as a de facto standard.  `pip` provides a wrapper and tooling around `setuptools`. The wheel format is a binary distribution format that intends to work around shortcomings of previous formats.

`virtualenv` is a tool to create isolated Python environments.


Python Packaging versus System Packaging

System software for production nodes will most often be installed with a system package manager: RPM/yum for Centos, apt/`.deb` for Ubuntu, and linux containers for CoreOS (just to name a few Linux distributions).  When available and when it is sufficient, the package maintainer version of software should most often be utilized for stability and interoperability with other system packages.  Package maintainer versions of python packages (as repackaged in the system format) contain the python modules with their paths set to use the system python interpreter, a record of their system package dependencies, and any support scripts (e.g. init files) and configuration set up as appropriate for the OS distribution.

It may be that one requires python software that does not have a system package form, or a more recent version than the canonical package, or a deployment (e.g. directory layout, and/or required interpreter, and/or virtualenv).  Additionally, if you are deploying in house python software, you may wish to use python packages as your distribution format. In these cases, it is desireable to distribute python packages.

An alternative is to convert python packages to the system package format.  A reference implementation using `fpm` and Docker is at https://github3.cisco.com/cloud-video/devops_utils/tree/master/pypedream . This does not allow virtualenv installations if that is a requirement, nor can this deal with binary packages.


The `setup.py` File

Python packages are distributed with a `setup.py` file that specifies the package name, the files to be distributed, dependencies on other python packages (including versions), and other metadata. An example may be seen at https://pythonhosted.org/an_example_pypi_project/setuptools.html .

Traditional pure source python packages (pre-wheel format) are distributed as tarballs.  Typically after expanding the archive one will be left with a directory with a setup.py script.  In such a directory one may run

    python setup.py install  # to install to the system

or

    python setup.py develop  # to install in place

Using `setuptools` this will attempt to locate and install the package and all of its dependencies recursively from the package index specified (https://pypi.python.org/simple/ by default).

Packages may also be installed with `pip`, e.g. `pip install decoupage`.


Python Package Index (PyPI)

A python package index (PyPI) is a website that hosts a collection of packages.  The official PyPI is at https://pypi.python.org/pypi . While there are codified standards for how a python package index should work (see e.g. https://www.python.org/dev/peps/pep-0301/ ), for purposes of package installation a PyPI may be implemented by a static file server , such as nginx or apache.

When one runs `pip install <package>`, `pip` will look in the package index listing -- https://pypi.python.org/simple/ by default -- for the package name and its dependencies.  If the link is available, it will pick a version of the package as appropriate for the system, download it, and install it.  This means you can host a PyPI exposing a directory in (e.g.) nginx and turning `autoindex on;`. Then you only need to copy the packages you want into subdirectories based on the name of the package.

Fetching Python Packages for Addition to a PyPI

The required directory structure may be discerned from https://pypi.python.org/simple/ . In brief, the lowercase name of the package to be installed is top level and the tarballs go into it.

Let's say you want to add https://pypi.python.org/pypi/a8e to your PyPI.


Testing Package Installation Using Virtualenv

Salt and Python Packaging

Salt has excellent utilities for installation of python packages

Control and iv-core Plane PyPIs

Mirroring package repos

Links


Python Packaging:

- https://packaging.python.org/

- https://packaging.python.org/distributing/

- http://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_packaging_api.html

- https://python-packaging-user-guide.readthedocs.io/

- https://www.python.org/dev/peps/pep-0376

- https://docs.python.org/3/distutils/examples.html#reading-the-metadata

Package Indices:

- https://pypi.python.org/pypi

- https://pypi.python.org/simple

- https://www.python.org/dev/peps/pep-0301/

setup.py:

- https://pythonhosted.org/an_example_pypi_project/setuptools.html

- https://docs.python.org/2/distutils/setupscript.html

pip:

https://pip.pypa.io/


setuptools:

- https://setuptools.readthedocs.io/en/latest/setuptools.html

virtualenv:

- https://virtualenv.pypa.io/

wheel:

http://pythonwheels.com/

http://wheel.readthedocs.io/en/latest/