PART I — Building Python Packages

6 min readFeb 12, 2021

Packages and modules are the most efficient way to organize the code into smaller units and improve quality. The code gets easier to change, test, reusable, and easier to understand. When correctly done, the contribution to bigger applications becomes easier thanks to packages and namespaces.

This post walks you through the basic steps of packaging your Python code.

1. Installing Python and configuring it.

pip is included in python 2 (2.7.9+) and python 3 (3.4+) when you install python. To install python for your OS please use the following link and instructions:

https://pip.pypa.io/en/stable/installing/

Python >=3.4 can self-bootstrap pip with the built-in ensurepip module. Refer to the standard library documentation for more details. Make sure to upgrade pip after ensurepip installs pip.

See the Using Linux Package Managers section if your Python reports No module named ensurepip on Debian and derived systems (e.g. Ubuntu).

2. Isolating the runtime environment

Application-level isolation is the easiest approach where the Python interpreter and the packages used for the application is isolated within. This is very easy to set up, and enough to ensure proper isolation for developing small projects and packages.

python -m venv my-env

my-env is the desired name for the new environment. The command creates a new my-env container (folder or directory) in the current path. the my-env directory is the home for the following directories out which you’ll be dealing with directory: this directory stores the new Python executable and any other executable / script provided by the packages referenced. To use the isolated environment, you need to activate it by running :

source my-env/bin/activate

and when you don’t want it anymore, then just use

deactivate

As a good practice, you should store all the dependencies your project has in the requirements.txt file. This file Please check here for further info on requirements file. Now, when your virtual env is ready, you run the following command to install the dependency packages in your virtual env.

pip install -r requirements.txt

There are other types of virtual environment such as virtualenv that are not discussed here.

Packaging your code

To package python, need a couple of tools to create a package which we have introduced in previous sections

PIP for installing packages from PyPI
venv for app-level isolation

The Python Packaging user guide recommends the following package creation and distributions:

setuptools which is used to create source distributions and the underlying project definition.
wheels to create built distributions
twin is the last tool that is simply responsible for uploading package distribution to PyPI.

The root directory of your package should contain setup.py , a script to define any metadata that you’d like to set up for the package.

here’s an instance of the setup.py code and setting some basic metadata:

from setuptools import setup
setup(
    name='newbies_package'
)

To see what commands are available, run the help switch:

python setup.py --help-commands

you can call the setup.py command to create a source (.zip or .tar, etc. ) or build distribution ( binary distribution), and even upload to the PyPI

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]   or: setup.py --help [cmd1 cmd2 ...]   or: setup.py --help-commands   or: setup.py cmd --help

for more complex build and distribution of the package, supply any of the optional arguments to the setup.py script. setup.cfg the file can be used to store default parameters alongside the source code. This file extracts and decouples the distribution flow of your project from the code, with the added benefit of sharing with others who can see how your package is built and distributed.

Below is an example setup.cfg file with some default arguments:

[global]
quiet=0[bdist_wheel]
universal=0[sdist]
format=tar

The example configuration ensures the source distribution is created in zip format, but the distribution is dependent on the python version. Also, the output on every command is shown.

The `Setuptools` module

The setuptools is an enhancement of distutil and uses it at its core for the packaging and distribution of the package. (please see https://setuptools.readthedocs.io/en/latest/setuptools.html for more details). When building packages and their distribution, the distutil module browses the package directory to include the files in the archive. By default, this is the order it includes the files:

All Python source files implied by the py_modules, packages, and scripts arguments
All C source files listed in the ext_modules argument
Files that match the glob pattern test/test*.py
Files named README, README.txt, setup.py, and setup.cfg

in case you’d like to override or explicitly exclude/include files, you may use MANIFEST.in file

include LICENSE 
recursive-include *.txt *.py
include HISTORY.txt
include README.txt 
include CHANGES.txt 
include CONTRIBUTORS.txt

other important metadata you can pass to the setup() functional areas listed here:

author: This is the name of the package author or organization that takes care of it.
author_email: This is the contact email address.
url: This is the URL of the project. license: This is the name of the license (GPL, LGPL, and so on) under which the package is distributed.
packages: This is a list of all package names in the package distribution; setuptools provides a small function called find_packages that can automatically find package names to include.
namespace_packages: This is a list of namespace packages within package distribution.
description: This includes a few sentences to describe the package.
long_description: This includes a full description that can be in reStructuredText (default) or other supported markup languages.
long_description_content_type: this defines the MIME-Type of long description; it is used to tell the package repository what kind of markup language is used for the package description.
keywords: This is a list of keywords that define the package and allow for better indexing in the package repository.

Additionally, you may pass other metadata to the project in the form of a list of classifiers. Python has more than 700 classifiers available on PyPI grouped in a few major categories such as (for more categories and namespaces please check https://pypi.org/classifiers/)

Operating system
Programming language
Framework
Environment
Topic

An Example setup script

from setuptools import setup  
setup(     
    name="myproject",     
    version="0.0.1",            
    description="mypackage project short description",
    long_description="""
         Longer description of mypackage project         
         possibly with some documentation and/or         
         usage examples     """,     
    install_requires=[         
         'dependency1',         
         'dependency2',         
         'etc',     
    ]
 )

To include version, you can use this in the __init__.py file:

# version as tuple 
VERSION = (0, 1, 1) # string created from tuple 
__version__ = ".".join([str(x) for x in VERSION])

then you can use the following setup.py script:

from setuptools import setup
import os
def get_version(version_tuple):
    if not isinstance(version_tuple[-1], int):
        return '.'.join(map(str,version_tuple[:-1])) +version_tuple[-1]
    return '.'.join(map(str,version_tuple))init = os.path.join(
   os.path.dirname(__file__),'src', 'some_package', '__init__.py'
)
version_line=list( filter(lambda l: l.startswith('VERSION'),open(init)))[0]PKG_VERSAION=get_version(eval(version_line.split('=')[-1]))
setup(
    name='some-package',version=PKG_VERSION,
)

Working with packages during development

When you need to use setuptools to install packages directly from the local project sources you can use:

pip install <project-path>

You can uninstall the packages you install ed locally, using

pip uninstall <package-name>

When you use the setup.py install command, the packages are copied to the site-packages directory of the current virtual environment. if you make a change to the source of the packages, you require to reinstall it. And you’re right, it’s a headache.

Instead, you can create a special link to the project sources in the development directory (site-packages) so the package sources can be edited without the need to reinstall them. The package sources are made available in the sys. path as if they were installed normally.

you can use pip for this matter as:

 pip install -e <project-path>

Namespacing your packages

as a best practice and to avoid dependency conflicts, you use namespace packages with which you store the source tree for each of the sub-packages independently as shown here

mypackag.sql/
--tech
   |__sql
       |__ __init__.py

and to install them independently

pip install tech.sql tech.templates

References:

1- Python Packaging Authority https://www.pypa.io/en/latest/

2- Building and Distributing Packages using Setuptools: https://setuptools.readthedocs.io/en/latest/setuptools.html#id1

3- Must have: Expert Python Programming — Third Edition, by Michal Jaworski, Tarek Ziade