How to Rock Python Packaging with Poetry and Briefcase

NOTE: Briefcase now automatically creates a new project with a pyproject.toml file by running briefcase new. I would recommend following the BeeWare Tutorial to setup a new project if you want to use Briefcase for packaging it.

As part of modernizing Gaphas, the diagramming widget for Python, I took another look at what the best practices are for packaging and releasing a new version of a Python library or application. There are new configuration formats and tools to make packaging and distributing your Python code much easier.

A Short Background on Packaging

There are two main use cases for packaging:

1. Packaging a Library - software that other programs will make use of.
2. Packaging an Application - software that a user will make use of.

This may not be a completely accurate definition because software does not always fit cleanly in to one of these bins, but these use cases will help to keep focus on what exactly we are trying achieve with the packaging.

The Library

The goal for packaging a library is to place it on the Python Packaging Index (PyPI), so other projects can pip install it. In order to distribute a library, the standard format is the Wheel. It allows for providing a built

distribution of files and metadata so that pip only needs to extract files out of the distribution and move them to the correct location on the target system for the package to be installed. In other words, nothing needs to be built and re-compiled.

Previously if you wanted to achieve this, it was common to have four configuration files:

1. setup.py - The setup script for building, distributing and installing modules using the Distutils.
2. requirements.txt - Allow easy install of requirements using pip install -r
3. setup.cfg - The setup configuration file
4. MANIFEST.in - The manifest template, directs sdist how to generate a manifest

The Application

The goal for packaging an application is get it in the formats where you can distribute it on the different platforms for easy installation by your users. For Windows this is often an exe or msi. For macOS this is an app. For Linux this is a deb, flatpak, appimage, or snap. There is a whole host of tools to do this like: py2exe, py2app, cx_Freeze, PyInstaller, and rumps.

pyproject.toml

On the packaging front, in May of 2016, PEP 518 was created. The PEP does a good job of describing all of the shortcoming of the setup script method to specify build requirements. The PEP also specified a new configuration format call pyproject.toml. If you aren't familiar with TOML, it is human-usable and is more simple than YAML.

The pyproject.toml replaced those four configuration files above using two main sections:

1. [build-system] - The build-system table contains the minimum requirements for the build system to execute.
2. [tool] - The tool table is where different tools can have users specify configuration data.

The Tools

Making use of this new configuration format, a tool called flit has been around since 2015 as a simple way to put Python Libraries on PyPI.

In 2017, Pipenv was created to solve pain points about managing virtualenvs and dependencies for Python Applications by using a new Pipfile to manage dependencies. The other major enhancement was the use of a lock file. While a Wheel is the important output for a Library, for an Application, the lock file becomes the important thing created for the project. The lock file contains the exact version of every dependency so that it can be repeatably rebuilt.

In 2018, a new project called Poetry combined some of the ideas from flit and Pipenv to create a new tool that aims to further simplify and improve packaging. Like flit, Poetry makes use of the pyproject.toml to manage configuration all in one place. Like Pipenv, Poetry uses a lock file (poetry.lock) and will automatically create a virtualenv if one does not already exist. It also has other advantages like exhaustive dependency resolution that we will explore more thoroughly below.

For Application distribution, I am going to focus on a single tool called Briefcase which along with the other set of BeeWare tools and libraries allows for you to distribute your program as a native application to Windows, Linux, macOS, iOS, Android, and the web.

Tutorial

With the background information out of the way, lets work through how you can create a new Python project from scratch, and then package and distribute it.

Initial Tool Installation

To do that, I am going to introduce one more tool (the last one I promise!) called cookiecutter. Cookiecutter provides Python project templates, so that you can quickly get up to speed creating a project that can be packaged and distributed without creating a bunch of files and boilerplate manually.

To install cookiecutter, depending on your setup and operating system, from a virtualenv you can run:

$ pip install cookiecutter

Next we are going to install Poetry. The recommended way is to run:

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

TestPyPI Account Sign-Up

As part of this tutorial we will be publishing packages. If you don't already have an account, please register for an account on TestPyPI. TestPyPI allows you to try distribution tools and processes without affecting the real PyPI.

Create Your Project

To create the Python project, we are going to use the Briefcase template, so run cookiecutter on this template:

$ cookiecutter https://github.com/pybee/briefcase-template

Cookiecutter will ask you for information about the project like the name, description, and software licence. Once this is finished, add any additional code to your project, or just keep it as is for this demo.

Change your directory to the app name you gave (I called mine dantestapp), and initialize git:

$ cd dantestapp
$ git init
$ git add .

Create a pyproject.toml Configuration

Poetry comes equipped to create a pyproject.toml file for your project, which makes it easy to add it to an existing or new project. To initiliaze the configuration run:

$ poetry init

The command guides you through creating your pyproject.toml config. It automatically pulls in the configuration values from the briefcase-template that we created earlier so using the default values by hitting enter after the first six questions will be fine. This is what it provided for an output:

Package name [dantestapp]: 
Version [0.1.0]: 
Description []: 
Author [Dan Yeaw <dan@yeaw.me>, n to skip]: 
License []: MIT
Compatible Python versions [^3.7]: 

Define Dependencies

The configuration generator then asks for you to define your dependencies:

Would you like to define your dependencies (require) interactively? (yes/no) [yes]

Hit enter for yes.

For the next prompt Search for package: enter in briefcase. We are setting briefcase as a dependency for our project to run.

Enter package # to add, or the complete package name if it is not listed: 
 [0] briefcase
 [1] django-briefcase

Type 0 to select the first option. and hit enter to select the latest version. You now need to repeat this process to also add Toga as a dependency. Toga is the native cross-platform GUI toolkit. Once you are done, hit enter again to complete searching for other dependencies.

Define Development Dependencies

At the next prompt the config generator is now asking us to define our development dependencies:

Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes]

Hit enter to select the default value which is yes.

We are going to make pytest a development dependency for the project.

At the prompt Search for package: enter in pytest.

Found 100 packages matching pytest

Enter package # to add, or the complete package name if it is not listed: 
 [ 0] pytest

You will get a long list of pytest packages. Type 0 to select the first option. and hit enter to select the latest version. Then hit enter again to complete searching for other development dependencies.

Complete the Configuration

The final step of the configuration generator summaries the configuration that it created. Notice that first three sections are tool tables for Poetry, and the final one is the build-system table.

[tool.poetry]
name = "dantestapp"
version = "0.1.0"
description = ""
authors = ["Dan Yeaw <dan@yeaw.me>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.7"
briefcase = "^0.2.8"
toga = "^0.2.15"

[tool.poetry.dev-dependencies]
pytest = "^4.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

The dependencies use a "caret requirement", like python = "^3.7". This makes use of semantic versioning. So in this example if Python 3.8 is released, then it will automatically update to this version. But, it won't update to 4.0 automatically, since that is a major version change. If we put in our configuration "^3.7.2", then it would automatically update to 3.7.3 which it is released, but not 3.8, since that is a new minor version.

There are also "tilde requirements" that are more restrictive. So if you enter `python = "~3.7" it will only allow update to the next patch level, like from 3.7.2 to 3.7.3. The combination of caret and tilde requirements allows you to get updates to your dependencies when they are released, but puts you in control to ensure that incompatible changes won't break your app. Nice!

The final prompt asks: Do you confim generation? (yes/no) [yes]. Go ahead and hit enter to confirm. Congrats, you have generated a pyproject.toml configuration!

Install Dependencies

OK, the hard work is over, we have created our project and finished the configuration. Now it is time to see how Poetry and Briefcase really shines.

To install the dependencies that you defined in the pyproject.toml, just run:

$ poetry install

Poetry includes an exhaustive dependency resolver, so it will now resolve all of the dependencies it needs to install Briefcase, Toga, and pytest. It will also create a poetry.lock file which ensures that anyone using your program would get the exact same set of dependencies that you used and tested with.

Notice that we also did not create or specify a virtual environment. Poetry automatically creates one prior to installing packages, if one isn't already activated. If you would like to see which packages are installed and which virtual environment Poetry is using you can run:

$ poetry show -v
or
$ poetry config --list

Bundle and Run your Application for Platform Distribution

For a Python Application, you want to bundle the application and all of its dependencies into a single package so that it can easily be installed on a users platform without the user manually install Python and other modules.

Briefcase allows you to package and run your app using your platform:

(Windows) $ poetry run python setup.py windows -s
(macOS)   $ poetry run python setup.py macos -s
(Linux)   $ poetry run python setup.py linux -s

Your app will launch, will just be a blank window at this point.

Also notice that it creates a folder with the platform name that you used above. Inside this folder, Briefcase has packaged your app for distribution on your platform. Briefcase also has distribution options for android, ios, and django.

Build your Library for Distribution on PyPI

$ poetry build

Building dantestapp (0.1.0)
 - Building sdist
 - Built dantestapp-0.1.0.tar.gz

 - Building wheel
 - Built dantestapp-0.1.0-py3-none-any.whl

The source distribution (sdist) and wheel are now in a new dist folder.

Publish your Library to PyPI

First we are going to add the TestPyPI repository to Poetry, so that it knows where to publish to. The default location is to the real PyPI.

$ poetry config repositories.test-pypi https://test.pypi.org/legacy/

Now simply run:

$ poetry publish -r test-pypi

The -r argument tells Poetry to use the repository that we configured. Poetry then will ask for your username and password. Congrats! Your package is now available to be viewed at https://test.pypi.org/project/your-project-name/ and can be pip installed with pip install -i https://test.pypi.org/simple/ your-project-name.