How to Rock Python Packaging with Poetry and Briefcase

As part of modernizing Gaphas, the diagramming widget for Python, I took another look at what the best practices are for packaging and releasing a new version of a Python library or application. There are new configuration formats and tools to make packaging and distributing your Python code much easier.

A Short Background on Packaging

There are two main use cases for packaging:

  1. Packaging a Library - software that other programs will make use of.
  2. Packaging an Application - software that a user will make use of.

This may not be a completely accurate definition because software does not always fit cleanly in to one of these bins, but these use cases will help to keep focus on what exactly we are trying achieve with the packaging.

The Library

The goal for packaging a library is to place it on the Python Packaging Index (PyPI), so other projects can pip install it. In order to distribute a library, the standard format is the Wheel. It allows for providing a built

distribution of files and metadata so that pip only needs to extract files out of the distribution and move them to the correct location on the target system for the package to be installed. In other words, nothing needs to be built and re-compiled.

Previously if you wanted to achieve this, it was common to have four configuration files:

  1. setup.py - The setup script for building, distributing and installing modules using the Distutils.
  2. requirements.txt - Allow easy install of requirements using pip install -r
  3. setup.cfg - The setup configuration file
  4. MANIFEST.in - The manifest template, directs sdist how to generate a manifest

The Application

The goal for packaging an application is get it in the formats where you can distribute it on the different platforms for easy installation by your users. For Windows this is often an exe or msi. For macOS this is an app. For Linux this is a deb, flatpak, appimage, or snap. There is a whole host of tools to do this like: py2exe, py2app, cx_Freeze, PyInstaller, and rumps.

pyproject.toml

On the packaging front, in May of 2016, PEP 518 was created. The PEP does a good job of describing all of the shortcoming of the setup script method to specify build requirements. The PEP also specified a new configuration format call pyproject.toml. If you aren't familiar with TOML, it is human-usable and is more simple than YAML.

The pyproject.toml replaced those four configuration files above using two main sections:

  1. [build-system] - The build-system table contains the minimum requirements for the build system to execute.
  2. [tool] - The tool table is where different tools can have users specify configuration data.

The Tools

Making use of this new configuration format, a tool called flit has been around since 2015 as a simple way to put Python Libraries on PyPI.

In 2017, Pipenv was created to solve pain points about managing virtualenvs and dependencies for Python Applications by using a new Pipfile to manage dependencies. The other major enhancement was the use of a lock file. While a Wheel is the important output for a Library, for an Application, the lock file becomes the important thing created for the project. The lock file contains the exact version of every dependency so that it can be repeatably rebuilt.

In 2018, a new project called Poetry combined some of the ideas from flit and Pipenv to create a new tool that aims to further simplify and improve packaging. Like flit, Poetry makes use of the pyproject.toml to manage configuration all in one place. Like Pipenv, Poetry uses a lock file (poetry.lock) and will automatically create a virtualenv if one does not already exist. It also has other advantages like exhaustive dependency resolution that we will explore more thoroughly below.

For Application distribution, I am going to focus on a single tool called Briefcase which along with the other set of BeeWare tools and libraries allows for you to distribute your program as a native application to Windows, Linux, macOS, iOS, Android, and the web.

Tutorial

With the background information out of the way, lets work through how you can create a new Python project from scratch, and then package and distribute it.

Initial Tool Installation

To do that, I am going to introduce one more tool (the last one I promise!) called cookiecutter. Cookiecutter provides Python project templates, so that you can quickly get up to speed creating a project that can be packaged and distributed without creating a bunch of files and boilerplate manually.

To install cookiecutter, depending on your setup and operating system, from a virtualenv you can run:

$ pip install cookiecutter

Next we are going to install Poetry. The recommended way is to run:

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

TestPyPI Account Sign-Up

As part of this tutorial we will be publishing packages. If you don't already have an account, please register for an account on TestPyPI. TestPyPI allows you to try distribution tools and processes without affecting the real PyPI.

Create Your Project

To create the Python project, we are going to use the Briefcase template, so run cookiecutter on this template:

$ cookiecutter https://github.com/pybee/briefcase-template

Cookiecutter will ask you for information about the project like the name, description, and software licence. Once this is finished, add any additional code to your project, or just keep it as is for this demo.

Change your directory to the app name you gave (I called mine dantestapp), and initialize git:

$ cd dantestapp
$ git init
$ git add .

Create a pyproject.toml Configuration

Poetry comes equipped to create a pyproject.toml file for your project, which makes it easy to add it to an existing or new project. To initiliaze the configuration run:

$ poetry init

The command guides you through creating your pyproject.toml config. It automatically pulls in the configuration values from the briefcase-template that we created earlier so using the default values by hitting enter after the first six questions will be fine. This is what it provided for an output:

Package name [dantestapp]: 
Version [0.1.0]: 
Description []: 
Author [Dan Yeaw <dan@yeaw.me>, n to skip]: 
License []: MIT
Compatible Python versions [^3.7]: 
Define Dependencies

The configuration generator then asks for you to define your dependencies:

Would you like to define your dependencies (require) interactively? (yes/no) [yes]

Hit enter for yes.

For the next prompt Search for package: enter in briefcase. We are setting briefcase as a dependency for our project to run.

Enter package # to add, or the complete package name if it is not listed: 
 [0] briefcase
 [1] django-briefcase

Type 0 to select the first option. and hit enter to select the latest version. You now need to repeat this process to also add Toga as a dependency. Toga is the native cross-platform GUI toolkit. Once you are done, hit enter again to complete searching for other dependencies.

Define Development Dependencies

At the next prompt the config generator is now asking us to define our development dependencies:

Would you like to define your dev dependencies (require-dev) interactively (yes/no) [yes]

Hit enter to select the default value which is yes.

We are going to make pytest a development dependency for the project.

At the prompt Search for package: enter in pytest.

Found 100 packages matching pytest

Enter package # to add, or the complete package name if it is not listed: 
 [ 0] pytest

You will get a long list of pytest packages. Type 0 to select the first option. and hit enter to select the latest version. Then hit enter again to complete searching for other development dependencies.

Complete the Configuration

The final step of the configuration generator summaries the configuration that it created. Notice that first three sections are tool tables for Poetry, and the final one is the build-system table.

[tool.poetry]
name = "dantestapp"
version = "0.1.0"
description = ""
authors = ["Dan Yeaw <dan@yeaw.me>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.7"
briefcase = "^0.2.8"
toga = "^0.2.15"

[tool.poetry.dev-dependencies]
pytest = "^4.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

The dependencies use a "caret requirement", like python = "^3.7". This makes use of semantic versioning. So in this example if Python 3.8 is released, then it will automatically update to this version. But, it won't update to 4.0 automatically, since that is a major version change. If we put in our configuration "^3.7.2", then it would automatically update to 3.7.3 which it is released, but not 3.8, since that is a new minor version.

There are also "tilde requirements" that are more restrictive. So if you enter `python = "~3.7" it will only allow update to the next patch level, like from 3.7.2 to 3.7.3. The combination of caret and tilde requirements allows you to get updates to your dependencies when they are released, but puts you in control to ensure that incompatible changes won't break your app. Nice!

The final prompt asks: Do you confim generation? (yes/no) [yes]. Go ahead and hit enter to confirm. Congrats, you have generated a pyproject.toml configuration!

Install Dependencies

OK, the hard work is over, we have created our project and finished the configuration. Now it is time to see how Poetry and Briefcase really shines.

To install the dependencies that you defined in the pyproject.toml, just run:

$ poetry install

Poetry includes an exhaustive dependency resolver, so it will now resolve all of the dependencies it needs to install Briefcase, Toga, and pytest. It will also create a poetry.lock file which ensures that anyone using your program would get the exact same set of dependencies that you used and tested with.

Notice that we also did not create or specify a virtual environment. Poetry automatically creates one prior to installing packages, if one isn't already activated. If you would like to see which packages are installed and which virtual environment Poetry is using you can run:

$ poetry show -v
or
$ poetry config --list

Bundle and Run your Application for Platform Distribution

For a Python Application, you want to bundle the application and all of its dependencies into a single package so that it can easily be installed on a users platform without the user manually install Python and other modules.

Briefcase allows you to package and run your app using your platform:

(Windows) $ poetry run python setup.py windows -s
(macOS)   $ poetry run python setup.py macos -s
(Linux)   $ poetry run python setup.py linux -s

Your app will launch, will just be a blank window at this point.

Also notice that it creates a folder with the platform name that you used above. Inside this folder, Briefcase has packaged your app for distribution on your platform. Briefcase also has distribution options for android, ios, and django.

Build your Library for Distribution on PyPI

$ poetry build

Building dantestapp (0.1.0)
 - Building sdist
 - Built dantestapp-0.1.0.tar.gz

 - Building wheel
 - Built dantestapp-0.1.0-py3-none-any.whl

The source distribution (sdist) and wheel are now in a new dist folder.

Publish your Library to PyPI

First we are going to add the TestPyPI repository to Poetry, so that it knows where to publish to. The default location is to the real PyPI.

$ poetry config repositories.test-pypi https://test.pypi.org/legacy/

Now simply run:

$ poetry publish -r test-pypi

The -r argument tells Poetry to use the repository that we configured. Poetry then will ask for your username and password. Congrats! Your package is now available to be viewed at https://test.pypi.org/project/your-project-name/ and can be pip installed with pip install -i https://test.pypi.org/simple/ your-project-name.

5 Steps to Build Python Native GUI Widgets for BeeWare

Part of my work at Ford Motor Company is to use Model-Based Systems Engineering through languages like SysML to help design safety in to complex automated and electrified technologies. In my free time I took over maintaining a UML tool called Gaphor with the aim of eventually turning it in to a simple SysML tool for beginners. I'm sure I'll be writing about this much more in the future.

Eventually I got really frustrated with the current set of GUI toolkits that are available for Python. I want the ability to write an app once and have it look and feel great on all of my devices, but instead I was dealing with toolkits that are wrapped or introspected around C or C++ libraries, or visually look like a blast from past. They made me feel like I was going against the grain of Python instead of writing great Pythonic code.

If you haven't heard of BeeWare yet, it is a set of software libraries for cross-platform native app development from a single Python codebase and tools to simplify app deployment. When I say cross-platform and native, I mean truly that. The project aims to build, deploy, and run apps for Windows, Linux, macOS, Android, iPhone, and the web. It is native because it is actually that platform's native GUI widgets, not a theme, icon pack, or webpage wrapper.

A little over a year ago, I started to contribute to the BeeWare project. I needed a canvas drawing widget for the app I am working on, I saw that this was not supported by BeeWare, so I went ahead and created it. Based on my experience, this blog post details how I would create a new widget from scratch, now that I have done it before, with the hope that it helps you implement your own widget as well.

If you are new to BeeWare, I recommend to start out with the Briefcase and Toga Tutorials, and then the First-time Contributor's Guide.

BeeWare Logo with Brutus the Bee and text

The current status of the BeeWare project, at the time of writing this, is that it is a solid proof of concept. Creating a simple app on macOS, Linux, or iOS is definitely possible. In fact there is an app called Travel Tips on Apple's App Store that was created by Russell Keith-Magee as a demonstration. Support for some of the other platforms like Windows and Android is lagging behind some, so except some very rough edges.

This alpha status may not be so exciting for you if you are just trying to build an app, but I think it is very exciting for those that want to contribute to an open source project. Although there are many ways to get involved, users keep asking how they can build a GUI widget that isn't yet supported. I think this is a great way to make a significant contribution.

A GUI widget forms the controls and logic that a user interacts with when using a GUI. The BeeWare project uses a GUI widget toolkit called Toga, and below is a view of what some of the widgets look like in Linux.

Example of Toga Widgets in a demo app

There are button, table, tree, and icon widgets in the example. Since I contributed a canvas drawing widget, I will be using that for the example of how you could contribute your own widget to the project.

There are three internal layers that make up every widget:

  1. The Interface layer
  2. The Implementation layer
  3. The Native layer

Software architecture of Toga

The Interface layer provides the public API for the GUI application that you are building. This is the code you will type to build your app using Toga.

The Interface layer calls public methods that are in the Toga_core portion of the project and this is where this Interface layer API is defined. Toga_core also provides any abstract functionality that is independent of the platform that Toga is running on, like setting up and running the app itself.

The Implementation layer connects Toga_core to the Toga_impl component. Toga_impl is the platform backends like Toga_ios, Toga_cocoa, and Toga_gtk. A factory method in Toga_core automatically selects and initializes the correct implementation objects based on the platform it is running on. Toga_dummy is also a type of Toga_impl backend, and it is used for smoke testing without a specific platform to find simple failures.

The Native layer connects the Toga_impl's to the Native Platform. For C language based platforms, Toga_impl directly call the native widgets. For example with Gtk+ on Linux, the Toga_gtk directly calls the Gtk+ widgets through PyGObject. For other platforms, more intermediate work may be required through a bridge or transpiler:

  • macOS and iOS, the Rubicon-ObjC project provides a bridge between Objective-C and Python.
  • Web, Batavia provides a javascript implementation of the Python virtual machine.
  • Android, VOC is a transpiler that converts Python in to Java bytecode.

I know there is a lot there, but understanding the software architecture of Toga together with the surrounding projects and interfaces will be key to implementing your own widget. With that background information out of the way, lets not delay any further, and jump in to building a widget.

Step 0

Pick your development platform

  • Normally pick the platform that you are most familiar with
  • macOS and Gtk+ are the most developed :thumbsup:
  • Is this a mobile only widget (camera, GPS, etc)?

This seems somewhat obvious, since the platform you select will most likely be based on the laptop or other device you are using right now. But do consider this. Most of my experience developing widgets are on Gtk+ and Cocoa so this is where I am coming from. Implementing widgets on other platforms is definitely needed as well, but it may be an additional challenge due to those platforms not as well developed with Toga yet. These other platforms may be more challenging, but they are also the areas where the BeeWare project needs the most help, so if you have some experience with them or feel especially brave, definitely go for it.

Step 1

Research your widget

  • How do you create this widget on different platforms
  • Think, brainstorm, whiteboard, and discuss how you would want to create and manipulate this widget with Python

For example, this is how you would draw a rectangle on a Canvas on different platforms:

  • Tkinter
canvas = Tk.Canvas()
canvas.create_rectangle(10, 10, 100, 100, fill="C80000")
canvas.pack()
  • wxpython
wx.Panel.Bind(wx.EVT_PAINT, OnPaint)
def OnPaint(self, evt):
    dc = wx.PaintDC()
    dc.SetBrush(wx.Brush(wx.Colour(200, 0, 0)))
    dc.DrawRectangle(10, 10, 100, 100)
  • HTML canvas
var c = document.getElementById("myCanvas");
var ctx = c.getContext("2d");
ctx.fillStyle = "rgb(200, 0, 0)";
ctx.fillRect(10, 10, 100, 100);
  • Gtk+
drawingarea = Gtk.DrawingArea()
drawingarea.connect("draw", draw)
def draw(da, ctx):
    ctx.set_source_rgb(200, 0, 0)
    ctx.rectangle(10, 10, 100, 100)
    ctx.fill()

Step 2

Create the Pythonic API for your interface layer

  • Write your API documentation first
  • The API provides the set of clearly defined methods of communication (layers) between the software components
  • Documentation Driven Development
  • This is iterative with Step 1

Start with explaining what your widget is and what it is used for. Think about the use cases of the widget. What types of apps are users going to build with this widget? These are the scenarios that your widget needs to be able to support. How would a user want to use your widget? When looking at the Canvas widgets from my research, I noticed that the current drawing widgets were very procedural, you have to create your canvas drawing using many steps. For example, you have to first set the color to draw with, then draw an object, and then fill in that object.

Python has the context manager and the "with" statement, and making use of this for a canvas allows the user to better break up the draw operations with some nesting. It also allows for automatically starting or closing drawing of a closed path for the user. This is an example of the types of things that you can take advantage of in an API that was designed for Python. It is easy to try to copy the API that you are familiar with, but I think you can really make your widget stand out by taking a step back and looking at how you can make an API that users will really enjoy using.

Here is an example of writing the initial explanation and widget API for the canvas widget:

The canvas is used for creating a blank widget that you can
draw on.

Usage
--

Simple usage to draw a colored rectangle on the screen using
the arc drawing object:

import toga
canvas = toga.Canvas(style=Pack(flex=1))
with canvas.fill(color=rgb(200, 0, 0)) as fill:
    fill.rect(10, 10, 100, 100)

Once that is complete, now might be a good time to ask for feedback to see if you have missed any use cases or if others have any ideas of how to improve the public API of the widget. One way to collect feedback would be to submit an issue or a "work in progress" pull request to the Toga project, or ask for feedback on the Gitter channel.

Next, start to work out the structure of your Toga_core code based on your API. I recommend creating the class and method definitions and add the docstrings to outline what each portion of the software does and what arguments and return values it provides. This is part of the overall documentation that will be generated by Sphinx for your widget, and creating this before writing your code will provide the next level of API documentation.

Here is an example of how that structure and docstrings would look for a canvas widget:

class Canvas(Context, Widget):
    """Create new canvas.

    Args:
        id (str):  An identifier for this widget.
        style (:obj:`Style`): An optional style object. 
        factory (:obj:`module`): A python module that is
            capable to return a implementation of this class.

     """
def rect(self, x, y, width, height):
    """Constructs and returns a :class:`Rect <Rect>`.

    Args:
        x (float): x coordinate for the rectangle.
        ...
    """

Step 3

Implement your Toga_core widget using TDD

  • Write a test for each function of the widget outlined in the API from Step 3
  • Check that the tests fail
  • Specify the implementation layer API
  • Write the core code for the widget to call the implementation layer

Test Driven Development is a test-first technique to write your tests prior to or in parallel with writing the software. I am being opinionated here, because you don't have to write your code using this process. But, I think this will really help you think about what you want from the code as you implement these different API layers.

Toga_core has a "tests" folder, and this is where you need to create your tests for the widget. Sometimes it can be challenging to know what tests to write, but in the previous step you already outlined what the use cases and scenarios are for using your widget, and the API to make use of the widget. Break this up in to atomic tests to check that you can successfully create the widget, and then make use of and modify the widget using all of the outlined scenarios.

Here is a test to check that the widget is created. The canvas._impl.interface is testing the call to the Toga_impl component ("_impl") and then back to the Toga_core component ("interface"). In other words we are testing that the canvas object is the same object as we go to the Implementation layer to the Toga_impl and then back across the Implementation layer to the Toga_core. The object should be equal as long as it was created successfully. The second line of the test assertActionPerformed is using the dummy backend to test that the canvas was created, and I'll discuss that more in Step 4 below.

def test_widget_created():
    assertEqual(canvas._impl.interface, canvas)
    self.assertActionPerformed(canvas, "create Canvas")
````
Further along in my test creation I also wanted to check that the user could modify
a widget that was already created. So I created a test that modifies the coordinates
and size of a rectangle. 
```Python
def test_rect_modify():
    rect = canvas.rect(-5, 5, 10, 15)
    rect.x = 5
    rect.y = -5
    rect.width = 0.5
    rect.height = -0.5
    canvas.redraw()
    self.assertActionPerformedWith(
            canvas, "rect", x=5, y=-5, width=0.5, height=-0.5
        )

Once you are done creating your tests and make sure that they are failing as expected, it is time to move on to filling in all of those Toga_core classes and objects that you left blank in the previous step.

Toga provides a base Widget class that all widgets derive from. It defines the interface for core functionality for children, styling, layout and ownership by specific App and Window. Below our class Canvas is derived from Widget and is initialized:

class Canvas(Widget):
    def __init__(self, id=None, style=None, factory=None):
        super().__init__(id=id, style=style, factory=factory)

As part of the class initialization, Toga also uses the factory method to determine the correct Toga_impl platform, and then connect it from the Toga_core to Toga_implself._impl and back the other way using interface=self:

        # Create a platform specific implementation of Canvas
        self._impl = self.factory.Canvas(interface=self)

Finally, we fill in our methods to call the creation of the rectangle on the Toga_impl component using the Implementation layer:

    def rect(self, x, y, width, height):
        self._impl.rect(
            self.x, self.y, self.width, self.height
        )

Step 4

Implement the Toga_impl widget on the dummy backend

  • Dummy is for automatic testing without a native platform
  • Code the implementation layer API endpoint, create a method for each call of the API
  • Check that all tests now pass

When your widget is integrated with Toga, we want unit tests to run with the test suite automatically during continuous integration. It may be difficult during these tests to start up every platform and check that your widget is working correctly, so there is a Toga_impl called dummy that doesn't require a platform at all. This allows for smoke testing to make sure that the widget correctly calling the Implementation layer API.

Now go ahead and implement the Toga_impl widget on the dummy backend. There needs to be methods for each call from the Toga_core to the Toga_impl. Below we check that the Canvas create and rect method actions were invoked through Implementation layer API calls.

class Canvas(Widget):
    def create(self):
        self._action("create Canvas")

    def rect(self, x, y, width, height):
        self._action(
            "rect", x=x, y=y, width=width, height=height
        )

You now should be able to run and pass all the tests that you created in Step 3.

Step 5

Implement the Toga_impl widget on your platform backend

  • Copy toga_dummy and create a new endpoint for the platform you chose in Step 1
  • Make use of the native interface API for this widget on your platform

If after your research in Step 1, you aren't feeling confident in how the widget should work on your platform, now would be a good time to take a break to go practice. Build a simple canvas drawing app for your platform using the native widgets. Once you have done that, now is the time to create the Toga_impl for your platform that calls those native widgets on your platform.

In my example, Gtk+ uses an event callback to do native drawing. So I create a Gtk.DrawingArea to draw on when my Canvas widget is created, and then I connect that drawing callback to the gtk_draw_callback function which then calls a method in Toga_core through the Implementation layer:

class Canvas(Widget):
    def create(self):
        self.native = Gtk.DrawingArea()
        self.native.interface = self.interface
        self.native.connect("draw", self.gtk_draw_callback)

    def gtk_draw_callback(self, canvas, gtk_context):
        self.interface._draw(self, draw_context=gtk_context)

Some platforms like Android or Cocoa will require transpiling or bridging to the native platform calls since those platforms using a different programming language. This may require the creation of extra native objects to, for example, reserve memory on those platforms. Here is an example of what this extra TogaCanvas class would like with the Cocoa platform:

class TogaCanvas(NSView):
    @objc_method
    def drawRect_(self, rect: NSRect) -> None:
        context = NSGraphicsContext.currentContext.graphicsPort()

Finally create each method for your native implementation. Below we create an implementation of the rectangle creation that calls Gtk+'s cairo drawing:

    def rect(self, x, y, width, height, draw_context):
        draw_context.rectangle(x, y, width, height)

After you most likely do some troubleshooting of your widget to get it to work properly with your platform, you now should have a complete widget!

Toga Tutorial 4 for a Canvas Widget

Tada! You did it, Submit a PR!

I would be interested in how it goes for you, drop me a line with your experience creating a widget.

2018-11-10: minor editorial updates.