Python API Development in Docker

Intention

In this article I will describe (in great detail) how I have configured a local dev environment to start up an application using Docker Compose and Docker containers. The result is a development environment that starts up quickly, is portable, and very closely resembles the final production environment.

Overview

Here are the steps you'll learn about in this post:

Create a specific Dockerfile that resembles the production dockerfile as closely as possible.
Create a docker-compose.yml file that builds the Dev Dockerfile. The docker-compose.yml contains default values for environment variables and other configurations.
Create a set_env_variables.sh shell script that can set environment variables with defaults and picks up values from a .env-dev file.
Create a reset_env_variables.sh shell script that can unset all environment variables and then sources the set_env_variables.sh shell script.
Create a start.sh shell script that makes configuring and starting the app a single simple call.
Create a stop.sh shell script that makes cleanly stopping the app a single simple call.
Leverage the "Dev Containers" VS Code extension by Microsoft for developing inside the container itself.
A full working example may be found at: https://github.com/generalui/python-flask-ariadne-api-starter.

Introduction

I often work on projects that must be handed off to other developers once created. Having been on the receiving end of this, I know that it can be a huge time-suck to try to get a project started locally with all the correct dependencies. A new developer on a project needs to get their dev environment set up quickly so they can get to the real work of build features and fixing bugs. A well designed dev environment can be the difference between new developers delivering value on day one versus getting stuck in setup for weeks.

Additionally, when I am working on local dev, I want my dev environment to match the production environment as closely as possible. How many times has code worked in local only to fail in production because the environment is slightly different?

I recently had the great opportunity to create a GraphQL API in Python. I had experience in GraphQl, but none in Python. On that note please forgive anything I've done that doesn't seem "Pythonic"! I am still a learner of course and always striving to improve.

The type of API is irrelevant in the scope of this article.

This app is using a PostgreSQL database but could be changed out to a different database. Changing out the database is outside of the scope of this article.

The code base for this article may be found at: https://github.com/generalui/python-flask-ariadne-api-starter. The repo may be used as a starting point for your own Python GraphQL API.

The Setup of the Python app itself is also outside of the scope of this article. Using the example app, we know that the app has a script or command to start the server. We will address how this script or command gets initialized later in the article.

Dockerfile †

The app has two Dockerfiles. One for deployment and one for development. I wanted the development file to resemble the production file as much as possible. the reason for two files is that in development, I have a number of dependencies (for testing, linting, profiling, etc) that I didn't need or want in production. I also have a number of apps that I wanted available in the container for dev that I didn't need or want in production.

Production Dockerfile

The production Dockerfile can be seen here.

Development Dockerfile

The development Dockerfile is similar to the production Dockerfile but with some additional apps and requirements installed just for dev. Again, I am trying to maintain parity between what is deployed in a release and what is run on the local dev machine.

Key similarities:

Python version
Running on Alpine
Using the same code base
Installing the same build tools for installing dependencies
Building with the same requirements.txt file
Removing the build tools after installing dependencies

Dockerfile-dev

Click to view file contents

Dockerfile
# Make the Python version into a variable so that it may be updated easily if / when needed. (ie "3.10")
ARG pythonVersion

# Using a python image itself so the app may be updated easily if / when needed. (ie "3.10")
# Start with a bare Alpine Linux to keep the container image small.
FROM python:${pythonVersion}-alpine

# Designate the `/app` folder inside the container as the working directory.
WORKDIR /app

# Copy the requirements (both prod and dev) files to the `/app` folder inside the container.
# Do this in a separate "COPY" so that the the image will update if either of these files change.
COPY ./requirements.txt ./requirements-dev.txt /app/
# Copy the code base to the work directory. This will ensure it is added to the volume.
COPY ./ /app/

# Execute everything under a single "RUN" to reduce the layer count.
# Upgrade pip
RUN pip install --upgrade pip && \
    # `libpq` is needed for Postgres commands.
    apk add --no-cache libpq \
    # These useful tools are only installed in the development environment.
    bash curl openssh git nodejs npm && \
    # Install `git-genui` for git commits.
    # This is only installed in the development environment.
    # See https://www.npmjs.com/package/git-genui
    npm install -g git-genui && \
    # Install build tools for installing dependencies.
    apk add --no-cache --virtual .build-deps \
    gcc \
    musl-dev \
    postgresql-dev \
    linux-headers && \
    # Install the PyPI dependencies using pip
    pip install --no-cache-dir -r requirements.txt && \
    # These are only installed in the development environment.
    pip install --no-cache-dir -r requirements-dev.txt && \
    # Remove the build tools now that we are done with them.
    apk del --no-cache .build-deps

# Inside the container, execute the Python script that starts the server.
# Only if `NO_AUTO_START` is NOT set.
# Otherwise, tail nothing so a process will continue and the container will run.
CMD ["bash", "-c", "if [ -z ${NO_AUTO_START} ]; then python /app/run.py; else tail -f /dev/null; fi"]

Of course the Docker files could be run as is with Docker cli. This would be just a bit ugly and complicated. I'll use docker-compose instead.

Docker Compose ††

Using a docker compose file to start and stop the docker container allows much more configurability for my environment and app.

docker-compose.yml

Click to view file contents

version: "3.8"

services:
    api:
    env_file: ${DOT_ENV_FILE:-.env-none}
    # Ensure specific environment variables are ALWAYS available.
    environment:
        - APP_NAME=${APP_NAME:-"Python Flask Ariadne API Starter Test"}
        - FLASK_APP=${FLASK_APP:-app.py}
        - FLASK_DEBUG_MODE=${FLASK_DEBUG_MODE:-false}
        - FLASK_ENV=${FLASK_ENV:-development}
        - FLASK_RUN_PORT=${FLASK_RUN_PORT:-5000}
        - LOG_TYPE=${LOG_TYPE:-}
        - NO_AUTO_START=${NO_AUTO_START:-}
        - POSTGRES_DB=${POSTGRES_DB:-pfaas_dev}
        - POSTGRES_DB_TEST=${POSTGRES_DB:-pfaas_test}
        - POSTGRES_HOST=${POSTGRES_HOST:-host.docker.internal}
        - POSTGRES_HOST_TEST=${POSTGRES_HOST_TEST:-host.docker.internal}
        - POSTGRES_PORT=${POSTGRES_PORT:-5432}
        - POSTGRES_PORT_TEST=${POSTGRES_PORT_TEST:-5432}
        - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-docker}
        - POSTGRES_PASSWORD_TEST=${POSTGRES_PASSWORD_TEST:-docker}
        - POSTGRES_USER=${POSTGRES_USER:-postgres}
        - POSTGRES_USER_TEST=${POSTGRES_USER_TEST:-postgres}
        - PYTHONUNBUFFERED=1
        - PYTHON_VERSION=${PYTHON_VERSION:-3.10}
        - SECRET_KEY=${SECRET_KEY:-some_real_good_secret}
        - SNAKEVIZ_PORT=${SNAKEVIZ_PORT:-8020}
        - SSL_ENABLED=${SSL_ENABLED:-}
    build:
        context: ./
        dockerfile: Dockerfile-dev
        args:
        pythonVersion: ${PYTHON_VERSION:-3.10}
    container_name: python-flask-ariadne-api-starter
    image: python-flask-ariadne-api-starter:dev
    command:
        - "sh"
        - "-c"
        - "if [ ${NO_AUTO_START:-} ]; then tail -f /dev/null; else python /app/run.py; fi"
    ports:
        - ${FLASK_RUN_PORT:-5000}:${FLASK_RUN_PORT:-5000}
        - ${SNAKEVIZ_PORT:-8020}:${SNAKEVIZ_PORT:-8020}
    volumes:
        - .:/app:delegated
        - ~/.gitconfig:/root/.gitconfig:delegated
        - ~/.ssh:/root/.ssh:delegated
        - python-flask-ariadne-api-starter-dev-root-vol:/root:delegated
    logging:
        options:
        max-size: "10m"
        max-file: "3"
volumes:
    python-flask-ariadne-api-starter-dev-root-vol:

The docker compose file is counting on a number of environment variables being available. Defaults have been defined in the event the variables haven't been set.

Initially, the DOT_ENV_FILE denotes the name of a .env file to pick up environment variables from. If this environment variable hasn't been set, then the default .env-none file is use which intentionally has no environment variables set in it.

To override defaults, create a .env-dev file (modeled after the included .env-SAMPLE file) and set the desired override values. The path to this file MUST be set to the DOT_ENV_FILE variable. I have created a script that ensures this, but more on that later.

Some of the defaults in the environment option are set for this app. Values for POSTGRES_DB and POSTGRES_DB_TEST, the database names for dev and test respectively, are set for this app. You would change these defaults to values appropriate for your situation. The database host values are set by default to host.docker.internal. This allows the container to connect to localhost of the local machine and not localhost of the container. Note that this doesn't work in Linux. For linux to connect to the host machine's localhost, use 172.17.0.1 instead.

In the build option, I tell docker compose where to find the docker file and what its name is. I also pass the python version (defaults to 3.10).

I've added a container name and image name to help easily identify them on the local machine. I have named them python-flask-ariadne-api-starter for this app, but they could be named whatever is convenient. The image version is tagged simply dev as this image will be overwritten with changes.

The command option is executed in the container once the container is built. I have created an optional NO_AUTO_START variable that will be set (or not) in the container. If it is set to a truthy value, the container will NOT automatically start the server. This may be useful for starting the container and then entering the container to do all dev inside. The server may then be started inside the container and played with exclusively in the container context. More on this later. If NO_AUTO_START is set to a truthy value, then I execute tail -f /dev/null. This tails nothing but provides a process to run so that the container will keep running.

The ports option exposes ports inside the container to the outside machine. The FLASK_RUN_PORT is the port that the server will be running on. As localhost inside the container can't be accessed by the local browser, the port is exposed. This is also true for the SNAKEVIZ_PORT port, the port that the Snakeviz ††† profiling info page is rendered on.

The volumes option is very important in the development environment. We create a number of volumes here. All of them are delegated †††† for better performance. This is geared towards developing INSIDE the container. I create these 4 volumes:

I map the local project directory to the working directory inside the container. Thus, changes made in the local are reflected inside the container and vice versa.
I map the local ~/.gitconfig file (in the user's home folder) to to the root user's home folder in the container. This supports git commands in the container as though you are operating on your local command line.
I map the local ~/.ssh folder (in the user's home folder) to to the root user's home folder in the container. This makes the local ssh keys available inside the container for git fetches, pulls and pushes.
I map the entire root user's home folder in the container to a volume. We named the volume python-flask-ariadne-api-starter-dev-root-vol for this app to differentiate it from other volumes on the local, but you could name it whatever is appropriate for your situation. Please note that the volume itself is defined at the bottom of the docker-compose.yml file under volumes.

This volume persists any other files that are created inside the root user's home folder inside the container. Files like .bashrc and .profile can be very useful inside the container. Additionally, if VS Code is being used and the "Dev Containers" ††††† extension is being used, the VS Code server and installed extensions are stored there. These will persist between starts and stops and helps make opening a lot faster.
The logging option limits the size of logs within the container. This ultimately helps with performance. If logs get too large, the container gets huge and can really slow down.

The docker-compose.yml file may be run on it's own to start the docker container, but I have created some convenience scripts that put everything in one place and simplify the process.

Start

There is a magic start.sh script in the root of the project. Starting the app in dev is as simple as running

bash ./start.sh

This will help set environment variables such as DOT_ENV_FILE by identifying if a .env-dev file exists or not. The script accepts a few flags:

-b or --build - Pass this flag if you explicitly want to rebuild the image.
-r or --reset_env - Pass this flag to reset environment variables back to defaults. Sometimes, I play around with environment variables and the values can get all crazy. This will reset them with one caveat in mind, it will read the values set in the .env-dev file. Clearing out this file will and passing this flag will completely reset the environment variables to defaults.
-n or --no_auto_start - Pass this flag to start the container without starting the server. This may be useful for starting the container and then entering the container to do all dev inside. The server may then be started inside the container and played with exclusively in the container context. More on this later.

start.sh

Click to view file contents

#!/bin/bash

# Defined some useful colors for echo outputs.
# Use BLUE for informational.
BLUE="\033[1;34m"
# Use Green for a successful action.
GREEN="\033[0;32m"
# Use YELLOW for warning informational and initiating actions.
YELLOW="\033[1;33m"
# Use RED for error informational and extreme actions.
RED="\033[1;31m"
# No Color (used to stop or reset a color).
NC='\033[0m'

# By default, set these variables to false.
build=false
no_auto_start=false
reset=false

# Checks if a specific param has been passed to the script.
has_param() {
    local term="$1"
    shift
    for arg; do
        if [[ $arg == "$term" ]]; then
            return 0
        fi
    done
    return 1
}

# If the `-b or --build` flag is passed, set build to true.
if has_param '-b' "$@" || has_param '--build' "$@"
then
    >&2 echo -e "${BLUE}Build requested${NC}"
    build=true
fi

# If the `-n or --no_auto_start` flag is passed, set no_auto_start to true.
if has_param '-n' "$@" || has_param '--no_auto_start' "$@"
then
    >&2 echo -e "${BLUE}No auto start requested${NC}"
    no_auto_start=true
fi

# If the `-r or --reset_env` flag is passed, set reset to true.
if has_param '-r' "$@" || has_param '--reset_env' "$@"
then
    >&2 echo -e "${BLUE}Reset environment variables requested${NC}"
    reset=true
fi

if [ "${reset}" = true ]
then
    # Reset the environment variables.
    source ./reset_env_variables.sh
else
    # Set the environment variables.
    source ./set_env_variables.sh
fi

if [ "${no_auto_start}" = true ]
then
    # Ensure the NO_AUTO_START environment variable is set to true.
    export NO_AUTO_START=true
fi

docker system prune --force

if [ "${build}" = true ]
then
    # Build and start the container.
    docker-compose up -d --build
else
    # Start the container.
    docker-compose up -d
fi

# Only execute if the `NO_AUTO_START` variable has NOT been set.
if [ -z ${NO_AUTO_START} ]
then
    # If CTRL+C is pressed, ensure the progress background PID is stopped too.
    function ctrl_c()
    {
        >&2 echo -e "${RED} => CTRL+C received, exiting${NC}"
        # Stop the progress indicator.
        kill $progress_pid
        wait $progress_pid 2>/dev/null
        # Cursor visible again.
        tput cnorm
        exit
    }

    function open_url()
    {
        [[ -x $BROWSER ]] && exec "$BROWSER" "$url"
        path=$(which xdg-open || which gnome-open || which open || which start) && exec "$path" "$url"
        >&2 echo -e "${YELLOW}Can't find the browser.${NC}"
    }

    # Creates a animated progress (a cursor growing taller and shorter)
    function progress() {
        # Make sure to use non-unicode character type locale. (That way it works for any locale as long as the font supports the characters).
        local LC_CTYPE=C
        local char="▁▂▃▄▅▆▇█▇▆▅▄▃▂▁"
        local charwidth=3
        local i=0
        # Cursor invisible
        tput civis
        while sleep 0.1; do
            i=$(((i + $charwidth) % $
                  How can we help?
                  
                    Can we help you apply these ideas on your project? Send us a message! You'll get
                    to talk with our awesome delivery team on your very first call.

Address

Phone

Email

Social

Python API Development in Docker

Jon Ryser

Intention

Overview

Introduction

Dockerfile †

Production Dockerfile

Development Dockerfile

Docker Compose ††

Start

How can we help?