Docker Compose Dependency Management Isn’t a Silver Bullet

Lately, my goal has been to streamline project setup processes in their READMEs, particularly when dockerizing a web app. The setup steps boil down to a simple 1) git clone {project} and 2) docker compose up. Ideally, developers now have a production-like version of the app up and running on a local port, ready for exploration.

However, usually, each project has its own quirks, and I had to do something unusual to make this setup work the way I’d expected. Here, I’ll delve into a peculiar database connection error I’ve encountered while working on a dockerized Python, Django, MariaDB web app.

The Error

With the web app all dockerized, I commit my progress and delete the local Git repository. Pruning the Docker networks, volumes, and caches follows suit.

Now, I start on the new codebase journey any fellow developer would — following the two-step setup outlined in the README.

Step one: git clone {project}. All good.

Step two: docker compose up. Womp, womp…

app-web-1    |   File "/usr/local/lib/python3.10/site-packages/MySQLdb/__init__.py", line 123, in Connect
app-web-1    |     return Connection(*args, **kwargs)
app-web-1    |   File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 185, in __init__
app-web-1    |     super().__init__(*args, **kwargs2)
app-web-1    | django.db.utils.OperationalError: (2002, "Can't connect to MySQL server on 'db' (115)")
app-web-1 exited with code 1

Upon running a fresh docker-compose up, I encountered this error consistently.

Can't connect to MySQL server on 'db' (115)

Oddly enough, on subsequent runs, the startup commands behaved as expected without any errors.

It soon becomes clear that the Django web service is trying to connect to the MariaDB db service before it’s actually ready to accept connections, despite the dependency management in Docker Compose, particularly the project’s docker-compose.yml file.

Project files

Here are three example files to understand the simple Docker architecture of this project.

1. docker-compose.yml

   
   version: "3.8"
   services:
     db:
       image: mariadb:10-focal
       restart: always
       volumes:
         - db-data:/var/lib/mysql
       environment:
         - MARIADB_DATABASE=app
         - MARIADB_ROOT_PASSWORD=password
       ports:
         - "3306:3306"
     web:
       build:
         context: .
         dockerfile: Dockerfile
       volumes:
         - .:/app
       ports:
         - "8080:80"
       command: python manage.py runserver
       depends_on:
         - db  # note that the Django web service is dependent on the MariaDB db service
       environment:
         - DATABASE_NAME=app
         - DATABASE_USER=app
         - DATABASE_PASSWORD=password
         - DATABASE_HOST=db
         - DATABASE_PORT=3306
   volumes:
     db-data:
   

2. Dockerfile

   
   FROM python:3.10-slim-bullseye
   
   # install dependencies...
   
   WORKDIR /app
   
   # other build steps...
   
   COPY entrypoint.sh /entrypoint.sh
   RUN chmod +x /entrypoint.sh
   
   ENTRYPOINT ["/entrypoint.sh"]
   
   CMD ./start_server.sh
   

3. entrypoint.sh

   
   #!/bin/bash
   
   # do not proceed if any command fails
   set -e
   
   # run migrations
   python manage.py migrate
   
   # run docker command
   exec "$@"
   

The solution

After researching the problem, incorporating some kind of wait-for-it.sh or a docker condition configuration is needed. This ensures the database is ready to accept connections before other services try to connect to it.

The method I chose to use is in line with “wait-for-it” and includes a new feature I learned about in bash. Specifically, that’s using redirections on the special /dev/tcp/host/port file. You can read more about this in the bash manual. By redirecting data to the /dev/tcp/host/port file, bash will either succeed in opening a TCP connection to the server or error out.

The gist of implementing this solution is to infinitely loop until echo > /dev/tcp/$DATABASE_HOST/$DATABASE_PORT exits 0 (no error). And wrap a timeout command around it so the loop doesn’t stall waiting on a response forever.

Knowing this, I modified the entrypoint.sh to block until a TCP connection could be formed with the db service successfully before proceeding with the web service startup procedure.

entrypoint.sh

#!/bin/bash

# do not proceed if any command fails
set -e

echo "waiting for the database..."
while ! timeout 1 bash -c "echo > /dev/tcp/$DATABASE_HOST/$DATABASE_PORT"; do
  sleep 0.37 # block for an arbitrary amount of time before checking again
done
echo "database is ready."

# run migrations
python manage.py migrate

# run docker command
exec "$@"

This solved the problem. No more database connection errors when running docker-compose up for the first time and any time after that!

In essence, the dependency management in Docker Compose, particularly with the depends_on attribute, isn’t a silver bullet that ensures readiness of services for their dependent counterparts.