Don’t Manage Your Python Environments, Just Use Docker Containers

Image by Author | Midjourney

Introduction

 
We all know it: managing Python environments can become cumbersome. Regardless of your experience level, environment management can sometimes give you that awful feeling in the pit of your stomach.

Tools like virtualenv or conda work well for isolated environments, but they can still lead to dependency headaches, version conflicts, and challenges in sharing reproducible setups. A more robust solution could be to utilize Docker containers — lightweight, isolated runtime environments — so you can keep your Python dependencies tidily packaged per project.

When you create a Docker container, you essentially snapshot an operating system and install everything you need (Python interpreter, libraries, system tools). Each container remains separate from your host machine, preventing library version clashes with other projects. This separation is especially useful for data scientists who juggle multiple projects, each requiring a distinct set of Python libraries and system dependencies.

Below, we’ll walk through how to set up and maintain a Docker-based Python environment, how to connect and code within it, and how to share it with collaborators.

Setting Up Your Python Docker Environment

 
Here’s how you can set up your Docker environment in 4 easy steps.

1. Install Docker: Although we won’t go into Docker fundamentals, ensure you have Docker installed and running on your system.
2. Create a Project Folder: Make a dedicated directory for your project, for instance my_project. Inside it, create a Dockerfile which defines your environment.
3. Create a requirements.txt file: Create a file with the required dependencies for your project. For our purposes we will use the following:

pandas==2.1.3
numpy==1.26.0
requests==2.31.0
matplotlib==3.8.0

4. Write Your Dockerfile: A basic Dockerfile for Python might look like this:

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

Here’s what the above Dockerfile is doing:

• FROM python:3.9-slim picks a minimal Python 3.9 image
• WORKDIR /app sets the working directory
• COPY requirements.txt . copies your requirements file into the container
• RUN pip install installs all dependencies listed in requirements.txt
• COPY . . copies the rest of your project files into the container

5. Build the Docker Image: Open a terminal in the same directory and run:

docker build -t my_project_image .

This command pulls the base Python image, installs your dependencies, and packages them into my_project_image.

Managing Your Container

 
After building the Docker image, create a container from it. Containers can be started, stopped, paused, and removed without affecting your host system.

Starting the Container

Here is the command to run in order to get the container started:

docker run -it --name my_project_container -v $(pwd):/app my_project_image /bin/bash

Here is what’s going on in the above command:

• -it gives you an interactive session (shell)
• --name my_project_container assigns a readable name
• -v $(pwd):/app mounts your local project folder into the container so you can edit files in real time
• /bin/bash tells Docker to open a bash shell inside the container

Pausing and Resuming

If you need to step away or reboot, type exit in the container or press Ctrl+D to end the session. You can later start it again like so:

docker start -i my_project_container

Inspecting or Debugging

For inspecting and debugging your container, the following commands can be helpful:

• docker ps -a shows all containers
• docker logs my_project_container views any stdout logs if your container runs a script
• docker container ls -a can be run to retrieve your container ID if you ever forget

Writing Code Inside the Container

 
So now you have a container running your dependencies. But what about your code?

Using a Text Editor or IDE

You can use any text editor or IDE you would normally use to write and edit your code, anything from nano to VS Code and beyond will work just fine.

Let’s say you open a new file called script.py. Because you mounted the local directory -v $(pwd):/app, any changes saved in the container to /app/script.py appear on your host file system in real-time. This allows you to also use an editor or IDE on your host machine.

So open up an editor, add some code, and save it as myproject/script.py (which is the same as /app/script.py in your container) to try it out.

Running Python Code

Within the container, you can directly run:

root@de85bfd0f8e0:/app# python script.py

The environment in the container includes all libraries specified in requirements.txt. If you want to test this out, edit the script.py file above and add the imports to the libraries included in the requirements.txt file.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

print("Hello world!")

And that’s it.

Removing the Environment When Done

 
One of Docker’s strengths is that you can wipe away an environment in seconds. If you no longer need a container, stop it and remove it:

docker stop my_project_container
docker rm my_project_container

The associated image can also be removed if you’re finished using it:

docker rmi my_project_image

This entire environment will vanish, leaving no trace of installed libraries on your system.

Sharing Your Container Setup

 
Another feature of Docker is the ability to share your container setup with others with ease.

Share the Dockerfile and requirements.txt

The simplest method is to commit your Dockerfile and requirements.txt to a version control system like Git. Anyone can build the image locally with:

docker build -t my_project_image .

Push Your Image to a Registry

For teams that want to skip rebuilding locally, you can push your image to a registry (e.g., Docker Hub) and let others pull it:

docker tag my_project_image your_dockerhub_username/my_project_image:latest
docker push your_dockerhub_username/my_project_image:latest

Your collaborators can then run:

docker pull your_dockerhub_username/my_project_image:latest
docker run -it your_dockerhub_username/my_project_image:latest /bin/bash

This reproduces the environment instantly.

Version Control for Environments

If your project’s dependencies change, update your Dockerfile and requirements, rebuild, and push again with a new tag or version. This ensures your environment is always documented and easily recreated by anyone at any time.

Conclusion

 
Docker containers offer a reliable, reproducible approach to Python environment management. By isolating each project within its own container, you eliminate the conflicts often seen with local tools like env or conda. You can easily version your environment, share it, or dispose of it when it’s no longer needed — no clutter left behind. If you’re looking for a more flexible and scalable strategy to handle your Python projects, adopting Docker can streamline your workflow and free you from the pitfalls of local environment juggling.
 
 

Matthew Mayo (@mattmayo13) holds a master’s degree in computer science and a graduate diploma in data mining. As managing editor of KDnuggets & Statology, and contributing editor at Machine Learning Mastery, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, language models, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.