Lesson 14 - Virtual Environments and Dependency Management

Introduction

As you work on multiple Python projects, you’ll quickly encounter a common problem: different projects need different versions of the same package. One project might need Django 3.2, while another requires Django 4.0. Installing both globally would cause conflicts.

Virtual environments solve this problem by creating isolated Python environments for each project. Each environment has its own Python interpreter and installed packages, completely separate from your system Python and other projects.

In this lesson, you’ll learn:

What virtual environments are and why they’re essential
How to create and activate virtual environments
Managing packages with pip
Tracking dependencies with requirements.txt
Best practices for project organization
Common troubleshooting techniques

The Problem: Dependency Conflicts

Imagine you’re working on three different projects:

Project A (Bookstore API):

Needs Django 3.2
Uses requests 2.25.0
Requires pandas 1.2.0

Project B (Data Analysis Tool):

Needs Django 4.0
Uses requests 2.28.0
Requires pandas 1.5.0

Project C (Machine Learning App):

Needs Flask 2.0
Uses numpy 1.21.0
Requires scikit-learn 1.0.0

If you install all these packages globally (system-wide), you’ll have conflicts:

# System Python
pip install django==3.2   # For Project A
pip install django==4.0   # Overwrites 3.2! Project A breaks!

The Solution: Each project gets its own isolated environment with its own packages.

System Python (clean, minimal)
├── Project A (virtual environment)
│   ├── Django 3.2
│   ├── requests 2.25.0
│   └── pandas 1.2.0
├── Project B (virtual environment)
│   ├── Django 4.0
│   ├── requests 2.28.0
│   └── pandas 1.5.0
└── Project C (virtual environment)
    ├── Flask 2.0
    ├── numpy 1.21.0
    └── scikit-learn 1.0.0

What is a Virtual Environment?

A virtual environment is a self-contained directory that contains:

Python interpreter (a copy or link to your system Python)
pip (package installer)
Installed packages specific to this project
Scripts to activate/deactivate the environment

Think of it like this:

Without virtual environments: All packages live in one shared apartment (system Python). Everyone fights over which version of Django to install.
With virtual environments: Each project gets its own apartment (isolated environment) with its own furniture (packages).

Creating Virtual Environments

Python 3.3+ includes venv module built-in for creating virtual environments.

Basic Syntax

python -m venv <environment_name>

Example: Creating a Virtual Environment

Let’s create a virtual environment for a bookstore project:

# Navigate to your project directory
cd ~/projects/bookstore

# Create virtual environment named 'venv'
python -m venv venv

Or on some systems:

python3 -m venv venv

What just happened?

This command created a new directory called venv containing:

bookstore/
├── venv/
│   ├── bin/          # (Linux/Mac) Executables (python, pip, activate)
│   ├── Scripts/      # (Windows) Executables
│   ├── lib/          # Installed packages
│   ├── include/      # C headers
│   └── pyvenv.cfg    # Configuration
├── main.py
└── README.md

Common environment names:

venv (most common)
env
.venv (hidden directory)
virtualenv

Activating Virtual Environments

Creating the environment isn’t enough—you need to activate it to use it.

On macOS/Linux:

source venv/bin/activate

On Windows (Command Prompt):

venv\Scripts\activate.bat

On Windows (PowerShell):

venv\Scripts\Activate.ps1

Note: If you get a PowerShell execution policy error, run:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

How to Know It’s Activated

When activated, your terminal prompt changes:

Before activation:

user@computer:~/projects/bookstore$

After activation:

(venv) user@computer:~/projects/bookstore$

The (venv) prefix indicates you’re inside the virtual environment.

Verifying the Active Environment

Check which Python you’re using:

# Check Python path
which python     # macOS/Linux
where python     # Windows

# Check Python version
python --version

# Check pip location
which pip        # macOS/Linux
where pip        # Windows

Inside venv:

/home/user/projects/bookstore/venv/bin/python

Outside venv:

/usr/bin/python  # System Python

Deactivating Virtual Environments

To leave the virtual environment and return to system Python:

deactivate

Your prompt returns to normal:

user@computer:~/projects/bookstore$

Installing Packages in Virtual Environments

Once your environment is activated, packages you install with pip go into this environment only—not your system Python.

Installing Packages

# Activate environment first
source venv/bin/activate  # macOS/Linux
# or
venv\Scripts\activate     # Windows

# Install packages
pip install django
pip install requests pandas numpy
pip install flask==2.0.1  # Specific version

Checking Installed Packages

pip list

Output:

Package    Version
---------- -------
Django     4.2.7
pip        23.3.1
requests   2.31.0
setuptools 68.2.0

Showing Package Details

pip show django

Output:

Name: Django
Version: 4.2.7
Summary: A high-level Python web framework
Home-page: https://www.djangoproject.com/
Author: Django Software Foundation
License: BSD-3-Clause
Location: /path/to/venv/lib/python3.11/site-packages
Requires: asgiref, sqlparse
Required-by:

Uninstalling Packages

pip uninstall django

Upgrading Packages

# Upgrade specific package
pip install --upgrade django

# Upgrade pip itself
pip install --upgrade pip

Managing Dependencies with requirements.txt

For professional projects, you need a way to track and share which packages your project needs. This is where requirements.txt comes in.

What is requirements.txt?

A text file listing all packages and their versions needed for your project.

Example requirements.txt:

django==4.2.7
requests==2.31.0
pandas==2.1.3
numpy==1.26.2
python-dateutil==2.8.2

Creating requirements.txt

After installing all your packages, generate the file:

pip freeze > requirements.txt

This captures the exact versions of all installed packages.

Example output:

cat requirements.txt

asgiref==3.7.2
Django==4.2.7
numpy==1.26.2
pandas==2.1.3
python-dateutil==2.8.2
pytz==2023.3
requests==2.31.0
six==1.16.0
sqlparse==0.4.4
urllib3==2.1.0

Installing from requirements.txt

When someone else (or you on another machine) wants to set up the same environment:

# Create and activate new environment
python -m venv venv
source venv/bin/activate  # or Windows equivalent

# Install all dependencies
pip install -r requirements.txt

This installs all packages with the exact versions specified.

Version Specifiers

You can specify version requirements flexibly:

# Exact version
django==4.2.7

# Minimum version
requests>=2.28.0

# Compatible release (same major.minor)
pandas~=2.1.0

# Version range
numpy>=1.20.0,<2.0.0

# Latest version (not recommended for production)
flask

Best Practice: Use exact versions (==) in requirements.txt for reproducibility.

Practical Example: Bookstore Project Setup

Let’s set up a complete bookstore project with virtual environment.

Step 1: Create Project Directory

mkdir bookstore-project
cd bookstore-project

Step 2: Create Virtual Environment

python -m venv venv

Step 3: Activate Environment

# macOS/Linux
source venv/bin/activate

# Windows
venv\Scripts\activate

Step 4: Install Dependencies

pip install flask pandas requests

Step 5: Create Application

Create app.py:

from flask import Flask, jsonify
import pandas as pd

app = Flask(__name__)

# Sample book data
books_data = {
    'title': ['Python Crash Course', 'Automate the Boring Stuff', 'Learning Python'],
    'author': ['Eric Matthes', 'Al Sweigart', 'Mark Lutz'],
    'price': [39.99, 29.99, 59.99],
    'rating': [4.7, 4.6, 4.5]
}

@app.route('/')
def home():
    return jsonify({'message': 'Welcome to Bookstore API'})

@app.route('/books')
def get_books():
    df = pd.DataFrame(books_data)
    return df.to_json(orient='records')

@app.route('/average-price')
def average_price():
    df = pd.DataFrame(books_data)
    avg = df['price'].mean()
    return jsonify({'average_price': f'${avg:.2f}'})

if __name__ == '__main__':
    app.run(debug=True)

Step 6: Save Dependencies

pip freeze > requirements.txt

Step 7: Project Structure

bookstore-project/
├── venv/                 # Virtual environment (don't commit to git)
├── app.py               # Application code
├── requirements.txt     # Dependencies
└── .gitignore          # Git ignore file

.gitignore content:

# Virtual environment
venv/
env/
.venv/

# Python
__pycache__/
*.py[cod]
*.so
*.egg-info/

# IDE
.vscode/
.idea/

Step 8: Run the Application

python app.py

Visit http://127.0.0.1:5000/books to see the JSON response.

When sharing with teammates:

Don’t include venv folder (add to .gitignore)
Include requirements.txt
Include setup instructions in README.md

README.md example:

# Bookstore API

## Setup

1. Create virtual environment:
   ```bash
   python -m venv venv

Activate environment:

source venv/bin/activate  # macOS/Linux
venv\Scripts\activate     # Windows

Install dependencies:
```
pip install -r requirements.txt
```
Run application:
```
python app.py
```


## Advanced: Multiple requirements Files

For larger projects, you might have different requirements for different environments:

```text
requirements/
├── base.txt          # Common dependencies
├── development.txt   # Dev tools (pytest, black, etc.)
├── production.txt    # Production-only packages
└── testing.txt       # Testing dependencies

base.txt:

django==4.2.7
requests==2.31.0
pandas==2.1.3

development.txt:

-r base.txt           # Include base requirements
black==23.11.0        # Code formatter
pytest==7.4.3         # Testing framework
ipython==8.18.1       # Interactive shell

production.txt:

-r base.txt
gunicorn==21.2.0      # Production server

Install for development:

pip install -r requirements/development.txt

Comparing Virtual Environment Tools

Built-in venv (Recommended for Beginners)

Pros:

Built into Python 3.3+
No installation needed
Official and well-supported

Cons:

Basic features only
Slower than some alternatives

virtualenv (Third-party)

Pros:

Faster than venv
More features
Works with Python 2

Cons:

Requires separate installation

pip install virtualenv
virtualenv venv

conda (Data Science Focus)

Pros:

Manages Python versions and packages
Great for data science
Handles non-Python dependencies

Cons:

Larger download
Slower

conda create -n myenv python=3.11
conda activate myenv

pipenv (Modern Alternative)

Pros:

Combines pip and venv
Automatic dependency resolution
Uses Pipfile instead of requirements.txt

Cons:

Additional tool to learn
Slower dependency resolution

pip install pipenv
pipenv install django
pipenv shell

poetry (Professional Projects)

Pros:

Modern dependency management
Handles packaging and publishing
Better dependency resolution

Cons:

Learning curve
Different workflow

pip install poetry
poetry new myproject
poetry add django

Best Practices

1. One Environment Per Project

✅ GOOD:
projects/
├── bookstore/
│   └── venv/
├── analytics/
│   └── venv/
└── api/
    └── venv/

❌ BAD:
projects/
└── shared-venv/  # Used by all projects

2. Name Environment Consistently

Use venv, env, or .venv consistently across projects.

3. Add to .gitignore

Never commit virtual environments to version control:

# .gitignore
venv/
env/
.venv/

4. Always Activate Before Working

# Start work session
cd myproject
source venv/bin/activate
python app.py

# End work session
deactivate

5. Keep requirements.txt Updated

After installing/updating packages:

pip freeze > requirements.txt

6. Use Exact Versions in Production

# Development (flexible)
django>=4.0

# Production (exact)
django==4.2.7

7. Separate Dev and Prod Dependencies

Use separate requirements files or mark optional dependencies.

8. Document Setup Process

Include clear setup instructions in README.md.

Common Issues and Solutions

Issue 1: “python: command not found”

Solution: Use python3 instead:

python3 -m venv venv

Issue 2: PowerShell Execution Policy Error

Solution:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Issue 3: Packages Installing Globally

Problem: Forgot to activate environment.

Solution: Always check for (venv) in prompt before pip install.

Issue 4: “No module named X” After Activation

Problem: Package installed in different environment.

Solution: Verify active environment:

which python  # Should point to venv/bin/python
pip list      # Check installed packages

Issue 5: Requirements.txt Too Large

Problem: pip freeze includes all dependencies.

Solution: Use pipreqs to generate minimal requirements:

pip install pipreqs
pipreqs /path/to/project

Real-World Workflow Example

Starting a New Project

# 1. Create project directory
mkdir my-data-project
cd my-data-project

# 2. Initialize git
git init

# 3. Create .gitignore
echo "venv/" > .gitignore
echo "__pycache__/" >> .gitignore

# 4. Create virtual environment
python -m venv venv

# 5. Activate it
source venv/bin/activate

# 6. Upgrade pip
pip install --upgrade pip

# 7. Install packages
pip install pandas numpy matplotlib jupyter

# 8. Save dependencies
pip freeze > requirements.txt

# 9. Create main script
touch main.py

# 10. Start coding!
code .  # Open in VS Code

Returning to Existing Project

# 1. Navigate to project
cd my-data-project

# 2. Activate environment
source venv/bin/activate

# 3. Verify environment
which python

# 4. Update packages if needed
pip install --upgrade -r requirements.txt

# 5. Start working
python main.py

Collaborating with Team

Team Member A (Creates project):

# Create and set up project
python -m venv venv
source venv/bin/activate
pip install django pandas requests
pip freeze > requirements.txt

# Commit to git (without venv/)
git add .
git commit -m "Initial project setup"
git push

Team Member B (Clones project):

# Clone repository
git clone https://github.com/team/bookstore.git
cd bookstore

# Set up environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Ready to work!
python manage.py runserver

Summary

In this lesson, we learned about Python virtual environments:

Key Concepts

Virtual environments isolate project dependencies
venv is built into Python 3.3+
Activation is required to use an environment
requirements.txt tracks project dependencies
pip freeze captures installed packages

Essential Commands

# Create
python -m venv venv

# Activate
source venv/bin/activate          # macOS/Linux
venv\Scripts\activate             # Windows

# Install packages
pip install package-name
pip install -r requirements.txt

# Save dependencies
pip freeze > requirements.txt

# Deactivate
deactivate

Best Practices

One environment per project
Never commit venv/ to git
Always activate before working
Keep requirements.txt updated
Use exact versions in production
Document setup process

Workflow

1. Create project directory
2. Create virtual environment (python -m venv venv)
3. Activate environment
4. Install packages
5. Save dependencies (pip freeze > requirements.txt)
6. Code your project
7. Deactivate when done

Why This Matters

Virtual environments are essential for professional Python development:

Avoid dependency conflicts between projects
Reproducible environments across machines
Clean system Python stays minimal
Easy collaboration with teammates
Professional best practice in industry

By mastering virtual environments, you’re not just learning a tool—you’re adopting a professional workflow that will serve you throughout your Python career. Every serious Python project uses virtual environments, from small scripts to large enterprise applications.

Now you’re ready to manage Python projects like a professional!

Lesson 13 - Modules and Packages

Overview of the NumPy Fundamentals Module

Lesson 14 - Virtual Environments and Dependency Management

Introduction#

The Problem: Dependency Conflicts#

What is a Virtual Environment?#

Creating Virtual Environments#

Basic Syntax#

Example: Creating a Virtual Environment#

Activating Virtual Environments#

On macOS/Linux:#

On Windows (Command Prompt):#

On Windows (PowerShell):#

How to Know It’s Activated#

Verifying the Active Environment#

Deactivating Virtual Environments#

Installing Packages in Virtual Environments#

Installing Packages#

Checking Installed Packages#

Showing Package Details#

Uninstalling Packages#

Upgrading Packages#

Managing Dependencies with requirements.txt#

What is requirements.txt?#

Creating requirements.txt#

Installing from requirements.txt#

Version Specifiers#

Practical Example: Bookstore Project Setup#

Step 1: Create Project Directory#

Step 2: Create Virtual Environment#

Step 3: Activate Environment#

Step 4: Install Dependencies#

Step 5: Create Application#

Step 6: Save Dependencies#

Step 7: Project Structure#

Step 8: Run the Application#

Step 9: Sharing the Project#

Comparing Virtual Environment Tools#

Built-in venv (Recommended for Beginners)#

virtualenv (Third-party)#

conda (Data Science Focus)#

pipenv (Modern Alternative)#

poetry (Professional Projects)#

Best Practices#

1. One Environment Per Project#

2. Name Environment Consistently#

3. Add to .gitignore#

4. Always Activate Before Working#

5. Keep requirements.txt Updated#

6. Use Exact Versions in Production#

7. Separate Dev and Prod Dependencies#

8. Document Setup Process#

Common Issues and Solutions#

Issue 1: “python: command not found”#

Issue 2: PowerShell Execution Policy Error#

Issue 3: Packages Installing Globally#

Issue 4: “No module named X” After Activation#

Issue 5: Requirements.txt Too Large#

Real-World Workflow Example#

Starting a New Project#

Returning to Existing Project#

Collaborating with Team#

Summary#

Key Concepts#

Essential Commands#

Best Practices#

Workflow#

Why This Matters#

Introduction

The Problem: Dependency Conflicts

What is a Virtual Environment?

Creating Virtual Environments

Basic Syntax

Example: Creating a Virtual Environment

Activating Virtual Environments

On macOS/Linux:

On Windows (Command Prompt):

On Windows (PowerShell):

How to Know It’s Activated

Verifying the Active Environment

Deactivating Virtual Environments

Installing Packages in Virtual Environments

Installing Packages

Checking Installed Packages

Showing Package Details

Uninstalling Packages

Upgrading Packages

Managing Dependencies with requirements.txt

What is requirements.txt?

Creating requirements.txt

Installing from requirements.txt

Version Specifiers

Practical Example: Bookstore Project Setup

Step 1: Create Project Directory

Step 2: Create Virtual Environment

Step 3: Activate Environment

Step 4: Install Dependencies

Step 5: Create Application

Step 6: Save Dependencies

Step 7: Project Structure

Step 8: Run the Application

Step 9: Sharing the Project

Comparing Virtual Environment Tools

Built-in venv (Recommended for Beginners)

virtualenv (Third-party)

conda (Data Science Focus)

pipenv (Modern Alternative)

poetry (Professional Projects)

Best Practices

1. One Environment Per Project

2. Name Environment Consistently

3. Add to .gitignore

4. Always Activate Before Working

5. Keep requirements.txt Updated

6. Use Exact Versions in Production

7. Separate Dev and Prod Dependencies

8. Document Setup Process

Common Issues and Solutions

Issue 1: “python: command not found”

Issue 2: PowerShell Execution Policy Error

Issue 3: Packages Installing Globally

Issue 4: “No module named X” After Activation

Issue 5: Requirements.txt Too Large

Real-World Workflow Example

Starting a New Project

Returning to Existing Project

Collaborating with Team

Summary

Key Concepts

Essential Commands

Best Practices

Workflow

Why This Matters