██FR█████ █INTELL███████████

The Python Developer’s Guide to Background Process Management


This content originally appeared on DEV Community and was authored by Mr Vi

As a Python developer, you’ve probably faced this scenario: you’ve built an amazing bot, API, or data processing script, but keeping it running 24/7 is a nightmare. SSH sessions disconnect, scripts crash, servers reboot, and suddenly your carefully crafted automation is offline.

In this comprehensive guide, we’ll explore different approaches to running Python applications in the background, from simple development solutions to production-ready process management.

The Problem with “Just Running Python”

When you develop locally, you run scripts like this:

python my_awesome_bot.py

But in production, this approach fails because:

  • SSH disconnections kill your process
  • Script crashes require manual intervention
  • Server reboots lose all running processes
  • Managing multiple bots becomes chaotic
  • No visibility into what’s actually running

Let’s fix this with proper process management.

Method 1: Screen – The Development Solution

Screen is perfect for development and testing. It creates persistent terminal sessions that survive SSH disconnections.

Getting Started with Screen 

# Install screen
sudo apt install screen  # Ubuntu/Debian
sudo yum install screen  # CentOS/RHEL

# Start a new session
screen

# Activate your virtual environment
source ./venv/bin/activate

# Run your script
python my_bot.py

# Detach (Ctrl+A, then D)
# Your script keeps running!

Managing Screen Sessions 

# List all sessions
screen -ls

# Reattach to a session
screen -r session_name

# Kill a session (from within screen)
# Ctrl+A, then K

When to use Screen:

  • ✅ Development and testing
  • ✅ Quick one-off scripts
  • ✅ When you need terminal access

Limitations:

  • ❌ No automatic restart on crashes
  • ❌ Manual process management
  • ❌ Sessions can be lost on server reboot
  • ❌ No built-in monitoring

Method 2: Systemd – The System Service Approach

Systemd is Linux’s service manager, perfect for production environments where you need guaranteed startup and restart capabilities.

Creating a Systemd Service

Create a service file:

sudo nano /etc/systemd/system/mybot.service

Add this configuration:

[Unit]
Description=My Python Bot Service
After=network.target

[Service]
Type=idle
Restart=always
RestartSec=3
User=myuser
WorkingDirectory=/home/myuser/bot
ExecStart=/home/myuser/bot/venv/bin/python /home/myuser/bot/my_bot.py
Environment=PATH=/home/myuser/bot/venv/bin

[Install]
WantedBy=multi-user.target

Managing the Service 

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable auto-start on boot
sudo systemctl enable mybot.service

# Start the service
sudo systemctl start mybot.service

# Check status
sudo systemctl status mybot.service

# View logs
sudo journalctl -u mybot.service -f

# Stop the service
sudo systemctl stop mybot.service

When to use Systemd:

  • ✅ Production servers
  • ✅ Critical services that must survive reboots
  • ✅ System-level integration
  • ✅ When you need fine-grained control

Limitations:

  • ❌ Requires root access
  • ❌ Complex configuration
  • ❌ System-level management
  • ❌ No built-in process monitoring

Method 3: Modern Process Managers – The Python Way

For Python developers, there’s a better way: Pyker – a process manager built specifically for Python applications.

Why Pyker?

Pyker combines the simplicity of screen with the power of systemd, but designed specifically for Python workflows:

  • Python-native: Built for Python developers, by Python developers
  • Virtual environment support: Native venv/conda integration
  • Zero configuration: Works out of the box
  • Beautiful monitoring: Real-time process tables
  • User-space installation: No root access required

Installation 

# One-line installation
curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.sh | bash

# Or with Python installer
curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.py | python3

# Manual installation
git clone https://github.com/mrvi0/pyker.git
cd pyker
python3 install.py

Basic Usage 

# Start a simple script
pyker start mybot my_bot.py

# Start with virtual environment
pyker start webapp app.py --venv ./venv

# List all processes
pyker list

# View logs in real-time
pyker logs mybot -f

# Get detailed process information
pyker info mybot

# Restart a process
pyker restart mybot

# Stop a process
pyker stop mybot

Advanced Features

Auto-restart on failure:

pyker start critical-service app.py --auto-restart

Virtual environment support:

# Works with any Python environment
pyker start ml-worker train.py --venv /path/to/conda/envs/pytorch
pyker start webapp app.py --venv ./venv
pyker start data-processor process.py --venv /home/user/projects/venv

Process monitoring:

pyker list

Shows a beautiful, adaptive table with:

  • Process status (✓ running, ✗ stopped, ⚠ error)
  • Real-time CPU and memory usage
  • Start/stop times
  • Virtual environment information
  • Script paths

Real-time logs:

pyker logs mybot -f        # Follow logs in real-time
pyker logs mybot -n 100    # Show last 100 lines

Detailed process information:

pyker info mybot

Shows comprehensive information including:

  • Virtual environment path and Python executable
  • Resource usage (CPU, memory)
  • Runtime information
  • Log file location
  • Auto-restart status

Tab completion:

pyker <TAB>                # Shows available commands
pyker stop <TAB>           # Completes with running process names
pyker start app script.py --<TAB>  # Shows --venv, --auto-restart

Real-World Examples

Discord Bot Management 

# Traditional approach with screen
screen
source ./venv/bin/activate
python discord_bot.py
# Ctrl+A, D

# With Pyker
pyker start discord-bot discord_bot.py --venv ./venv --auto-restart
pyker logs discord-bot -f

Web API Deployment 

# With systemd (complex setup)
sudo nano /etc/systemd/system/api.service
# ... complex configuration ...
sudo systemctl enable api.service

# With Pyker (simple)
pyker start api app.py --venv ./venv --auto-restart

Data Processing Pipeline 

# Multiple related processes
pyker start scraper scraper.py --venv ./data-env
pyker start processor process_data.py --venv ./data-env  
pyker start uploader upload_to_s3.py --venv ./data-env

# Monitor all processes
pyker list

# Check specific process
pyker info scraper

Machine Learning Workflows 

# Training with proper environment
pyker start training train_model.py --venv /opt/conda/envs/pytorch --auto-restart

# Inference service
pyker start inference inference_server.py --venv ./ml-env

# Data preprocessing
pyker start preprocess preprocess_data.py --venv ./data-env

Comparison: Choosing the Right Method

Method Complexity Auto-restart Virtual Env Monitoring Best For
Screen Low ❌ Manual Basic Development, testing
Systemd High ✅ Manual System logs Production servers
Pyker Low ✅ Native Built-in Python applications

Best Practices for Production

1. Always Use Virtual Environments 

# Create and activate environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run with Pyker
pyker start mybot my_bot.py --venv ./venv

2. Enable Auto-restart for Critical Services 

pyker start critical-service app.py --auto-restart

3. Implement Proper Logging 

import logging
import sys

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler('app.log')
    ]
)

logger = logging.getLogger(__name__)

def main():
    logger.info("Application started")
    try:
        # Your application logic
        pass
    except Exception as e:
        logger.error(f"Application error: {e}")
        raise

if __name__ == "__main__":
    main()

4. Handle Graceful Shutdowns 

import signal
import sys
import logging

logger = logging.getLogger(__name__)

def signal_handler(sig, frame):
    logger.info("Received shutdown signal, cleaning up...")
    # Cleanup code here
    sys.exit(0)

# Register signal handlers
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)

5. Monitor Your Processes 

# Regular status checks
pyker list

# Monitor logs
pyker logs mybot -f

# Check system resources
htop

Troubleshooting Common Issues

Process Won’t Start 

# Test the script manually first
python my_bot.py

# Check Pyker logs
pyker logs mybot

# Verify process information
pyker info mybot

Virtual Environment Issues 

# Verify venv path exists
ls -la ./venv/bin/python

# Test with absolute path
pyker start mybot my_bot.py --venv /full/path/to/venv

# Check Python executable
./venv/bin/python --version

High Resource Usage 

# Monitor resource usage
pyker list  # Shows CPU and memory columns

# Check system resources
htop
top

# Kill problematic processes
pyker stop process_name

Log Management 

# View log files directly
ls -la ~/.pyker/logs/

# Clean old logs
rm ~/.pyker/logs/*.log.*

# Configure log rotation
nano ~/.pyker/config.json

Security Considerations

User Permissions 

# Run as non-root user
pyker start mybot my_bot.py --venv ./venv

# Check process ownership
ps aux | grep mybot

Environment Variables 

# Set environment variables
export API_KEY="your-secret-key"
pyker start mybot my_bot.py --venv ./venv

Network Security 

# Bind to localhost only
pyker start api app.py --venv ./venv
# Configure your app to bind to 127.0.0.1:8000

Conclusion

Running Python applications in production doesn’t have to be complicated. The key is choosing the right tool for your needs:

  • Development/Testing: Use Screen for quick testing
  • System Services: Use Systemd for critical system services
  • Python Applications: Use Pyker for modern Python process management

Pyker offers the perfect balance of simplicity and power for Python developers. It’s designed specifically for Python workflows, supports virtual environments natively, and provides beautiful monitoring without the complexity of systemd.

The best part? You can start simple and scale up. Begin with Screen for development, then move to Pyker for production, and only use Systemd when you need deep system integration.

Your Python applications deserve better than manual restarts and SSH sessions. Choose the right tool, and let your code run reliably 24/7.

What’s your experience with running Python applications in production? Have you tried any of these approaches? Share your thoughts in the comments below!

python #devops #processmanagement #automation #production #linux #systemd #backgroundprocesses


This content originally appeared on DEV Community and was authored by Mr Vi