Automation: subprocess, shutil, and os

What you'll learn

• Run another program from Python with subprocess.run and capture its output
• Check exit codes and handle failures instead of ignoring them
• Copy, move, and delete files and whole folders with shutil
• Read and set environment variables and walk folder trees with os

Concept

Automation is the everyday superpower of Python: replacing a checklist of manual steps with a script that runs them in order, every time, the same way. Three standard modules cover most of it. subprocess runs other programs. shutil moves files around. os talks to the operating system about the environment and the filesystem. With pathlib from the files lesson, that is the whole toolkit for the scripts that keep servers, laptops, and build pipelines tidy.

subprocess.run(command) starts a program and waits for it to finish. Pass the command as a list, ["git", "status", "--short"], so each argument is separate and nothing is interpreted by a shell; that is both safer and more portable than one string. capture_output=True collects what the program printed into .stdout and .stderr, text=True decodes them into strings, and the returned object's .returncode tells you whether it succeeded, zero meaning yes. Pass check=True to raise CalledProcessError automatically on a non-zero code, so failures cannot be silently ignored.

shell=True is the option to be careful with. It hands your command string to the system shell, which enables pipes and wildcards but also means any user-provided text in the string is interpreted as shell syntax, the same injection problem as SQL. Use the list form by default, and if you truly need shell features, never place untrusted text in the command.

shutil does what its name says, shell utilities: copy and copy2 copy a file (the second keeps timestamps), copytree copies a folder, move renames or relocates, rmtree deletes a folder and everything in it with no confirmation, and make_archive builds a zip or tar.

os.environ is the dictionary of environment variables, the standard way configuration reaches scripts in production. os.walk yields every folder, its subfolders, and its files, top down, which is how you process a whole tree; pathlib's rglob does the same for a pattern.

Code

Running a program and reading its output. The command is the pinned Python itself, so the example works everywhere Python does, and the exit code is checked explicitly.

# run_program.py
import subprocess
import sys

result = subprocess.run([sys.executable, "-c", "print('hello from a child process')"],
                        capture_output=True, text=True, encoding="utf-8")
print(result.returncode)
print(result.stdout.strip())
print(repr(result.stderr))

version = subprocess.run([sys.executable, "--version"], capture_output=True, text=True)
print(version.stdout.split()[0], version.stdout.split()[1][:4])

0
hello from a child process
''
Python 3.14

Failures: a non-zero exit code is a value to check, or an exception with check=True. A timeout stops a program that runs too long.

# failures.py
import subprocess
import sys

bad = subprocess.run([sys.executable, "-c", "import sys; print('failing'); sys.exit(3)"], capture_output=True, text=True)
print(bad.returncode, bad.stdout.strip())

try:
    subprocess.run([sys.executable, "-c", "raise SystemExit(1)"], check=True, capture_output=True)
except subprocess.CalledProcessError as err:
    print("raised:", type(err).__name__, "code", err.returncode)

try:
    subprocess.run([sys.executable, "-c", "import time; time.sleep(5)"], timeout=0.5)
except subprocess.TimeoutExpired:
    print("timed out after 0.5s")

3 failing
raised: CalledProcessError code 1
timed out after 0.5s

Files and folders with shutil: copy, copy a tree, move, archive, and remove. Everything happens inside a scratch folder created for the demo.

# files_folders.py
import shutil
from pathlib import Path

root = Path("scratch")
shutil.rmtree(root, ignore_errors=True)
(root / "src").mkdir(parents=True)
(root / "src" / "app.py").write_text("print('app')\n", encoding="utf-8")
(root / "src" / "util.py").write_text("x = 1\n", encoding="utf-8")

shutil.copy2(root / "src" / "app.py", root / "app_backup.py")
shutil.copytree(root / "src", root / "src_copy")
shutil.move(str(root / "app_backup.py"), str(root / "src_copy" / "backup.py"))
archive = shutil.make_archive(str(root / "release"), "zip", root / "src")

print(sorted(p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file()))
print(Path(archive).name, Path(archive).stat().st_size > 0)
shutil.rmtree(root / "src_copy")
print((root / "src_copy").exists())

['release.zip', 'src/app.py', 'src/util.py', 'src_copy/app.py', 'src_copy/backup.py', 'src_copy/util.py']
release.zip True
False

Environment variables and walking a tree. os.environ is read with a default, and os.walk reports each folder's files; sorting the subfolder list in place makes the walk order predictable, and rglob reaches the same files by pattern.

# env_walk.py
import os
from pathlib import Path

os.environ["APP_ENV"] = "staging"
print(os.environ.get("APP_ENV"), os.environ.get("MISSING_VAR", "fallback"))

root = Path("tree")
for sub in ("src", "tests"):
    (root / sub).mkdir(parents=True, exist_ok=True)
(root / "README.md").write_text("# tree\n")
(root / "src" / "app.py").write_text("")
(root / "tests" / "test_app.py").write_text("")

for folder, subfolders, files in os.walk(root):
    subfolders.sort()            # sorting in place also fixes the order os.walk visits them
    print(f"{Path(folder).as_posix()}: dirs={subfolders} files={sorted(files)}")

print(sorted(p.name for p in root.rglob("*.py")))

staging fallback
tree: dirs=['src', 'tests'] files=['README.md']
tree/src: dirs=[] files=['app.py']
tree/tests: dirs=[] files=['test_app.py']
['app.py', 'test_app.py']

Try it yourself

Create backup.py, a command that takes a source folder and a destination folder, copies the source tree into a timestamped subfolder of the destination, zips it with make_archive, and deletes the unzipped copy. Print the archive path and size. Then add a --verify flag that runs python -m zipfile -l <archive> with subprocess.run and prints its output. Run it on a folder with a few small files.

Common mistakes

• Building a command as one string with shell=True. Use a list of arguments; the shell is only for shell features, and never with user-supplied text.
• Ignoring the return code. A failed command that nobody checks looks like success; use check=True or test returncode.
• rmtree without a guard. It deletes the folder and everything below it instantly; check the path is what you think, and never pass a path built from unvalidated input.

What's next

• subprocess.run with a list of arguments runs a program; capture_output, text, check, and timeout cover the everyday needs.
• shutil copies, moves, archives, and removes files and trees; os.environ and os.walk read the environment and the filesystem.
• Avoid shell=True, check return codes, and treat rmtree with respect.

You can automate a machine. Next you put the whole Beginner level to work in a project: a file-organizing command with classes for its rules, a dry-run flag, and a log.