9.1 Module Fundamentals
A module is just a .py file — any file that defines functions, classes, or
variables can be imported and reused elsewhere, instead of being copy-pasted. This is
how a program scales past "one script": related code lives in its own file/namespace,
and other code pulls in only what it needs.
Set up a tiny module to work with for the rest of this section (writing it to a
temporary directory keeps this self-contained — normally a module is just a .py file
sitting next to the code that imports it):
import sys, tempfile, os
demo_dir = tempfile.mkdtemp()
with open(os.path.join(demo_dir, "greetings.py"), "w") as f:
f.write(
'"""A tiny demo module for chapter 9."""\n'
'print("greetings module is being imported")\n'
"\n"
'LANGUAGE = "en"\n'
"\n"
"def hello(name):\n"
' return f"Hello, {name}!"\n'
)
sys.path.insert(0, demo_dir) # so `import greetings` can find it (9.2 covers this search path)
import greetings
print(greetings.hello("Ada")) # greetings module is being imported / Hello, Ada!
greetings module is being imported Hello, Ada!
Import forms:
| Form | Effect |
|---|---|
import module |
binds the whole module object to module |
import module as alias |
same, under a shorter/clearer name |
from module import name |
binds just name directly, no module. prefix needed |
from module import name as alias |
same, renamed |
from module import * |
binds every public name from the module directly — generally avoided, since it makes it unclear where a name came from |
import math as m
from math import sqrt
from math import sqrt as square_root
print(m.sqrt(16)) # 4.0
print(sqrt(16)) # 4.0
print(square_root(16)) # 4.0 -> three different ways to reach the same function
4.0 4.0 4.0
Note: __all__ isn't only a package-level tool — 9.3 shows it controlling
from package import *, but assigning __all__ = [...] works exactly the same way in
any plain module too. It's the direct, in-file fix for the "unclear where a name came
from" problem with from module import * flagged in the table above: it lets a module
declare exactly which names a wildcard import is allowed to pull in.
Special module members — every module carries some built-in attributes automatically, without needing to define them:
| Attribute | Holds |
|---|---|
__name__ |
the module's name — "__main__" if run directly instead of imported (9.4) |
__doc__ |
its docstring — the string literal at the top of the file, if any |
__file__ |
the path to the source file it was loaded from |
__package__ |
the package it belongs to ("" for a top-level module, 9.3 covers packages) |
__dict__ |
its entire namespace as a regular dict — every name the module defines |
print(greetings.__name__) # greetings
print(greetings.__doc__) # A tiny demo module for chapter 9.
print(greetings.__file__.endswith("greetings.py")) # True
print(repr(greetings.__package__)) # '' -> top-level module, not part of a package
print("hello" in greetings.__dict__) # True -> everything it defines lives here
greetings A tiny demo module for chapter 9. True '' True
dir() lists the names available in a namespace — call it on a module to see
everything it defines (its own names plus the special attributes above), sorted
alphabetically:
print(dir(greetings))
# ['LANGUAGE', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__',
# '__name__', '__package__', '__spec__', 'hello']
# in practice, usually filtered down to just the "public API":
public_names = [name for name in dir(greetings) if not name.startswith("_")]
print(public_names) # ['LANGUAGE', 'hello']
['LANGUAGE', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'hello'] ['LANGUAGE', 'hello']
In practice — this is the anatomy of every real Python package. Essentially every
modern open-source library and production service repository organizes its code as a
package with exactly this shape (often under a src/ layout) plus a pyproject.toml
declaring its metadata and dependencies — the greetings module here is a miniature of
what a real project's own package looks like.
9.2 The Import System and Search Path
sys.path is the list of directories Python searches, in order, for a module named
in an import statement — the first match wins:
import greetings
│
▼
sys.path searched in order:
1. '' — the running script's own directory (or the current directory for -c/-m)
2. directories listed in the PYTHONPATH environment variable, if set
3. the standard library's own directories
4. site-packages — where pip-installed third-party packages live (9.7)
│
▼
first directory containing greetings.py (or a greetings/ package) wins
That first entry is exactly why inserting demo_dir at the front of sys.path in 9.1
made import greetings work — without it, Python would never have looked there:
import sys
print(sys.path[0] == demo_dir) # True -> the insert() in 9.1 put it at the very front of the search order
print(len(sys.path) > 1) # True -> the standard library's directories are in here too
True True
Common mistake — naming a personal script after a standard-library module (random.py,
json.py, types.py...) silently shadows the real one. Since entry 1 of
sys.path is the running script's own directory, Python finds your file first and
never even looks at the stdlib version — no error, no warning, just mysterious
AttributeErrors when code expects the real module's functions:
import subprocess, tempfile, os
shadow_dir = tempfile.mkdtemp()
with open(os.path.join(shadow_dir, "random.py"), "w") as f:
f.write('print("MY random.py got imported, not the stdlib one")\n')
# run in a fresh process, cwd = shadow_dir, so '' in sys.path resolves there
result = subprocess.run(
["python3", "-c", "import random; print(random.__file__); print(hasattr(random, 'choice'))"],
capture_output=True, text=True, cwd=shadow_dir,
)
print(result.stdout)
# MY random.py got imported, not the stdlib one
# <shadow_dir>/random.py
# False -> the real random.choice() is simply gone, shadowed by our own file
MY random.py got imported, not the stdlib one /tmp/tmpk8n6tvmm/random.py False
A module is only ever executed once per process. The first import runs the
file top to bottom and stores the resulting module object in sys.modules, keyed by
name; every import after that — anywhere else in the program — just hands back the
same cached object instead of re-running the file:
import greetings
│
▼
already in sys.modules?
├── yes → return the cached module object (no re-execution)
└── no → read greetings.py, execute its code top to bottom,
store the module object in sys.modules, then return it
greetings module is being imported only printed once back in 9.1, even though this
notebook will go on to reference greetings many more times — this is why:
import greetings # runs again -> no "being imported" print this time, it's cached
print("greetings" in sys.modules) # True
print(sys.modules["greetings"] is greetings) # True -> the exact same object, not a fresh copy
True True
.pyc files — the first time a module is imported, Python compiles its source to
bytecode and saves that compiled form next to it, in a __pycache__ subfolder, so the
next process to import it can skip recompiling:
demo_dir/
├── greetings.py
└── __pycache__/
└── greetings.cpython-312.pyc <- compiled bytecode, tagged with the Python version
Python checks the source file's modification time (or a hash) against the .pyc — if
greetings.py hasn't changed, the cached bytecode is reused; if it has, Python
recompiles and overwrites the .pyc automatically. This only applies to imported
modules — a script run directly (python script.py) is compiled in memory each time and
does not get a __pycache__ entry for itself.
cache_dir = os.path.join(demo_dir, "__pycache__")
print(os.listdir(cache_dir)) # ['greetings.cpython-312.pyc'] -> version number matches your interpreter
['greetings.cpython-313.pyc']
reload() — since a plain import is a no-op once a module is cached (as seen
above), editing a module's source and importing it again does not pick up the change
in an already-running process. importlib.reload() forces a module to re-execute in
place, replacing its old namespace with the freshly-run one — mainly useful in
interactive sessions (a REPL, a notebook) to pick up edits without restarting:
import importlib
print(greetings.hello("Ada")) # Hello, Ada!
# edit the source on disk, simulating someone changing the module's code
with open(os.path.join(demo_dir, "greetings.py"), "a") as f:
f.write('\ndef hello(name):\n return f"Hi there, {name}!"\n')
import greetings
print(greetings.hello("Ada")) # Hello, Ada! -> plain import is a no-op, still the old version
importlib.reload(greetings) # re-executes greetings.py in place
# greetings module is being imported
print(greetings.hello("Ada")) # Hi there, Ada! -> now picks up the change
Hello, Ada! Hello, Ada! greetings module is being imported Hi there, Ada!
Common mistake — reload() only affects lookups made after it runs. Any object
already created from the old version of a class keeps behaving like the old class;
only new objects, created after the reload, get the updated definition:
with open(os.path.join(demo_dir, "shape.py"), "w") as f:
f.write("class Circle:\n def area(self, r):\n return 'old formula'\n")
import shape
old_circle = shape.Circle()
print(old_circle.area(2)) # old formula
with open(os.path.join(demo_dir, "shape.py"), "w") as f:
f.write("class Circle:\n def area(self, r):\n return 3.14159 * r * r\n")
importlib.reload(shape)
print(old_circle.area(2)) # old formula -> this instance is still tied to the OLD class
new_circle = shape.Circle()
print(new_circle.area(2)) # 12.56636 -> a NEW instance uses the reloaded class
old formula old formula 12.56636
There's a sharper version of this same surprise: reload() doesn't just leave old
instances behaving oddly — it replaces the class object itself, so an old instance
stops being recognized as an instance of the "same" class at all:
print(isinstance(old_circle, shape.Circle)) # False -> shape.Circle now refers to a brand-new class object
print(type(old_circle) is shape.Circle) # False, same reason
print(isinstance(new_circle, shape.Circle)) # True -> new_circle was built from that reloaded class
False False True
9.3 Packages and Namespaces
A package is a directory of modules with an __init__.py file — the file is what
tells Python "this directory is importable as a single unit," and it runs once, the
first time the package is imported (much like a module's own top-level code):
shapes/ <- a package (has __init__.py)
├── __init__.py <- runs once, on first import of "shapes"
├── circle.py <- shapes.circle
├── square.py <- shapes.square
└── solids/ <- a sub-package (its own __init__.py)
├── __init__.py
└── sphere.py <- shapes.solids.sphere
Build exactly this layout and import from it:
pkg_dir = tempfile.mkdtemp()
sys.path.insert(0, pkg_dir)
os.makedirs(os.path.join(pkg_dir, "shapes", "solids"))
with open(os.path.join(pkg_dir, "shapes", "__init__.py"), "w") as f:
f.write('"""shapes package."""\nprint("shapes package initialized")\n__all__ = ["circle"]\n')
with open(os.path.join(pkg_dir, "shapes", "circle.py"), "w") as f:
f.write("def area(r):\n return 3.14159 * r * r\n")
with open(os.path.join(pkg_dir, "shapes", "square.py"), "w") as f:
f.write("def area(side):\n return side * side\n")
with open(os.path.join(pkg_dir, "shapes", "solids", "__init__.py"), "w") as f:
f.write("")
import shapes
print(shapes.__all__) # shapes package initialized / ['circle']
from shapes import circle
print(circle.area(2)) # 12.56636
shapes package initialized ['circle'] 12.56636
Relative imports — inside a package, a module can import a sibling using a leading
dot instead of spelling out the full package path: . means "this same package," ..
means "one level up." They only work inside a package — never in a standalone script:
with open(os.path.join(pkg_dir, "shapes", "solids", "sphere.py"), "w") as f:
f.write(
"from ..circle import area as circle_area # .. -> up one level, into 'shapes'\n"
"\n"
"def volume(r):\n"
" return (4 / 3) * circle_area(r) * r\n"
)
from shapes.solids import sphere
print(sphere.volume(2)) # 33.51029333333333
33.51029333333333
__init__.py controls from package import * via __all__ — a list of the names
that wildcard import is allowed to bring in. shapes/__init__.py set __all__ =
["circle"] above, so square is deliberately left out, even though it's right there in
the package:
before = set(dir())
from shapes import *
after = set(dir())
print("circle" in after) # True -> listed in __all__
print("square" in after) # False -> not listed, so * skipped it
True False
Namespace packages — __init__.py is actually optional. A directory with no
__init__.py at all still works as an importable package; it just doesn't get to run
any package-level setup code, since there's no file to run:
ns_dir = tempfile.mkdtemp()
sys.path.insert(0, ns_dir)
os.makedirs(os.path.join(ns_dir, "plugins")) # note: no __init__.py in "plugins" at all
with open(os.path.join(ns_dir, "plugins", "tool_a.py"), "w") as f:
f.write('NAME = "tool_a"\n')
import plugins.tool_a as tool_a
print(tool_a.NAME) # tool_a -> imported fine, no __init__.py needed
tool_a
9.4 Scripts versus Modules and __main__
The exact same file can be run (python script.py) or imported
(import script) — what changes is __name__ (9.1): it's "__main__" only when the
file is the one executed directly, and the module's own name otherwise. The
if __name__ == "__main__": guard uses that to separate "define these things" (always
runs) from "now actually do something with them" (only when run directly, not when
someone else imports this file just to reuse a function from it):
import subprocess
script_dir = tempfile.mkdtemp()
script_path = os.path.join(script_dir, "greet.py")
with open(script_path, "w") as f:
f.write(
"def hello(name):\n"
' return f"Hello, {name}!"\n'
"\n"
'print("module __name__ is:", __name__)\n'
"\n"
'if __name__ == "__main__":\n'
' print(hello("World")) # only runs when executed directly\n'
)
run_directly = subprocess.run(["python3", script_path], capture_output=True, text=True)
print(run_directly.stdout)
# module __name__ is: __main__
# Hello, World!
run_imported = subprocess.run(["python3", "-c", "import greet"],
capture_output=True, text=True, cwd=script_dir)
print(run_imported.stdout)
# module __name__ is: greet -> the guarded print never runs
module __name__ is: __main__ Hello, World! module __name__ is: greet
In practice — a data-processing script that's also a reusable module. A common
shape in data-engineering codebases: a script with functions useful enough that other
code wants to import them, but which should also do a specific job when run directly
(e.g. python transform.py input.csv) — the __main__ guard is exactly what lets one
file serve both purposes without the "do the actual work" part running on every import.
Common mistake — a script run directly is executed as sys.modules['__main__'],
never under its own filename. If that same file also gets imported elsewhere in the
same program (directly, or indirectly through some path-manipulation trick), Python
treats it as two unrelated modules and runs its top-level code — including every class
definition — twice, producing two distinct class objects for what looks like "the
same" class. isinstance() checks across that boundary then fail in a very confusing
way, because they're comparing an instance of one execution's class against the other
execution's class:
import runpy
dupe_path = os.path.join(script_dir, "shape2.py")
with open(dupe_path, "w") as f:
f.write("class Shape:\n pass\n")
# simulates "python shape2.py" -- the file executes as __main__, in its own fresh namespace
main_ns = runpy.run_path(dupe_path, run_name="__main__")
MainShape = main_ns["Shape"]
sys.path.insert(0, script_dir)
import shape2 # a separate, independent execution of the SAME file -- cached as 'shape2'
ImportedShape = shape2.Shape
instance = MainShape()
print(MainShape is ImportedShape) # False -> two distinct class objects
print(isinstance(instance, ImportedShape)) # False -> even though both are named "Shape"
False False
9.5 Command-Line Arguments
sys.argv is the raw list of arguments a script was launched with — argv[0] is
always the script's own path, everything after it came from the command line, all as
plain strings (no parsing, no type conversion, no --flag handling):
argv_script = os.path.join(script_dir, "cli_demo.py")
with open(argv_script, "w") as f:
f.write("import sys\nprint('raw argv:', sys.argv)\n")
result = subprocess.run(["python3", argv_script, "alpha", "beta", "--flag"],
capture_output=True, text=True)
print(result.stdout)
# raw argv: ['<path>/cli_demo.py', 'alpha', 'beta', '--flag']
raw argv: ['/tmp/tmpmse3ic9o/cli_demo.py', 'alpha', 'beta', '--flag']
argparse (stdlib) is the standard way to build an actual command-line interface on
top of sys.argv — named/positional arguments, type conversion, --help text, and
error messages for missing/invalid input, all generated for you:
argparse_script = os.path.join(script_dir, "cli_argparse.py")
with open(argparse_script, "w") as f:
f.write(
"import argparse\n"
"\n"
'parser = argparse.ArgumentParser(description="Greet someone.")\n'
'parser.add_argument("name", help="who to greet")\n'
'parser.add_argument("--shout", action="store_true", help="shout the greeting")\n'
"args = parser.parse_args()\n"
"\n"
'message = f"Hello, {args.name}!"\n'
"print(message.upper() if args.shout else message)\n"
)
plain = subprocess.run(["python3", argparse_script, "Ada"], capture_output=True, text=True)
print(plain.stdout) # Hello, Ada!
shouted = subprocess.run(["python3", argparse_script, "Ada", "--shout"], capture_output=True, text=True)
print(shouted.stdout) # HELLO, ADA!
Hello, Ada! HELLO, ADA!
Note: action="store_true" above defines a flag — an argument that
takes no value of its own. Its mere presence on the command line sets args.shout to
True; leaving it off leaves it False. Contrast that with "name", a normal
argument that requires a value to go with it.
In practice — this is the foundation click/typer build on. Plenty of internal
DevOps and data tools are built directly on argparse exactly like this; more
ergonomic third-party CLI libraries like click and typer are themselves layered on
top of the same underlying idea — declare the arguments/flags a command accepts, and
get parsing, --help text, and error messages generated automatically.
9.6 Standard Library Orientation
Python ships with a large "batteries included" standard library — no install needed,
just import. A few of the ones reached for constantly:
| Module | For |
|---|---|
os |
filesystem paths, environment variables, running processes |
sys |
interpreter internals — sys.argv (9.5), sys.path (9.2), sys.exit() |
math |
numeric functions (sqrt, floor, trig, constants like pi) |
datetime |
dates, times, durations, formatting/parsing timestamps |
json |
encoding/decoding JSON — the de facto data-interchange format |
tomllib |
reading TOML — the format pyproject.toml (9.7) itself is written in (read-only — writing TOML still needs a third-party package) |
collections |
extra container types (deque, Counter, defaultdict, namedtuple) |
itertools |
building blocks for efficient looping (chain, product, groupby) |
re |
regular expressions (15) |
pathlib |
an object-oriented alternative to os.path for filesystem paths |
random |
pseudo-random numbers, shuffling, sampling |
subprocess |
launching and talking to external processes (used throughout this section's demos) |
functools |
higher-order function helpers — reduce, lru_cache (6.8.2, 6.8.4), wraps |
argparse |
command-line argument parsing (9.5) |
logging |
structured, leveled logging (8.1–8.3) |
import json, pathlib
from collections import Counter
data = json.dumps({"a": 1, "b": 2})
print(data) # {"a": 1, "b": 2}
print(json.loads(data)) # {'a': 1, 'b': 2}
print(pathlib.Path("/tmp/example/file.txt").suffix) # .txt
print(Counter("mississippi")) # Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})
{"a": 1, "b": 2}
{'a': 1, 'b': 2}
.txt
Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})import tomllib
config_text = """
name = "demo"
version = "1.0"
[dependencies]
requests = ">=2.0"
"""
config = tomllib.loads(config_text) # tomllib.load(f) reads directly from an open binary file
print(config) # {'name': 'demo', 'version': '1.0', 'dependencies': {...}}
print(config["dependencies"]["requests"]) # >=2.0
{'name': 'demo', 'version': '1.0', 'dependencies': {'requests': '>=2.0'}}
>=2.09.7 Third-Party Packages and Environments
The standard library (9.6) covers a lot, but not everything — PyPI (the Python
Package Index) hosts third-party packages, installed with pip. Installing
globally makes every project on the machine share one set of package versions, which
breaks the moment two projects need different versions of the same dependency — the
standard fix is a virtual environment: an isolated, per-project install location.
python -m venv .venv create an isolated environment in ./.venv
source .venv/bin/activate activate it (Windows: .venv\Scripts\activate)
pip install requests installs into .venv only, not system-wide
pip freeze > requirements.txt record exact installed versions
pip install -r requirements.txt reproduce that exact environment elsewhere
deactivate leave the virtual environment
Once activated, python/pip inside that shell point at the environment's own
interpreter and its own separate site-packages — code can check which one it's
running in:
import sys
print(sys.prefix) # the interpreter currently in use
print(sys.base_prefix) # the underlying system/base interpreter
in_venv = sys.prefix != sys.base_prefix
print("inside a virtual environment" if in_venv else "NOT inside a virtual environment")
/tmp/claude-0/-home-claude/486c0d09-ed3e-5ead-aaa8-c4d85a824c07/scratchpad/pym/venv313 /usr inside a virtual environment
In practice — every CI/CD pipeline and Docker build does this. A Dockerfile or
CI config that runs pip install -r requirements.txt (or poetry install against a
lockfile) inside a fresh virtual environment is exactly this pattern, automated — it's
the mechanism that turns "works on my machine" into "works identically on every
machine," by pinning the exact package versions a project was built and tested against.
9.9 Additional Modules and Packages Concepts
Overflow bucket for this chapter — small or unclassified items that clearly belong to this domain but not yet to a specific section above.
Circular imports — module a importing b, while b also imports a. Whether
this works depends on what gets imported and when it's needed: importing the whole
module is usually fine (the name being looked up on it isn't needed until later, by
which point both modules have finished loading), but importing a specific name before
it exists yet fails immediately:
circ_dir = tempfile.mkdtemp()
sys.path.insert(0, circ_dir)
# the version that FAILS: each module needs a specific name from the other,
# immediately, before either has finished running
with open(os.path.join(circ_dir, "a.py"), "w") as f:
f.write("from b import from_b # needs from_b to exist RIGHT NOW\n\ndef from_a():\n return 'a calls ' + from_b()\n")
with open(os.path.join(circ_dir, "b.py"), "w") as f:
f.write("from a import from_a # a is only half-built at this point\n\ndef from_b():\n return 'b'\n")
failing = subprocess.run(["python3", "-c", "import a; print(a.from_a())"],
capture_output=True, text=True, cwd=circ_dir)
last_line = failing.stderr.strip().splitlines()[-1]
print(last_line.replace(os.path.join(circ_dir, "a.py"), "a.py"))
# ImportError: cannot import name 'from_a' from partially initialized module 'a'
# (most likely due to a circular import) (a.py)
ImportError: cannot import name 'from_a' from partially initialized module 'a' (most likely due to a circular import) (a.py)
The usual fix: import the module itself instead of pulling a specific name out of it
at the top. The attribute lookup (b.from_b()) then happens inside the function, at
call time — long after both modules have finished loading:
with open(os.path.join(circ_dir, "a.py"), "w") as f:
f.write("import b\n\ndef from_a():\n return 'a calls ' + b.from_b()\n")
with open(os.path.join(circ_dir, "b.py"), "w") as f:
f.write("import a # only needs the 'a' module object to exist, not any name on it yet\n\ndef from_b():\n return 'b'\n")
working = subprocess.run(["python3", "-c", "import a; print(a.from_a())"],
capture_output=True, text=True, cwd=circ_dir)
print(working.stdout.strip()) # a calls b
a calls b
Two other standard remedies, for when "import the module, not the name" doesn't fit:
- Deferred (local) import — move the
importfrom the top of the file into the function that actually needs it. By the time that function runs, both modules have long since finished loading, so the circularity is never even hit at import time. - Extract the shared piece — pull whatever both modules need out into a third
module that neither depends on the other for, and have both
aandbimport from it instead of from each other. This is usually the better long-term fix, since it also tends to meanaandbweren't cleanly separated responsibilities to begin with.
In practice — a very real, common pain point in Django projects. Django's own
models.py files frequently need to reference each other (an Order model with a
foreign key to a Customer model defined in a different app, which in turn references
something back), and hitting exactly this circular-import error while wiring up model
relationships is a rite of passage for anyone who's built a Django app of non-trivial
size.
Running a module with python -m — instead of pointing at a file path
(python pkg/runner.py), -m names a module by its import path
(python -m pkg.runner) and runs it through the normal import machinery. That
difference matters for relative imports (9.3): they need to know their package context,
which only exists when the module was located via -m, not when its file was executed
directly:
m_dir = tempfile.mkdtemp()
os.makedirs(os.path.join(m_dir, "pkgtest"))
with open(os.path.join(m_dir, "pkgtest", "__init__.py"), "w") as f:
f.write("")
with open(os.path.join(m_dir, "pkgtest", "helper.py"), "w") as f:
f.write("VALUE = 42\n")
with open(os.path.join(m_dir, "pkgtest", "runner.py"), "w") as f:
f.write("from .helper import VALUE # relative import - needs package context\nprint('VALUE:', VALUE)\n")
via_module = subprocess.run(["python3", "-m", "pkgtest.runner"], capture_output=True, text=True, cwd=m_dir)
print(via_module.stdout.strip()) # VALUE: 42
via_path = subprocess.run(["python3", os.path.join("pkgtest", "runner.py")],
capture_output=True, text=True, cwd=m_dir)
print(via_path.stderr.strip().splitlines()[-1])
# ImportError: attempted relative import with no known parent package
VALUE: 42 ImportError: attempted relative import with no known parent package