Gradient AtlasML Materials

Data science roadmap › Python for Data Science › Chapter 16

File Handling

Persisting and reading data on disk.

16.1 File Fundamentals

Why file handling. A variable lives only in the running process's memory (RAM) — the instant the program ends (or crashes), everything it held is gone. File handling is how a program persists data beyond its own lifetime, by asking the operating system to read and write it to disk instead:

Storage Lifetime Example
In-memory (a variable) only while the program is running a list, a dict
Temporary storage until the OS/user cleans it up; often explicit the system temp directory, a cache
Permanent storage (a file) survives the program ending and the machine rebooting a .txt, .csv, .json file on disk

Python's built-in open() is the entry point for all of this — it hands back a file object connected to a real file on disk, through which the program reads or writes data.

Text vs. binary — every file is really just a sequence of bytes on disk; the difference is what those bytes are interpreted as:

Kind Interpreted as Typical use
Text characters, via a text encoding (default: UTF-8) .txt, .py, .csv, .json — human-readable
Binary raw bytes, no encoding/decoding at all images, executables, pickled data

Encoding is exactly the boundary between the two — a str only becomes bytes (what actually gets written to disk) by going through one:

In [1]3 lines
message = "hi"
print(message)                      # hi -> just a str in memory
print(message.encode("utf-8"))   # b'hi' -> the actual bytes that would be written to disk

hi
b'hi'

16.2 Text File Open Modes

open(path, mode)'s second argument controls both what is allowed (read/write) and where it starts:

Mode Meaning
"r" read an existing file (the default if mode is omitted); error if missing
"w" write — creates the file if missing, truncates (erases) it if it exists
"a" append — creates the file if missing, writes are added after existing content
"x" exclusive creation — creates a new file; errors if it already exists
"r+" read and write on an existing file, without truncating it
"w+" write and read, but truncates first, exactly like "w"
"a+" append and read

A trailing "t" (text) is the implicit default for every mode above — "rt" means exactly the same thing as "r". Reading, writing, and appending, in order:

The examples below use tempfile.mkdtemp() to create a fresh temporary directory, so running this notebook doesn't scatter files around your filesystem; in your own code you'd normally pass a plain path (e.g. "notes.txt" or "/home/you/notes.txt") instead:

In [2]20 lines
import tempfile, os

work_dir = tempfile.mkdtemp()
path = os.path.join(work_dir, "notes.txt")

f = open(path, "w")   # 'w' - create (or truncate) for writing
f.write("first line\n")
f.close()

f = open(path, "r")    # 'r' - read an existing file (the default mode)
print(f.read())            # first line
f.close()

f = open(path, "a")   # 'a' - append, creating the file if it doesn't exist
f.write("second line\n")
f.close()

f = open(path, "r")
print(f.read())   # first line \n second line
f.close()

first line

first line
second line

In practice — every config-reading or report-writing script does this. A DevOps script reading a YAML/.env-style config file, or a nightly job writing out a report file, uses exactly this open/read-or-write/close shape — the only real question in practice is almost always which mode ("r", "w", "a") fits the task, since the mechanics are identical across every one of them.

"x" refuses to clobber an existing file, and "r+" writes into an existing file without erasing what was already there:

In [3]12 lines
try:
    open(path, "x")   # 'x' - exclusive creation, fails if the file already exists
except FileExistsError as e:
    print("FileExistsError:", e)   # [Errno 17] File exists: '...notes.txt'

f = open(path, "r+")   # 'r+' - read AND write on an existing file, no truncation
f.write("XXXX")            # overwrites starting at position 0, doesn't erase the rest
f.close()

f = open(path, "r")
print(f.read())   # XXXXt line \n second line \n -> only the first 4 characters were overwritten
f.close()

FileExistsError: [Errno 17] File exists: '/tmp/tmpc0oemt3i/notes.txt'
XXXXt line
second line

"w+" combines writing with reading back, and "rt" demonstrates the implicit "t":

In [4]11 lines
scratch_path = os.path.join(work_dir, "scratch.txt")

f = open(scratch_path, "w+")   # 'w+' - write AND read, but truncates the file first
f.write("hello")
f.seek(0)                                  # move back to the start to read what was just written
print(f.read())                            # hello
f.close()

f = open(scratch_path, "rt")   # 'rt' - the same as plain "r"; 't' (text) is the implicit default
print(f.read())                        # hello
f.close()

hello
hello

16.3 Binary File Open Modes

Every text mode from 16.2 has a binary counterpart — just append "b" instead of the implicit "t", and reads/writes deal in raw bytes instead of str:

Mode One-liner
"rb" read raw bytes from an existing file
"wb" write raw bytes, creating/truncating the file
"ab" append raw bytes to the end of the file
"xb" exclusive creation, in binary
"rb+" read and write raw bytes on an existing file, no truncation
"wb+" write and read raw bytes, truncating first
"ab+" append and read raw bytes
In [5]9 lines
bin_path = os.path.join(work_dir, "data.bin")

f = open(bin_path, "wb")     # 'wb' - write raw bytes, no text encoding involved
f.write(bytes([72, 105, 33]))   # writes those exact byte values directly
f.close()

f = open(bin_path, "rb")      # 'rb' - read raw bytes back
print(f.read())   # b'Hi!'
f.close()

b'Hi!'

16.4 The File Object

open() returns a file object carrying a handful of properties about how it was opened, plus state-query methods for what it currently allows:

Property / method Gives
.name the path (or descriptor) it was opened with
.mode the mode string it was opened with
.encoding the text encoding in use (text mode only)
.closed True once .close() has run, False while still open
.writable() whether writing is currently allowed
.readable() whether reading is currently allowed

.write(s) sends s to the file and returns how many characters were written:

In [6]15 lines
profile_path = os.path.join(work_dir, "profile.txt")
f = open(profile_path, "w", encoding="utf-8")

print(f.name == profile_path)   # True -> the exact path it was opened with
print(f.mode)                            # w
print(f.encoding)                       # utf-8
print(f.closed)                          # False -> still open
print(f.writable())                     # True
print(f.readable())                    # False -> opened write-only

n = f.write("Ada Lovelace\n")   # write() returns the number of characters written
print(n)                                     # 13

f.close()
print(f.closed)   # True -> now closed

True
w
utf-8
False
True
False
13
True

Giving open() a path. A relative path ("notes.txt") is resolved against the current working directory — wherever the process happens to be running from, not where the .py file lives — while an absolute path always names the same file regardless of the working directory. Building a path by hand with + and "/" breaks on Windows, whose separator is "\\" — os.path.join() (or pathlib's / operator, 9.6) picks whichever separator the running OS actually needs (1.1.5's platform independence, applied to paths):

In [7]8 lines
from pathlib import Path

joined = os.path.join(work_dir, "sub", "notes.txt")           # portable - right separator for the OS
pathlib_version = Path(work_dir) / "sub" / "notes.txt"      # pathlib's / operator does the same job
print(str(pathlib_version) == joined)   # True -> two ways to build the identical path

relative = "relative_notes.txt"   # resolved against the CURRENT WORKING DIRECTORY, not this file's location
print(os.path.abspath(relative) == os.path.join(os.getcwd(), relative))   # True

True
True

In practice — deployment scripts that run on more than one OS. A DevOps script built and tested on a developer's Mac/Windows machine but deployed to a Linux production server will silently break if it hardcodes a path with "/" or "\\" — os.path.join()/pathlib.Path are what make the exact same script work unmodified in both places.

Going deeper. For new code, prefer pathlib.Path over os.path — it wraps a path in an object with useful methods (.exists(), .suffix, .read_text(), ...) instead of loose functions that take and return plain strings. os.path is shown here because it's still extremely common in existing codebases, and reading it fluently matters even when it isn't your first choice for new work.

16.5 Reading Character Data from Text Files

Four ways to pull characters back out of a text file, from least to most structured:

Call Reads
f.read(n) exactly n characters (or fewer, if the file ends first)
f.read() everything left in the file, as one str
f.readline() one line at a time, including its trailing \n; "" once nothing's left
f.readlines() every remaining line, all at once, as a list of strings
In [8]18 lines
poem_path = os.path.join(work_dir, "poem.txt")
with open(poem_path, "w") as f:
    f.write("roses are red\nviolets are blue\n")

f = open(poem_path, "r")
print(f.read(5))    # roses -> read(n) reads exactly n CHARACTERS
print(f.read())        # ' are red\nviolets are blue\n' -> read() with no args reads the REST
f.close()

f = open(poem_path, "r")
print(f.readline())   # roses are red\n -> ONE line, including the trailing newline
print(f.readline())   # violets are blue\n
print(f.readline())   # "" -> empty string once there's nothing left
f.close()

f = open(poem_path, "r")
print(f.readlines())   # ['roses are red\n', 'violets are blue\n']
f.close()

roses
 are red
violets are blue

roses are red

violets are blue


['roses are red\n', 'violets are blue\n']

The idiomatic way to process a file line by line is to iterate over the file object directly — it's itself a lazy iterator (14.1, 14.2) that yields one line per next() call, instead of loading every line into memory at once like readlines() does:

In [9]6 lines
f = open(poem_path, "r")
for line in f:              # one line at a time, lazily
    print(line.strip())
f.close()
# roses are red
# violets are blue

roses are red
violets are blue

16.6 The with Statement

An open file holds an OS-level resource (a file descriptor) that needs to be released with .close() — but if an exception happens between open() and .close(), that .close() call is skipped entirely, leaking the resource. The concrete failure story: a program that opens many files without ever closing them eventually hits the OS's open-file-descriptor limit and crashes with OSError: [Errno 24] Too many open files. with open(...) as f: fixes this structurally: open()'s return value is a context manager — it implements __enter__/__exit__ (13.1) — and with guarantees __exit__ (which closes the file) runs when the block ends, no matter how it ends:

In [10]13 lines
log_path = os.path.join(work_dir, "log.txt")

with open(log_path, "w") as f:   # calls f.__enter__() on entry, f.__exit__() on exit
    f.write("with a context manager")
    print(f.closed)   # False -> still open INSIDE the block
print(f.closed)   # True -> closed automatically the instant the block ends

try:
    with open(log_path, "w") as f:
        f.write("partial")
        raise ValueError("something went wrong")
except ValueError:
    print(f.closed)   # True -> __exit__ ran and closed the file EVEN THOUGH an exception was raised

False
True
True

Writing your own context manager

with isn't limited to files — any object implementing __enter__(self) and __exit__(self, exc_type, exc_val, exc_tb) (13.1) can follow with. __enter__ runs on entry, and its return value becomes whatever as binds to. __exit__ always runs on exit, whether the block finished normally or raised an exception, and it receives that exception's type, value, and traceback as its three arguments (all None if nothing went wrong). Returning True from __exit__ tells with to suppress the exception instead of letting it propagate; returning None/False (the usual case) lets it propagate normally after cleanup runs:

In [11]16 lines
class ManagedFile:
    def __init__(self, path, mode):
        self.path = path
        self.mode = mode

    def __enter__(self):
        self.file = open(self.path, self.mode)
        return self.file   # this becomes "f" in "with ManagedFile(...) as f:"

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.file.close()
        return False   # False/None - don't suppress any exception that occurred

with ManagedFile(log_path, "w") as f:
    f.write("hand-written context manager")
print(f.closed)   # True -> __exit__ closed it, exactly like the built-in file's does

True

The class above is boilerplate for something conceptually simple: "run this setup code, then always run that cleanup code." contextlib.contextmanager turns a generator function into a context manager without writing a class at all — everything before a single yield runs as __enter__, the yielded value becomes what as binds to, and everything after yield runs as __exit__ (wrapped in try/finally so the cleanup still runs if the with block raises). This is the same send/close/yield from machinery from 14.6, repurposed to drive a with block instead of being called by hand:

In [12]13 lines
from contextlib import contextmanager

@contextmanager
def managed_file(path, mode):
    f = open(path, mode)
    try:
        yield f   # this becomes "f" in "with managed_file(...) as f:"
    finally:
        f.close()   # always runs when the block ends, even on an exception

with managed_file(log_path, "w") as f:
    f.write("contextlib.contextmanager version")
print(f.closed)   # True -> closed automatically, same guarantee as the built-in with

True

In practice — wrapping a database connection or a distributed lock. Production code commonly wraps things far more consequential than a file in exactly this pattern: a context manager that acquires a database connection or a distributed lock (e.g. via Redis) on entry and guarantees it's released on exit — even if the code inside the with block raises — so a crash never leaves a lock held or a connection leaked.

16.7 Other Structured File Formats

CSV, JSON, Excel spreadsheets, and PDFs all have real internal structure — rows and columns, sheets, pages — that plain .read()/.write() (16.5) knows nothing about; it only sees characters or bytes. Each format instead gets its own dedicated module that understands that structure:

Format Structure Python module Built-in?
CSV plain text; rows of delimiter-separated values csv yes (stdlib)
JSON plain text; nested key-value/array structure json (9.6) yes (stdlib)
Excel (.xlsx) a zip archive of XML; sheets, cells, formulas, styles openpyxl no — pip install openpyxl (9.7)
PDF binary; pages of positioned text, fonts, images pypdf, pdfplumber no — pip install pypdf (9.7)

csv is the simplest and ships with Python — csv.writer/csv.reader work row-by-row as lists, and csv.DictReader reads each row as a dict keyed by the header row instead:

In [13]22 lines
import csv

csv_path = os.path.join(work_dir, "people.csv")

with open(csv_path, "w", newline="") as f:   # newline="" - stops csv from adding blank lines itself
    writer = csv.writer(f)
    writer.writerow(["name", "age"])              # header row
    writer.writerow(["Ada", 28])
    writer.writerow(["Grace", 34])

with open(csv_path, "r", newline="") as f:
    for row in csv.reader(f):
        print(row)
# ['name', 'age']
# ['Ada', '28']
# ['Grace', '34']

with open(csv_path, "r", newline="") as f:
    for row in csv.DictReader(f):   # each row as a dict, keyed by the header
        print(row)
# {'name': 'Ada', 'age': '28'}
# {'name': 'Grace', 'age': '34'}

['name', 'age']
['Ada', '28']
['Grace', '34']
{'name': 'Ada', 'age': '28'}
{'name': 'Grace', 'age': '34'}

json ships with Python too (9.6 introduced the json module for its data-structure side; here's the file-handling side) and mirrors csv's shape: json.dump(obj, f) writes a Python object out to an open file as JSON text, and json.load(f) reads that text back into Python objects:

In [14]13 lines
import json

json_path = os.path.join(work_dir, "people.json")
people = [{"name": "Ada", "age": 28}, {"name": "Grace", "age": 34}]

with open(json_path, "w") as f:
    json.dump(people, f)   # writes the Python object out as JSON text

with open(json_path, "r") as f:
    loaded = json.load(f)   # reads the JSON text back into Python objects

print(loaded)                     # [{'name': 'Ada', 'age': 28}, {'name': 'Grace', 'age': 34}]
print(loaded == people)   # True -> round-tripped exactly

[{'name': 'Ada', 'age': 28}, {'name': 'Grace', 'age': 34}]
True

In practice — JSON is the default format for config and API payloads. Application config files (config.json), the request/response bodies of virtually every REST API, and data exchanged between microservices are almost always JSON — json.dump()/ json.load() (or the .model_dump_json()/.model_validate_json() pair from Pydantic, 20.3/20.5) are two of the most frequently used file-adjacent calls in real Python services.

Excel and PDF are both binary formats with no standard-library reader — the usual choice is a third-party package (pip install, 9.7). Their shape (once installed) — illustrative only: not run in this notebook, since no third-party dependency is installed here:

# Excel (.xlsx) - openpyxl
import openpyxl
wb = openpyxl.load_workbook("data.xlsx")   # a Workbook: one or more Sheets
sheet = wb.active                                     # or wb["SheetName"]
print(sheet["A1"].value)                            # read one cell
sheet["B2"] = "new value"                          # write one cell
wb.save("data.xlsx")

# PDF - pypdf
from pypdf import PdfReader
reader = PdfReader("document.pdf")
print(len(reader.pages))                            # number of pages
print(reader.pages[0].extract_text())        # the text on page 1

(Illustrative only, as noted above — not executed in this notebook.) Both libraries mirror the same core idea as csv/json: parse the file's real structure (sheets and cells; pages and their content) into Python objects, instead of handing back an undifferentiated blob of bytes.