20.1 Why Data Validation and Parsing Matter
Data validation is checking that a piece of data actually has the shape a program expects (right types, required fields present, values within sensible ranges) before using it. Parsing is the related job of turning untyped input — JSON from an API request, a row from a config file, raw text from an LLM — into real, typed Python objects. Both matter for the same reason: any data arriving from outside the program (a user, a network call, a file) cannot be trusted to already match what the code assumes.
Common mistake — trusting a type hint to actually check anything. Type hints
(6.6.1) are documentation for humans and static-analysis tools (mypy) — Python itself
never looks at them at runtime. A function can be called with completely the wrong
type and nothing stops it:
def total_price(price: float, quantity: int) -> float:
return price * quantity
print(total_price(9.99, 3)) # 29.97 -> works as intended
print(total_price("9.99", 3)) # '9.999.999.99' -> a str hinted as a float, silently WRONG, no error
29.97 9.999.999.99
The obvious fix — check everything by hand — works, but doesn't scale. Every field
needs its own isinstance/range check, every nested object needs the same treatment
recursively, and the checks have to be kept in sync with the shape by hand forever:
def make_user_manual(name, age): # a plain function - every check written out by hand
if not isinstance(name, str):
raise TypeError("name must be a str")
if not isinstance(age, int):
raise TypeError("age must be an int")
if age < 0:
raise ValueError("age must be non-negative")
return {"name": name, "age": age}
print(make_user_manual("Ada", 28)) # {'name': 'Ada', 'age': 28}
try:
make_user_manual("Ada", -5)
except ValueError as e:
print("ValueError:", e) # age must be non-negative
{'name': 'Ada', 'age': 28}
ValueError: age must be non-negative20.2 What Pydantic Is
Pydantic is a third-party library (pip install pydantic, 9.7 — not stdlib) that
takes the type hints already being written anyway and makes them enforced at
runtime: define a schema once, as an ordinary-looking class, and every value that
enters through it is actually checked and converted to match.
At a glance, a Pydantic model looks exactly like a dataclass (10.9) — a class with
type-hinted attributes and no manual __init__. The difference is what happens on
construction:
from dataclasses import dataclass
from pydantic import BaseModel, ValidationError
@dataclass
class UserDC:
name: str
age: int
class UserModel(BaseModel): # looks almost identical to the dataclass above
name: str
age: int
dc = UserDC(name="Ada", age="not a number") # a dataclass never checks the hints
print(dc.age, type(dc.age)) # not a number <class 'str'> -> silently wrong
try:
UserModel(name="Ada", age="not a number") # pydantic DOES check
except ValidationError:
print("ValidationError raised - pydantic caught it")
not a number <class 'str'> ValidationError raised - pydantic caught it
ValidationError is a plain subclass of the built-in ValueError (7.4) — no special
except machinery is needed to catch it, and it slots directly into ordinary
try/except (7.2) alongside every other exception.
This chapter targets Pydantic v2. Pydantic went through a major rewrite between v1
and v2 (2023) that renamed several core APIs — a lot of tutorials, blog posts, and
Stack Overflow answers still written against v1 use names that raise AttributeError
(or silently don't exist) under the v2 install this chapter uses:
| v1 (older tutorials) | v2 (used throughout this chapter) |
|---|---|
.dict() |
.model_dump() (20.3) |
.json() |
.model_dump_json() (20.3) |
@validator |
@field_validator (20.4) |
class Config: |
model_config = {...} (20.5) |
pip show pydantic (9.7) confirms which major version is actually installed before
trusting either style of example.
20.3 Defining Models with BaseModel
Every Pydantic schema is a class that subclasses pydantic.BaseModel, with fields
declared exactly like class attributes with type hints. Creating an instance runs
validation immediately — construction is the validation step:
class User(BaseModel):
name: str
age: int
u = User(name="Ada", age=28)
print(u) # name='Ada' age=28
print(u.name, u.age) # Ada 28
try:
User(name="Ada", age="not a number")
except ValidationError as e:
print(e)
# 1 validation error for User
# age
# Input should be a valid integer, unable to parse string as an integer [type=int_parsing, ...]
name='Ada' age=28
Ada 28
1 validation error for User
age
Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='not a number', input_type=str]
For further information visit https://errors.pydantic.dev/2.13/v/int_parsing.errors() returns every failure, not just the first. The rest of this chapter
mostly reaches for e.errors()[0]["msg"] as a shorthand once the pattern is familiar,
but a real validation failure is a list — one entry per field that failed — and worth
seeing in full at least once. Two independently-broken fields in the same call produce
two separate entries:
try:
User(name=123, age="not a number") # TWO fields wrong at once
except ValidationError as e:
print(len(e.errors())) # 2 -> one entry per failing field, not just the first
for err in e.errors():
print(err["loc"], "->", err["msg"])
# ('name',) -> Input should be a valid string
# ('age',) -> Input should be a valid integer, unable to parse string as an integer
2
('name',) -> Input should be a valid string
('age',) -> Input should be a valid integer, unable to parse string as an integerValidation isn't just rejection — a value that's the wrong type but the right shape
gets coerced (parsed) into the declared type rather than being turned away, which is
exactly the "parsing" half of this chapter's scope. .model_dump()/.model_dump_json()
go the other direction, turning a model back into a plain dict/JSON string:
u2 = User(name="Ada", age="28") # a string that LOOKS like an int - pydantic parses it
print(u2.age, type(u2.age)) # 28 <class 'int'> -> coerced, not just accepted as-is
print(u2.model_dump()) # {'name': 'Ada', 'age': 28} -> back to a plain dict
print(u2.model_dump_json()) # '{"name":"Ada","age":28}' -> a JSON string
28 <class 'int'>
{'name': 'Ada', 'age': 28}
{"name":"Ada","age":28}20.4 Field Constraints, Nested Models and Custom Validators
Field() attaches constraints beyond "the right type" — numeric bounds, a default
built fresh per instance (avoiding 6.2's mutable-default-argument gotcha), and more.
Optional fields work exactly as in plain type hints:
from pydantic import Field
from typing import Optional
class Product(BaseModel):
name: str
price: float = Field(gt=0) # must be greater than 0
description: Optional[str] = None # optional - defaults to None if omitted
tags: list[str] = Field(default_factory=list) # a fresh empty list per instance
p1 = Product(name="Mouse", price=19.99)
print(p1.description, p1.tags) # None []
try:
Product(name="Bad", price=-5)
except ValidationError as e:
print(e.errors()[0]["msg"]) # Input should be greater than 0
None [] Input should be greater than 0
In practice — this is a signup-form API endpoint. Validating that a price is
positive, a password meets a minimum length, or an age falls in a plausible range is
exactly what a real /signup or /products endpoint does before the data ever
reaches the database — Field() constraints are how that validation gets written
once, on the schema, instead of scattered across every handler that touches the data.
Nested models validate recursively — a field typed as another BaseModel accepts a
plain dict and converts it, and a validation failure reports the exact nested path
that failed:
class Address(BaseModel):
city: str
zip_code: str
class Customer(BaseModel):
name: str
address: Address # a nested model - validated recursively
orders: list[str] # a list field
c = Customer(
name="Ada",
address={"city": "London", "zip_code": "SW1A"}, # a plain dict - parsed into an Address
orders=["order-1", "order-2"],
)
print(c.address.city) # London -> a real Address instance, not a dict
print(type(c.address)) # <class '__main__.Address'>
try:
Customer(name="Ada", address={"city": "London"}, orders=[]) # missing zip_code
except ValidationError as e:
print(e.errors()[0]["loc"]) # ('address', 'zip_code') -> the exact nested field that failed
London
<class '__main__.Address'>
('address', 'zip_code')@field_validator covers rules Field() can't express — arbitrary logic, and even
transforming the value on the way in:
from pydantic import field_validator
class SignupForm(BaseModel):
username: str
@field_validator("username")
@classmethod
def username_must_be_alnum(cls, value):
if not value.isalnum():
raise ValueError("username must be alphanumeric")
return value.lower() # validators can also TRANSFORM the value
form = SignupForm(username="Ada123")
print(form.username) # ada123 -> lowercased by the validator
try:
SignupForm(username="not valid!")
except ValidationError as e:
print(e.errors()[0]["msg"]) # Value error, username must be alphanumeric
ada123 Value error, username must be alphanumeric
@model_validator covers the other half of validation @field_validator can't
reach — a rule that depends on more than one field at once. Once a model has
interdependent fields (a date range, a password/confirmation pair, a min/max bound),
single-field validators can't see the other field to compare against;
@model_validator(mode="after") runs once every field has already been individually
validated, with self giving access to the whole model:
from pydantic import model_validator
from datetime import date
class Booking(BaseModel):
start_date: date
end_date: date
@model_validator(mode="after") # runs AFTER every individual field is already validated
def check_dates(self):
if self.start_date >= self.end_date:
raise ValueError("start_date must be before end_date")
return self
b = Booking(start_date="2026-01-01", end_date="2026-01-10")
print(b.start_date, b.end_date) # 2026-01-01 2026-01-10
try:
Booking(start_date="2026-01-10", end_date="2026-01-01") # end before start
except ValidationError as e:
print(e.errors()[0]["msg"]) # Value error, start_date must be before end_date
2026-01-01 2026-01-10 Value error, start_date must be before end_date
20.5 Parsing External Data and Common Usage Patterns
The pattern that shows up everywhere Pydantic is used: .model_validate() parses a
plain dict (or any dict-like object), and .model_validate_json() parses a raw
JSON string directly — both are how external data (a config file, an HTTP request
body, an LLM's reply) enters a program as a validated model instead of an untrusted
blob:
class Config(BaseModel):
debug: bool
max_retries: int
raw_json = '{"debug": "true", "max_retries": "3"}' # e.g. loaded from a config file or API response
config = Config.model_validate_json(raw_json) # parses JSON text directly
print(config.debug, type(config.debug)) # True <class 'bool'>
print(config.max_retries, type(config.max_retries)) # 3 <class 'int'>
raw_dict = {"debug": 0, "max_retries": 5}
config2 = Config.model_validate(raw_dict) # parses a plain dict instead
print(config2.debug) # False -> 0 coerced to a bool
True <class 'bool'> 3 <class 'int'> False
In practice — validating a webhook payload. An incoming webhook from Stripe,
GitHub, or a payment provider is untrusted external JSON arriving over HTTP — precisely
the case .model_validate_json() exists for. A real webhook handler runs the request
body through a BaseModel like Config above before trusting a single field of it,
rejecting anything malformed before it can reach business logic.
model_config switches on model-wide behaviour — "frozen": True makes instances
immutable after creation, useful for value objects that should never change once
validated:
class Point(BaseModel):
model_config = {"frozen": True} # instances become immutable after creation
x: int
y: int
point = Point(x=1, y=2)
try:
point.x = 99
except ValidationError as e:
print(e.errors()[0]["type"]) # frozen_instance
frozen_instance
20.6 Pydantic in API Development
Web frameworks like FastAPI use a BaseModel directly as a request/response
schema: the incoming JSON body is validated against it automatically, a mismatch
becomes an HTTP 422 response with the exact field errors, and the same model doubles
as the source for auto-generated API documentation (OpenAPI/Swagger) — none of that
written by hand. The mechanism underneath is exactly .model_validate() (20.5)
catching ValidationError and turning it into a structured response, reproducible with
plain Pydantic and no server at all:
class CreateItemRequest(BaseModel): # the exact shape of the JSON body an API endpoint expects
name: str
price: float
def handle_request(body: dict): # stands in for what a web framework does per request
try:
item = CreateItemRequest.model_validate(body)
except ValidationError as e:
return {"status": 422, "errors": e.errors()} # exactly what FastAPI returns automatically
return {"status": 200, "item": item.model_dump()}
print(handle_request({"name": "Mouse", "price": 19.99}))
# {'status': 200, 'item': {'name': 'Mouse', 'price': 19.99}}
print(handle_request({"name": "Mouse"})) # missing "price"
# {'status': 422, 'errors': [{'type': 'missing', 'loc': ('price',), 'msg': 'Field required', ...}]}
{'status': 200, 'item': {'name': 'Mouse', 'price': 19.99}}
{'status': 422, 'errors': [{'type': 'missing', 'loc': ('price',), 'msg': 'Field required', 'input': {'name': 'Mouse'}, 'url': 'https://errors.pydantic.dev/2.13/v/missing'}]}What FastAPI adds on top is purely the wiring — reading the actual HTTP body and
calling this same validation automatically. fastapi/uvicorn aren't installed here
(a real server needs a running process and an open port, out of scope for a notebook
cell), so this is shown for reference rather than executed:
from fastapi import FastAPI
app = FastAPI()
@app.post("/items")
def create_item(item: CreateItemRequest): # FastAPI validates the JSON body against this model
return {"received": item.model_dump()} # invalid bodies never reach this line - FastAPI
# already replied with a 422 automatically
20.7 Pydantic for LLM Output Enforcement
An LLM's raw reply is just text — exactly the kind of untrusted, unstructured input
20.1 opened with. The modern pattern for getting structured, reliable output out of
one (used by OpenAI's structured outputs/function calling, LangChain, and libraries
like instructor) is built entirely on what's already in this chapter: describe the
wanted shape as a BaseModel, hand the model's JSON Schema to the LLM as part of
the prompt/API call, then run whatever text comes back through
.model_validate_json() (20.5). .model_json_schema() generates that schema directly
from the class — no separate schema to hand-write or keep in sync:
import json
class MovieReview(BaseModel): # the exact shape we want an LLM to reply in
title: str
rating: int = Field(ge=1, le=5)
summary: str
print(json.dumps(MovieReview.model_json_schema(), indent=2))
# {
# "properties": {
# "title": {"title": "Title", "type": "string"},
# "rating": {"maximum": 5, "minimum": 1, "title": "Rating", "type": "integer"},
# "summary": {"title": "Summary", "type": "string"}
# },
# "required": ["title", "rating", "summary"],
# "title": "MovieReview",
# "type": "object"
# }
{
"properties": {
"title": {
"title": "Title",
"type": "string"
},
"rating": {
"maximum": 5,
"minimum": 1,
"title": "Rating",
"type": "integer"
},
"summary": {
"title": "Summary",
"type": "string"
}
},
"required": [
"title",
"rating",
"summary"
],
"title": "MovieReview",
"type": "object"
}Parsing the reply back is identical to parsing any other external JSON — and a reply
that breaks the contract (here, a rating outside 1–5) is caught the same way any other
invalid input would be, which is exactly what an application needs in order to retry or
re-prompt instead of silently accepting garbage:
llm_output_good = '{"title": "Inception", "rating": 5, "summary": "A mind-bending heist thriller."}'
review = MovieReview.model_validate_json(llm_output_good)
print(review.title, review.rating) # Inception 5
llm_output_bad = '{"title": "Inception", "rating": 11, "summary": "Great movie"}' # rating out of range
try:
MovieReview.model_validate_json(llm_output_bad)
except ValidationError as e:
print(e.errors()[0]["msg"]) # Input should be less than or equal to 5
Inception 5 Input should be less than or equal to 5
The natural production follow-up to catching a ValidationError here isn't just to
give up — it's to feed e.errors() straight back to the LLM as a repair prompt,
describing exactly what was wrong, and re-request a corrected reply:
def call_llm_stand_in(prompt):
# stands in for a real LLM API call - a canned (invalid) first reply
return '{"title": "Inception", "rating": 11, "summary": "Great movie"}'
def call_llm_repair_stand_in(repair_prompt):
# stands in for the LLM's reply once told exactly what was wrong with its first attempt
return '{"title": "Inception", "rating": 5, "summary": "Great movie"}'
raw = call_llm_stand_in("Review the movie Inception.")
try:
review = MovieReview.model_validate_json(raw)
except ValidationError as e:
repair_prompt = (
f"Your last reply failed validation with these errors: {e.errors()}\n"
f"Please reply again, fixing those fields."
)
raw_fixed = call_llm_repair_stand_in(repair_prompt) # re-request from the LLM
review = MovieReview.model_validate_json(raw_fixed) # validate the repaired reply
print(review.title, review.rating) # Inception 5 -> succeeded after one repair round-trip
Inception 5
In practice, a library that talks to a real LLM API bundles both halves of this
together — sending the schema and calling .model_validate_json() on the reply — so
the model class is the entire integration. The openai package isn't installed here
(this would make a real network call), so shown for reference only:
from openai import OpenAI
client = OpenAI()
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[{"role": "user", "content": "Review the movie Inception."}],
response_format=MovieReview, # the schema from 20.7, sent straight to the API
)
review = completion.choices[0].message.parsed # already a validated MovieReview instance