Technology handbook
Python Packages — Standard Library & Ecosystem
153 Python modules and packages, from the standard library through the web, data, testing and AI stacks.
- Topics
- 153
- Sections
- 14
Text & data types
re
Regular expressions. Compile once with re.compile when a pattern is reused, and prefer named groups.
- Regular expressions search, split and substitute text.
- Compiling once pays off when a pattern is reused.
- Catastrophic backtracking can hang on a crafted input.
string
Constants such as ascii_letters, plus Template for simple $name substitution.
- Constants and a template class support simple substitution.
- Template substitution is safer than format for untrusted patterns.
- It is far less capable than f-strings for ordinary formatting.
textwrap
Wraps, fills, indents and dedents text — dedent is what makes triple-quoted strings readable.
- Text is wrapped, filled, indented and dedented.
- dedent is what makes a triple-quoted block readable in code.
- It counts characters, not display width, so wide glyphs misalign.
difflib
Sequence comparison: unified diffs and get_close_matches for fuzzy lookup.
- Sequences are compared and human-readable diffs produced.
- Close-match lookup is a cheap spelling suggestion.
- Comparison is quadratic, so large inputs are slow.
unicodedata
Character properties and normalisation. NFC before comparing user-supplied text.
- Character properties and normalisation forms are exposed.
- Normalising is what makes two visually identical strings compare equal.
- Comparing unnormalised text fails in ways that are invisible on screen.
struct
Packs and unpacks C structs to and from bytes, for binary file and wire formats.
- Bytes are packed and unpacked to a declared layout.
- It is how a binary protocol or file header is read.
- Endianness and alignment must be stated or the platform decides.
codecs
Explicit encoder and decoder access for stream-level transcoding.
- Text is encoded and decoded between byte representations.
- It is the lower layer under file and stream encoding.
- An error handler choice silently changes the data on failure.
dataclasses
Generates __init__, __repr__ and __eq__ from annotations. frozen=True gives a hashable value object.
- A decorator generates init, repr and comparison from fields.
- The class declaration becomes the schema.
- A mutable default needs a factory rather than a literal.
enum
Enum, IntEnum, StrEnum and Flag for named constants that type checkers understand.
- Named constants are grouped into a type.
- Members are singletons, so identity comparison works.
- Adding a member can break exhaustive matching elsewhere.
typing
Optional, Literal, Protocol, TypedDict, TypeVar and the generic machinery.
- Type annotations are expressed for the checker.
- Protocols describe a shape without requiring inheritance.
- Nothing is enforced at runtime by the annotations themselves.
Collections & algorithms
collections
defaultdict, Counter, deque, OrderedDict and namedtuple — the data structures stdlib got right.
- Specialised containers cover counting, defaults, ordering and queues.
- Counter and defaultdict remove the commonest dictionary boilerplate.
- deque appends and pops at both ends in constant time.
itertools
Lazy combinatorics and iteration: chain, groupby, product, islice, accumulate.
- Iterator building blocks compose lazily.
- Nothing is materialised, so infinite sequences are usable.
- Consuming an iterator twice yields nothing the second time.
functools
cache, lru_cache, partial, reduce, wraps and singledispatch.
- Caching, partial application and reduction are provided.
- lru_cache memoises a pure function in one line.
- Caching on a method keeps instances alive indefinitely.
operator
Function forms of the operators, so sorted(key=itemgetter(1)) beats a lambda.
- Operators are available as plain functions.
- itemgetter and attrgetter are faster than equivalent lambdas.
- It is less readable than a lambda for anything unusual.
heapq
A binary heap over a plain list — priority queues and nlargest/nsmallest.
- A list is maintained as a binary heap.
- It gives the smallest item in constant time with cheap insertion.
- It is a min-heap, so a max-heap needs negated keys.
bisect
Binary search and insertion into a sorted list, in O(log n).
- A sorted sequence is searched and maintained.
- Binary search is what makes an ordered insert cheap.
- The sequence must already be sorted or results are wrong.
array
Compact typed numeric arrays when a list of ints is too much memory and NumPy is too much dependency.
- A compact sequence of one numeric type is stored.
- It uses far less memory than a list of the same numbers.
- NumPy is better for anything beyond simple storage.
copy
copy and deepcopy, and __deepcopy__ for types that need to control it.
- Shallow and deep copies of objects are produced.
- A shallow copy shares nested objects with the original.
- Deep copying is expensive and can follow more than expected.
graphlib
TopologicalSorter for dependency ordering without hand-rolling the traversal.
- A dependency graph is sorted topologically.
- It is the standard answer for ordering tasks by dependency.
- A cycle raises rather than producing a partial order.
statistics
Mean, median, mode, variance and quantiles without pulling in NumPy.
- Mean, median, variance and related measures are computed.
- It needs no third-party dependency for basic statistics.
- It is far slower than NumPy on large datasets.
Files & paths
pathlib
Object-oriented paths. Path.read_text(), glob and / composition replace most of os.path.
- Filesystem paths are objects with methods.
- The slash operator joins correctly on every platform.
- Some older APIs still expect strings.
os.path
The older string-based path functions, still what you meet in existing code.
- The older string-based path manipulation functions.
- Most existing code still uses it, so it must be read.
- pathlib is the preferred interface for new code.
shutil
High-level file operations: copy trees, move, remove trees, archive and disk_usage.
- Files and directory trees are copied, moved and removed.
- It provides the high-level operations os lacks.
- Recursive removal is unforgiving of a wrong path.
glob
Shell-style wildcard matching, with ** recursion when recursive=True.
- Paths are matched against shell-style wildcards.
- Recursive matching finds files across a tree in one call.
- Ordering is not guaranteed, so results need sorting.
fnmatch
Glob-style matching against a name you already hold, rather than against the filesystem.
- Names are matched against shell-style patterns.
- It matches strings rather than touching the filesystem.
- Its case sensitivity follows the platform.
tempfile
Secure temporary files and directories that clean themselves up as context managers.
- Temporary files and directories are created safely.
- Safe creation avoids the race a predictable name invites.
- Without a context manager the file may not be removed.
io
StringIO and BytesIO in-memory file objects, and the buffering layers under open.
- Text, binary and in-memory streams share one interface.
- In-memory buffers let code that expects a file be tested.
- Wrapping a binary stream as text introduces an encoding.
mmap
Memory-maps a file so it can be read and sliced like bytes without loading it.
- A file is mapped into memory and accessed as bytes.
- Random access to a huge file avoids reading it whole.
- The mapping must be closed or the file stays locked.
filecmp
Compares files and directory trees, shallow or by content.
- Files and directory trees are compared.
- A shallow comparison uses metadata and is very fast.
- A shallow match is not proof the contents are identical.
OS & processes
os
Environment, process ids, file descriptors and directory operations.
- Processes, environment and filesystem are reached directly.
- It is the portable layer over operating system calls.
- Behaviour still differs per platform in places.
sys
argv, stdin/stdout, exit, the module path and interpreter details.
- Interpreter state, arguments and streams are accessed.
- It is where the module path and exit behaviour live.
- Mutating the module path at runtime makes imports unpredictable.
subprocess
Runs external commands. Pass a list of arguments; shell=True is an injection risk.
- External commands are run with their input and output captured.
- Passing arguments as a list is what avoids shell injection.
- A filled output pipe deadlocks a process that is not read.
signal
Handles SIGTERM and SIGINT — where graceful shutdown is implemented.
- Operating system signals are handled.
- Handling termination is what allows a clean shutdown.
- Handlers run between bytecodes, not instantly.
platform
Identifies the OS, machine and Python build, for diagnostics and conditional behaviour.
- The running system and interpreter are identified.
- It is how platform-specific behaviour is selected.
- Branching on it is a smell when a portable API exists.
getpass
Reads a password without echoing it, and reports the current user.
- A password is read without echoing it.
- Suppressing the echo is the entire purpose.
- It fails where there is no real terminal.
argparse
Declarative CLI parsing with subcommands, types and generated help.
- Command-line arguments are declared, parsed and validated.
- Declaring arguments generates the help text.
- Complex subcommand trees become hard to follow.
logging
Hierarchical loggers, handlers, formatters and filters. Configure at the edge, log everywhere else.
- Messages are emitted through configurable loggers and handlers.
- Levels and handlers are what separate noise from signal.
- Configuring it from a library imposes choices on the application.
contextlib
contextmanager, suppress, ExitStack and closing for resources that must be released.
- Context managers are created and composed.
- A generator becomes a context manager with one decorator.
- suppress can hide an error that mattered.
atexit
Registers cleanup to run on normal interpreter shutdown.
- Functions run when the interpreter exits normally.
- It is the place for final cleanup and flushing.
- It does not run on a hard kill or a crash.
Concurrency & async
asyncio
The event loop, tasks, gather, TaskGroup, queues and timeouts.
- A coroutine runs only via
asyncio.run(), an await, or asyncio.create_task(). - Calling a coroutine function merely returns a coroutine object and warns if never awaited.
- The loop keeps only weak references to tasks, so an unreferenced task can be collected mid-run.
threading
Threads, locks and events. Useful for I/O; the GIL limits CPU-bound gains.
- Threads share memory inside one interpreter process.
- I/O-bound work still benefits, because I/O releases the global lock.
asyncio.to_thread is the documented way to move a blocking call off the loop.
multiprocessing
Separate processes with real parallelism, plus pools, queues and shared memory.
- Separate processes each get their own interpreter.
- That is how CPU-bound work uses more than one core.
- Arguments are pickled, so unpicklable objects cannot cross.
concurrent.futures
ThreadPoolExecutor and ProcessPoolExecutor behind one submit/map interface.
- Thread and process pools share one submission interface.
- Swapping the executor is what moves work between threads and processes.
- An exception surfaces only when the future's result is retrieved.
queue
Thread-safe FIFO, LIFO and priority queues for producer/consumer work.
- Thread-safe queues pass work between threads.
- The locking that makes it safe is already handled.
- An unbounded queue hides a producer outpacing its consumer.
selectors
High-level I/O multiplexing, the layer asyncio's selector loop is built on.
- Many sockets are monitored for readiness.
- It is the portable layer beneath the event loop.
- Writing directly against it is rarely worth the effort now.
anyio
Structured concurrency that runs on asyncio or Trio, with cancel scopes and task groups.
- One API runs on both asyncio and Trio.
- Structured concurrency is its organising idea.
- Libraries may still assume one backend.
trio
An async library built around nurseries, where no task can be silently orphaned.
- Structured concurrency organises tasks into nurseries.
- A nursery guarantees child tasks are awaited and errors surface.
- Much of the ecosystem targets asyncio instead.
uvloop
A libuv-backed event loop that drops into asyncio for a substantial speed-up.
- A faster event loop replaces the default.
- It is a drop-in change with a measurable gain.
- It does not run on every platform.
Networking & web
urllib.request
The stdlib HTTP client. Workable, but httpx or requests is nicer for anything real.
- HTTP requests are made without a dependency.
- It is available everywhere the standard library is.
- Its API is clumsy compared with the third-party clients.
http.client
The low-level HTTP protocol implementation under urllib.
- The low-level HTTP protocol client.
- It is what higher-level clients are built upon.
- Connection reuse and retries are yours to manage.
http.server
A development-only static server. Never expose it.
- A simple HTTP server for local use.
- Serving a directory is a one-line command.
- It is explicitly not for production.
socket
BSD sockets — TCP, UDP and Unix domain.
- Network sockets are created and used directly.
- It is the substrate under every network protocol here.
- Message framing is the application's responsibility.
ssl
TLS wrapping for sockets, certificate verification and context configuration.
- Sockets are wrapped in TLS.
- Certificate and hostname verification are the security.
- Disabling verification silently removes the protection.
ipaddress
Parses and reasons about IPv4/IPv6 addresses and networks.
- Addresses and networks are parsed and compared.
- Containment checks replace error-prone string parsing.
- Mixing address families raises rather than comparing.
email
Constructs and parses MIME messages, including attachments.
- Messages are constructed and parsed, including attachments.
- It handles the encoding rules correspondence actually requires.
- Its layered API is notoriously awkward.
smtplib
Sends mail over SMTP, with TLS and authentication.
- Mail is sent over SMTP.
- It needs no dependency to deliver a message.
- Deliverability depends on domain configuration, not the library.
requests
The classic synchronous HTTP client: sessions, retries via adapters and simple JSON handling.
- A widely used synchronous HTTP client.
- Its interface is the reason it became a de facto standard.
- It is synchronous, so it blocks an async loop.
httpx
A requests-compatible client that also does async, HTTP/2 and connection pooling.
- An HTTP client with sync and async interfaces.
- One library covers both styles and HTTP/2.
- Its defaults differ from requests in ways that surprise.
aiohttp
An async HTTP client and server built directly on asyncio.
- An asynchronous HTTP client and server.
- The session is what pools connections and must be reused.
- A session per request defeats the pooling entirely.
websockets
An asyncio WebSocket client and server with a clean async-iterator API.
- Websocket clients and servers are built on asyncio.
- It is a focused, standards-compliant implementation.
- Reconnection logic is the application's responsibility.
Dates & math
datetime
Dates, times and deltas. Prefer timezone-aware objects; naive ones cause the hard bugs.
- Dates, times and durations are represented.
- A naive datetime has no zone, which is where bugs start.
- Arithmetic yields timedelta rather than a number.
zoneinfo
IANA timezones from the system database, which retires the pytz dependency.
- IANA time zones are available in the standard library.
- Real zone data is what makes conversions correct.
- On some systems the data package must be installed.
time
Clocks: monotonic for durations, perf_counter for benchmarks, time for wall clock.
- Clocks, sleeping and epoch conversions are provided.
- The monotonic clock is the right one for measuring durations.
- Wall clock time can jump backwards.
calendar
Calendar arithmetic, weekday names and leap-year handling.
- Calendar data and month arithmetic are provided.
- It answers questions about months and weekdays directly.
- It works in dates, not in instants.
math
Scalar mathematics, including isclose, comb, gcd and hypot.
- Floating-point mathematical functions are provided.
- isclose is how floats should actually be compared.
- Its functions are scalar, not vectorised.
decimal
Exact decimal arithmetic with controllable precision and rounding. Money belongs here, not in float.
- Exact decimal arithmetic with configurable precision.
- It is the correct type for money.
- It is markedly slower than floating point.
fractions
Exact rational numbers, for when repeated rounding would accumulate.
- Exact rational arithmetic is provided.
- No precision is lost the way it is with floats.
- Denominators grow, and with them the cost.
random
Pseudo-random numbers and sampling. Deterministic from a seed, and unsuitable for secrets.
- Pseudo-random numbers and selections are generated.
- Seeding makes a sequence reproducible for tests.
- It is predictable and must not be used for secrets.
Testing & typing
unittest
The stdlib xUnit framework, plus unittest.mock which everyone uses regardless of runner.
- The standard library test framework with class-based tests.
- It ships with Python, so it needs no dependency.
- Its assertion style is more verbose than pytest's.
doctest
Runs the examples in docstrings as tests, keeping documentation honest.
- Examples in docstrings are executed as tests.
- It keeps documentation honest by running it.
- It is brittle against any change in output formatting.
pytest
The de facto runner: plain assert, fixtures, parametrisation and a deep plugin ecosystem.
- Tests are plain functions using plain assert statements.
- Fixtures supply setup by parameter name.
- Parametrisation turns one test into a table of cases.
pytest-asyncio
Runs async test functions and manages the event loop for them.
- Asynchronous tests run under pytest.
- It supplies the loop the coroutine needs.
- The mode must be configured or async tests are skipped silently.
hypothesis
Property-based testing that generates inputs and shrinks any failure to a minimal case.
- Inputs are generated to find counterexamples.
- It explores cases nobody would have written by hand.
- A failure must be minimised to be understandable.
coverage.py
Line and branch coverage measurement, with reporting and thresholds.
- Executed lines and branches are measured.
- Coverage shows what was never run at all.
- High coverage is not evidence of good assertions.
tox / nox
Runs the suite across several interpreters and dependency sets in isolated environments.
- Tests run across environments and interpreter versions.
- It catches what only breaks on another version.
- Each environment is built and installed, which is slow.
mypy
The established static type checker, configurable from strict down to gradual.
- Static type checking verifies annotations.
- It catches a class of errors before anything runs.
- Strictness is configurable, so adoption can be gradual.
pyright / basedpyright
A fast type checker with excellent inference, and the engine behind Pylance.
- A fast type checker with strong editor integration.
- Its inference is more aggressive than mypy's.
- The two disagree, so a codebase should pick one.
ty
Astral's type checker, built for speed on large codebases.
- A very fast type checker written in Rust.
- Speed makes checking viable on every keystroke.
- It is young, so coverage of edge cases is still growing.
pydantic
Runtime validation and settings from type hints, with a Rust core in v2.
- Data is validated and coerced from annotated models.
- Models are the schema and the runtime check together.
- Version two changed behaviour substantially from version one.
attrs
Class boilerplate generation with validators and converters; the ancestor of dataclasses.
- Classes are generated from declared attributes.
- It predates dataclasses and remains more capable.
- It overlaps dataclasses, so a project should choose one.
freezegun
Freezes time in tests so date-dependent behaviour is deterministic.
- The clock is frozen or moved for tests.
- Controlling time is what makes a time-dependent test deterministic.
- It patches globally, which can surprise unrelated code.
factory-boy
Builds realistic test objects from declarative factories.
- Test objects are built from declarative factories.
- Factories keep fixture setup readable as models grow.
- Overly clever factories hide what a test depends on.
responses / respx
Mock HTTP at the client layer for requests and httpx respectively.
- Outbound HTTP is intercepted in tests.
- Mocking at the HTTP layer keeps the client code real.
- Each targets a different client library.
Web frameworks
fastapi
Async APIs typed with Pydantic, generating OpenAPI and docs from the signatures.
- APIs are built from type annotations with validation and docs.
- Annotations produce validation and OpenAPI together.
- Its concurrency model requires care with blocking calls.
django
Batteries included: ORM, migrations, admin, auth, forms and templates.
- A batteries-included framework with ORM, admin and auth.
- The admin interface alone justifies it for many projects.
- Its conventions are pervasive and hard to opt out of.
flask
A minimal WSGI framework with blueprints and a large extension ecosystem.
- A minimal framework that stays out of the way.
- Small surface area makes it quick to learn.
- Everything beyond routing is an extension you choose.
starlette
The ASGI toolkit FastAPI is built on — routing, middleware, websockets and background tasks.
- A lightweight ASGI toolkit.
- It is the foundation FastAPI is built on.
- It is deliberately low level.
litestar
An opinionated, fully typed ASGI framework with DI and layered configuration.
- An ASGI framework with dependency injection built in.
- Injection and layered configuration are its distinguishing features.
- Its community is smaller than the established alternatives.
uvicorn
The ASGI server, usually run behind gunicorn workers in production.
- An ASGI server for asynchronous applications.
- It is the usual way to run modern Python web apps.
- A single worker uses a single core.
gunicorn
A WSGI process manager, and the supervisor for uvicorn workers.
- A process manager that supervises worker processes.
- It is how multiple cores are used in production.
- It needs an ASGI worker class for async frameworks.
djangorestframework
Serializers, viewsets, routers and permissions on top of Django.
- REST APIs are built on top of Django.
- Serializers and viewsets remove most boilerplate.
- Its abstractions are deep and can obscure behaviour.
sqlalchemy
The Python ORM and Core query builder, with a fully typed 2.0 API.
- A SQL toolkit and object-relational mapper.
- The core and the ORM can be used independently.
- Version two changed the idiomatic query style substantially.
alembic
Schema migrations for SQLAlchemy, autogenerated from model changes.
- Database migrations are generated and applied.
- Migrations are files, which makes schema change reviewable.
- Autogenerated migrations must be read before they are run.
celery
Distributed task queues with scheduling, retries and result backends.
- Distributed task queues run background work.
- It is the established choice for Python background jobs.
- It needs a broker and adds real operational weight.
redis-py
The Redis client, sync and async, with cluster support.
- The Redis client for Python.
- It covers caching, locks and pub/sub in one client.
- Connection pooling must be configured for concurrency.
psycopg
The Postgres adapter, version 3 with async support and server-side binding.
- The PostgreSQL adapter for Python.
- Version three supports both sync and async.
- Server-side cursors are needed for very large result sets.
jinja2
Template rendering with autoescaping, inheritance and macros.
- Templates render text and HTML from context data.
- Autoescaping is what prevents injection in HTML output.
- Autoescaping is not on by default in every configuration.
Data & scientific
numpy
N-dimensional arrays and vectorised maths. The foundation everything numeric builds on.
- Arrays hold uniform types and operate element-wise.
- Vectorised operations run in compiled code, which is the speed.
- A slice is a view, so writing through it changes the original.
pandas
DataFrames for tabular data: joins, grouping, reshaping and time series.
- Tabular data is loaded, reshaped, grouped and aggregated.
- The index is what alignment happens on, not row position.
- Chained assignment may write to a copy rather than the frame.
polars
A Rust-backed DataFrame library with a lazy engine and much lower memory use.
- A fast dataframe library with lazy evaluation.
- The lazy API optimises a whole query before executing.
- Its API differs enough from pandas to need relearning.
scipy
Optimisation, integration, interpolation, signal processing and statistics.
- Scientific algorithms build on NumPy arrays.
- It covers optimisation, integration, signal and statistics.
- Its surface is large and organised by subpackage.
matplotlib
The plotting workhorse, with full control over every element of a figure.
- Plots are built from figures and axes.
- The object-oriented interface scales where the stateful one does not.
- Headless rendering needs a non-interactive backend.
seaborn
Statistical plots over matplotlib, with sensible defaults.
- Statistical plots are produced with sensible defaults.
- It reduces a common chart to a single call.
- Customising beyond its defaults means dropping to matplotlib.
plotly
Interactive charts for notebooks and the web.
- Interactive charts render in the browser.
- Interactivity is what it offers over static plotting.
- Its output embeds data, so files grow quickly.
pyarrow
Arrow memory format and Parquet I/O — the interchange layer between data tools.
- Columnar data is represented and exchanged in memory.
- It is the interchange format between data tools.
- It is a heavy dependency for a small task.
duckdb
An in-process analytical database that queries DataFrames and Parquet files directly in SQL.
- An embedded analytical database queries files directly.
- It runs SQL over Parquet and CSV without loading them.
- It is analytical, not a transactional store.
openpyxl
Reads and writes modern Excel workbooks, formatting included.
- Excel workbooks are read and written.
- It handles the format most business data actually arrives in.
- Large workbooks are slow and memory-hungry.
jupyter
Notebooks for exploratory work, with the kernel protocol behind them.
- Code, output and prose live in an executable notebook.
- Interleaving is what makes exploration and explanation one artifact.
- Cells can run out of order, producing unreproducible state.
ML & AI
scikit-learn
Classical machine learning: pipelines, model selection and metrics with one consistent API.
- Classical machine learning with a consistent estimator API.
- One fit and predict interface covers every model.
- It is not designed for deep learning.
pytorch
Tensors, autograd and neural networks, with eager execution that makes debugging tractable.
- Tensors and neural networks with dynamic graphs.
- Dynamic graphs make debugging ordinary Python debugging.
- GPU and CUDA versions must match the install.
tensorflow
The graph-based deep learning framework, with Keras as its high-level API.
- A machine learning platform with a large deployment story.
- Its serving and mobile tooling are mature.
- Its API has changed substantially across major versions.
transformers
Pretrained models and pipelines for text, vision and audio from the Hugging Face Hub.
- Pretrained models are loaded and fine-tuned.
- A shared interface covers thousands of published models.
- Model downloads are large and cached on disk.
anthropic
The official SDK for the Claude API, with streaming, tool use and structured outputs.
- The official client for Claude models.
- Streaming, tools and structured output are supported directly.
- Retries and rate limits still need handling by the caller.
langchain
Composition primitives for LLM applications: chains, agents, retrievers and memory.
- Components for models, retrievers, tools and agents.
- It standardises the wiring between pieces.
- Its abstractions are worth understanding before depending on them.
llama-index
Indexing and retrieval for RAG over your own documents.
- Data is indexed and queried for retrieval-augmented generation.
- It focuses on the ingestion and indexing half.
- Its defaults hide choices that matter at scale.
sentence-transformers
Embedding models for semantic search and similarity.
- Sentence and document embeddings are produced.
- It makes semantic similarity available in a few lines.
- Embeddings from different models are not comparable.
onnxruntime
Runs exported models fast across CPU, GPU and mobile targets.
- Models run in a portable, optimised format.
- One format runs across languages and hardware.
- Not every operator converts cleanly.
Automation & scraping
playwright
Browser automation with auto-waiting, in both sync and async flavours.
- Browsers are automated for testing and scraping.
- Auto-waiting removes most timing flakiness.
- Browser binaries are installed separately from the package.
selenium
The long-established WebDriver automation library.
- Browsers are driven through the WebDriver standard.
- Its standardisation and language coverage are the strengths.
- Waiting is explicit, which is the usual source of flakiness.
beautifulsoup4
Forgiving HTML parsing and traversal for pages that are not well-formed.
- HTML is parsed and navigated forgivingly.
- It copes with malformed markup that a strict parser rejects.
- Its speed depends on the parser it is given.
lxml
Fast XML and HTML parsing with full XPath support.
- XML and HTML are parsed quickly with XPath support.
- XPath is what makes complex extraction concise.
- It is a compiled dependency, which complicates installation.
scrapy
A crawling framework with scheduling, pipelines, throttling and retries.
- A framework for large-scale crawling.
- Scheduling, throttling and pipelines come built in.
- It is heavy for fetching a handful of pages.
pillow
Image opening, conversion, resizing and drawing.
- Images are opened, converted and transformed.
- It is the standard image library for Python.
- Decoding untrusted images has a history of vulnerabilities.
rich
Tables, progress bars, syntax highlighting and tracebacks in the terminal.
- Terminal output gains colour, tables and progress bars.
- It makes command-line output genuinely readable.
- Its markup can interfere with piped output.
typer
CLIs built from type hints, on top of Click.
- Command-line interfaces are built from type annotations.
- Annotations generate parsing and help text together.
- It is a layer over click, which shows through at the edges.
click
Composable command-line interfaces with decorators, groups and prompts.
- Composable command-line interfaces are built with decorators.
- Nested commands and groups compose cleanly.
- Its decorator stacking becomes dense quickly.
watchdog
Filesystem event monitoring, for rebuild and reload loops.
- Filesystem changes are observed and acted upon.
- It is the basis of most watch-and-reload tools.
- Editors produce several events for one logical save.
paramiko
SSH and SFTP from Python, for remote execution and file transfer.
- SSH and SFTP connections are made from Python.
- It automates remote operations without shelling out.
- Host key handling must be deliberate rather than disabled.
boto3
The AWS SDK, covering every service with paginators and waiters.
- AWS services are called from Python.
- It covers every service with a consistent interface.
- Credential resolution order is a frequent source of confusion.