Open and read URLs
An application needs to retrieve a resource over a supported URL scheme.
Python · Module
urllib
urllib is a standard-library package that groups modules for opening, parsing, and otherwise working with URLs. Its modules also cover request errors and robots.txt parsing.
How it works
The urllib package groups standard-library modules for URL work. Use urllib.request to open URLs, urllib.parse to split, combine, quote, or encode URL data, urllib.error for request-related exceptions, and urllib.robotparser for robots.txt files.
Syntax
import urllib
import urllib.request
import urllib.error
import urllib.parse
import urllib.robotparserKey points
- urllib is a package rather than one single URL-handling operation.
- Use urllib.request when an application needs to open or read a URL.
- Handle exceptions raised by URL-opening operations through urllib.error.
- Use urllib.parse to split and combine URL components and perform URL quoting.
- Use urllib.robotparser when interpreting robots.txt files.
Example: Build a URL with query parameters
from urllib.parse import urlencode, urlsplit, urlunsplit
parts = urlsplit("https://example.com/search")
query = urlencode({"q": "python", "page": 2})
url = urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
print(url)Parses a base URL, encodes query parameters, and combines the components into a complete URL.
When to use it
Manipulate URL structure
Code needs to build, split, resolve, or quote URL values.
Parse crawler directives
A crawler or client needs to inspect robots.txt rules.
Handle URL request errors
Network code needs to distinguish request failures from normal results.
Interview questions
QWhat is urllib, and which modules does it contain?
urllib is a package containing focused modules for URL operations. Its documented modules include urllib.request for opening and reading URLs, urllib.error for exceptions from urllib.request, urllib.parse for URL parsing, and urllib.robotparser for robots.txt files.
Follow-up: Which module should be used to construct a URL from components? urllib.parse, using its parsing, quoting, or combining functions.
QWhat is urllib.parse used for?
urllib.parse provides a standard interface for breaking URLs into components, combining components into a URL, and resolving a relative URL against a base URL. It also provides URL quoting functionality.
Follow-up: What does urlsplit return? It returns a five-item named tuple, SplitResult or SplitResultBytes, containing the URL's split components.
QHow does urllib.request.urlopen work?
urllib.request supplies functions and classes for opening URLs, particularly HTTP URLs, and supports concerns such as authentication, redirects, and cookies. Its urlopen function accepts a URL string or a Request object.
Follow-up: What can data mean in urlopen? It specifies additional data sent to the server; if supplied, it must be an appropriate object rather than None.
Related
urllib.requesturllib.errorurllib.parseurllib.robotparser