URL parser: how a URL breaks into parts

The parts, one by one

Two details worth memorizing. First, the query starts at the first ? and runs to the first #; the fragment runs to the end of the string. Second, the port belongs to the authority section together with the host, but parsers usually hand you them separately, as they should: the port is transport plumbing, the host is identity.

Query strings: the fiddly part

Parameters look simple and behave subtly. Three conventions do most of the work:

• Encoding. Spaces cannot live in a URL, so they become %20 or a + depending on who did the encoding. Reserved characters (&, =, ?, #) inside a value get percent-encoded too, which is why q=a%26b is the value a&b and not two parameters.
• Repeated keys. size=41&size=42 is how lists travel. Some stacks read both values; some keep only the last. APIs that need arrays in one key often use a=1&a=2 or the PHP-style a[]=1; there is no single standard, so check what your target expects.
• Blank values. ?debug is a flag with an empty value. Parsers disagree on whether it survives, so flag-style parameters deserve an explicit test.

To see all of this on any real URL, paste it into our URL Parameter Inspector: every query parameter appears decoded, with the raw original beside it and a rebuilt URL that updates as you edit. It runs entirely in your browser.

Parse a URL in Python

Python ships a parser in the standard library, urllib.parse. No install, no dependencies:

from urllib.parse import urlparse, parse_qs

url = "https://shop.example.com:8443/catalog/shoes?color=red&size=41&size=42&page=2#reviews"
parts = urlparse(url)

parts.scheme    # 'https'
parts.netloc    # 'shop.example.com:8443'  (user info stays here if present)
parts.hostname  # 'shop.example.com'      (lowercased, port removed)
parts.port      # 8443                    (int, or None)
parts.path      # '/catalog/shoes'
parts.query     # 'color=red&size=41&size=42&page=2'  (still encoded)
parts.fragment  # 'reviews'               (no leading #)

The query string turns into usable data with parse_qs:

params = parse_qs(parts.query)
# {'color': ['red'], 'size': ['41', '42'], 'page': ['2']}

params['size']   # ['41', '42']   repeated keys become lists
params['missing']  # KeyError: check with .get() instead

Three behaviors bite everyone once. parse_qs drops blank values: parse_qs("debug=&page=2") returns {'page': ['2']}, and only keep_blank_values=True keeps debug. Repeated keys always fold into a list, so take params['size'][0] when you want one. And urlparse does not decode anything: parts.query is still percent-encoded, parse_qs is the step that decodes. The round trip back to text is urlencode, which encodes spaces as + by default:

from urllib.parse import urlencode
urlencode({"q": "red sneakers"})   # 'q=red+sneakers'

Parse a URL in JavaScript

JavaScript's URL class does the same job, built into every browser and Node:

const u = new URL("https://shop.example.com:8443/catalog/shoes?color=red&size=41&size=42&page=2#reviews");

u.protocol   // 'https:'
u.hostname   // 'shop.example.com'
u.port       // '8443'   (empty string when default)
u.pathname   // '/catalog/shoes'
u.search     // '?color=red&size=41&size=42&page=2'
u.hash       // '#reviews'  (includes the #)

The query string gets its own interface, searchParams, which decodes for you:

u.searchParams.get("color")        // 'red'
u.searchParams.getAll("size")      // ['41', '42']
new URLSearchParams({q: "red sneakers"}).toString()  // 'q=red+sneakers'

Two differences from Python to keep straight. new URL() refuses relative input: new URL("example.com/catalog") throws, because it cannot know the scheme; pass a base as the second argument or use the fully qualified form. And unlike Python, searchParams keeps blank values, so ?debug= survives as an empty string.

Traps that cause most parsing bugs

• The missing scheme. Browsers hide https:// in the address bar, so people paste example.com/catalog into code that expects a full URL. Python's urlparse does not throw: it silently puts everything in path and leaves netloc empty, and the bug surfaces later. JavaScript's URL throws instead. Validate the scheme explicitly when input comes from humans.
• Encoded slashes. %2F inside a path segment is a slash that some servers decode early and others reject outright, so a path meant to carry an encoded slash can 404 depending on the stack in front of you. APIs that accept encoded slashes often ask for them double-encoded; read their docs, do not assume.
• Fragment versus path. Everything after # never reaches the server: the browser strips it before the request leaves the machine. Anything the server needs to know cannot live in the fragment, which is why single-page apps use history.pushState paths instead of # routes.
• Internationalized domains. A host with non-ASCII letters, like a German münchen, travels as punycode on the wire: münchen.example becomes xn--mnchen-3ya.example. Both forms can appear in logs and redirects; normalize with IDNA encoding before comparing hosts.