Encoding

URL Encoding: Why Your Query Strings Break and How to Fix Them for Good

· 3 min read

Sooner or later every developer ships the bug: a search box works perfectly until someone types an ampersand, a plus sign, or a hash — and suddenly parameters vanish, phone numbers gain spaces, redirects go nowhere. The culprit is always the same: characters that mean something in URL syntax were sent as data without armor.

URLs are a tiny language (RFC 3986) where ? & = # / have grammatical roles. Percent-encoding is how data containing those characters travels through that language safely: each dangerous byte becomes % plus two hex digits. The bugs come from encoding the wrong scope, the wrong number of times, or the wrong dialect.

Reserved, unreserved, and context

RFC 3986 divides characters into unreserved — letters, digits, hyphen, underscore, period, tilde — which never need encoding, and reserved — : / ? # [ ] @ ! $ & ' ( ) * + , ; = — which are URL structure. The subtlety: whether a reserved character needs encoding depends on where it sits. A / in the path is a separator; a / inside a query value is data and must be %2F. A ? starts the query; a ? inside a value must be %3F.

This context-dependence is why 'just encode the URL' is not a coherent instruction. Encoding a complete URL turns its skeleton into data (https%3A%2F%2F…); not encoding values corrupts the parse. The only strategy that scales is component-wise construction: build the URL from parts, encoding each dynamic value as you insert it — which is precisely what URLSearchParams (JavaScript) and equivalent APIs do for you.

// Fragile: manual concatenation
const bad = base + "?q=" + query + "&lang=" + lang;

// Robust: component-wise
const u = new URL(base);
u.searchParams.set("q", query);    // encodes as needed
u.searchParams.set("lang", lang);
u.toString();

The space schism: %20 vs +

Two encodings for the space character coexist for historical reasons. RFC 3986 percent-encoding says %20, valid anywhere in a URL. HTML form encoding (application/x-www-form-urlencoded) — the format of submitted forms and, by convention, query strings — says +. Both are 'correct' in their dialect, and the collision produces one very specific bug family: a literal plus sign in data (phone numbers +66…, timezone offsets, Base64 values) gets decoded as a space by form-style parsers.

Symptoms are unmistakable once known: phone numbers arriving with leading spaces, Base64 tokens failing on 'invalid character space'. The rule: when a value must survive a query string, encode + as %2B; when decoding form-style data, translate + to space — and know which dialect each layer of your stack speaks.

Double encoding and other pathologies

Every encoding bug leaves a fingerprint in the data. Learn four signatures and you can diagnose from logs alone:

  • %2520 or %253D — double encoding: % itself was re-encoded to %25. One layer encoded a value that another layer encoded again. Decode twice to recover; remove the redundant layer. Double encoding also breaks signatures and cache keys.
  • Truncated values after & or # — a raw separator in unencoded data split the parse. The missing tail of the value is in the 'next' parameter.
  • †/ ภmojibake — not a percent-encoding bug at all: UTF-8 bytes were interpreted as Latin-1 upstream. Fix the charset, not the encoding.
  • %u0E01-style sequences — the non-standard escape() from ancient JavaScript; needs unescape(), and the producing code needs replacing with encodeURIComponent.

Unicode: percent-encoding works on bytes

Non-ASCII characters are UTF-8-encoded first, then each byte is percent-escaped: é → %C3%A9 (two bytes), Thai ก → %E0%B8%81 (three), many emoji → four. One visible character becoming up to twelve encoded characters explains both why encoded URLs balloon and why decoding with the wrong charset produces garbage. Practical limits follow: browsers and CDNs cap URL length (2–8 KB territory) — non-ASCII-heavy data in query strings hits those caps three times faster than ASCII.

In JavaScript specifically, remember the two functions have different jobs: encodeURIComponent escapes everything reserved (use it for values); encodeURI preserves URL structure (use it, rarely, to sanitize a whole URL). Using encodeURI on a value silently leaves & = + unescaped — a bug that passes every test until real data arrives.

Decode-side discipline

Reading URLs from logs or webhooks, apply the same rules in reverse: parse structure first (split on ? & =), then decode each value once. Decoding before splitting turns encoded separators back into live ones and corrupts the parse — the mirror image of the encoding bug. And treat decoded values as untrusted input: percent-encoding is transport armor, and attackers rely on filters that inspect only the encoded form. Decode, then sanitize, in that order.

← All guides