Invisible Unicode characters: the complete field guide
Every piece of text has two layers: the one you read, and a layer of Unicode characters that render as nothing. They are real bytes, they survive every copy and paste, and they break search, dedupe and code. Some of them are load-bearing punctuation in real languages, which is why "delete everything invisible" is the wrong fix.
Most of the time you never notice the second layer. Then a search that should match doesn't, a form says a field is too long when it looks fine, a build fails on a line that looks identical to the working one, and you lose an hour to a character with no pixels.
This is a field guide to that layer: what lives there, why it exists, and when it is junk.
The main families
Zero-width characters. The classic is the zero-width space, U+200B, an invisible word-break hint. Its siblings are the zero-width non-joiner (U+200C) and zero-width joiner (U+200D), which control how neighboring characters connect. We wrote a whole post on the zero-width space, because it causes more mystery bugs than the rest combined.
Space impostors. Unicode has more than a dozen characters that look exactly like a space but are not one: the non-breaking space (U+00A0), the narrow no-break space (U+202F), thin and hair spaces, the em quad, the ideographic space. They break string matching, because a phrase typed with a normal space and the same phrase with a non-breaking one are different strings.
Directional marks. Left-to-right and right-to-left marks (U+200E, U+200F) and their embedding and isolate relatives steer how mixed-direction text lays out. Essential for Arabic and Hebrew, stray noise everywhere else. The right-to-left override (U+202E) is known for a nastier trick: it can make a filename display with its extension reversed, so an executable reads as a PDF. Security tools flag it for that reason.
Control and format characters. The C0 and C1 control blocks, the byte-order mark (U+FEFF), the word joiner (U+2060), invisible math operators, and the soft hyphen (U+00AD), which only becomes visible when a line happens to break at it.
Tag characters. A block at U+E0000 to U+E007F that mirrors ASCII invisibly. Its one legitimate modern use is inside flag emoji. Outside of that, a run of tag characters is invisible ASCII riding along with your text, which makes it an ideal place to hide a fingerprint.
The braille blank. U+2800 looks like a space but is technically a braille pattern with no dots raised. It slips past "remove whitespace" logic because it is not classified as whitespace, which is why it turns up in blank-looking usernames and empty-looking messages.
Where they come from
Almost never from typing. They arrive in text through:
- Word processors and CMSs that quietly insert non-breaking spaces and soft hyphens for layout.
- Websites that inject zero-width characters to defeat copy-paste scrapers, or to fingerprint which subscriber leaked a document. Adding an invisible, per-recipient pattern to text is a documented watermarking technique.
- AI assistants and chat interfaces, which have their own typographic habits. Some model outputs have carried unusual invisible characters; we dug into that story in Does ChatGPT watermark its text?
- PDFs and e-books, where extraction tools translate layout artifacts into spacing characters.
When invisible characters are legitimate
This part matters, because a cleaner that strips everything invisible corrupts real writing in a dozen languages and shatters emoji.
Zero-width spaces mark word boundaries in Thai, Khmer, Lao and Myanmar. Joiners shape Arabic and Indic scripts, and they assemble emoji: a family emoji is several people glued together with zero-width joiners. Directional marks keep Hebrew punctuation on the correct side. The non-breaking space is correct French typography before a colon.
The right behavior is contextual: remove what is junk for your language, preserve what carries meaning. Any tool that promises to "remove all invisible characters" without that distinction will eventually mangle something you cared about.
How to check your own text
If you suspect a string is carrying passengers, you have options: a hex dump, an editor that renders invisibles, or a paste into a Unicode inspector. We compared the practical ones in how to see hidden characters in any text. The short version for macOS is pbpaste | hexdump -C, where a zero-width space shows up as the bytes e2 80 8b.
Text cleaning is coming next to tdy.
tdy is a Mac menu-bar app that already tidies every link you copy. Removing the invisible layer, without breaking the languages and emoji that need it, is the next upgrade. One email when it ships, nothing else.
Join the launch list