Code point, code unit, byte
Three different counts. They disagree for most of the world's text, and most bugs here come from treating them as one.
- A code point is the number Unicode assigns a character. The rocket emoji is U+1F680, 128,640 in decimal. Code points run to U+10FFFF.
- A code unit is the fixed-size piece an encoding is built from. A JavaScript string is UTF-16, so any code point above U+FFFF is stored as two, a surrogate pair. One rocket emoji has length 2, and
charCodeAt(0)returns 55,357. - A byte is what a file or socket carries. UTF-8 uses one byte to U+007F, two to U+07FF, three to U+FFFF and four beyond. The rocket is one code point, two UTF-16 code units, four UTF-8 bytes:
f0 9f 9a 80.
Why the obvious implementation is wrong
Walking the characters and taking charCodeAt of each is correct for ASCII and wrong for everything else, because it returns UTF-16 code units rather than bytes. Feed it an emoji and you get d83d de80, two numbers larger than any byte, each a surrogate half. Paste that into a decoder and nothing comes back. This tool converts through TextEncoder and TextDecoder, assuming nothing about how many bytes a character takes.
Splitting bytes at an arbitrary position is the same mistake a layer down. Cutting UTF-8 to fit a 255-byte column can land mid-character and leave a lead byte with no continuation, which is where U+FFFD comes from.
Common problems
- An odd number of hex digits means one was lost in the copy. Reported rather than padded, since either guess changes your data.
- A group larger than 255 means unseparated decimal, or the wrong base. Reading
72 105as hex asks for a byte worth 261. - Not valid UTF-8, with a single byte between 0x80 and 0xFF, means the bytes came from Latin-1 or Windows-1252: an e-acute is 0xE9 there and 0xC3 0xA9 in UTF-8. A leading
0xEF 0xBB 0xBFis a byte order mark, invisible in the decoded text and a common reason two identical-looking files do not compare equal. - Another converter's output containing d83d or dc00 is that tool emitting UTF-16 code units and calling them bytes. 0xD800 to 0xDFFF are surrogates, never bytes, and text converted that way does not survive a round trip.
Frequently asked questions
Is this ASCII or UTF-8?
UTF-8, which contains ASCII: the first 128 code points are identical byte for byte.
Can I convert binary back to text?
Yes, in every base, and the input can be messy: spaces, commas and newlines all separate groups, and prefixes are stripped. Binary and hex are fixed width, so the grouping is thrown away and the digits re-split from the left, which makes 0100 1000 and 01001000 the same input. Decimal and octal cannot work that way, since an unpadded 12 145 is not 12145: with separators each group is one byte, and with none the digits split into threes only when the count divides by three, and are read as a single number when it does not.
How is this different from Base64?
Base64 packs three bytes into four printable characters for transport: compact, not readable. A base writes each byte as its own group, from a single decimal digit up to eight binary ones, with every byte legible. Use the Base64 encoder to move data, including files, which this tool does not take.
Why does one emoji produce four bytes?
Because UTF-8 encodes most emoji in four bytes, and many in more than one code point on top of that. A flag is two regional indicators, and a family emoji is several people joined by zero-width joiners. The byte count is the honest answer; the character count depends on what you mean by a character.