Charset and Character Encoding in Java

A character encoding defines how text characters are represented as bytes and how those bytes are converted back into characters. Java provides the Charset API and StandardCharsets constants to handle text reliably when reading and writing data.

A Java String represents text as a sequence of characters, while files and network streams typically contain bytes. A charset specifies the mapping between characters and their byte representations. UTF-8 is a widely used encoding that supports characters from many writing systems and represents each Unicode code point using one to four bytes.

Java provides the `java.nio.charset.Charset` class to represent character encodings. The `StandardCharsets` class contains constants such as `UTF_8`, `UTF_16`, `US_ASCII`, and `ISO_8859_1`. Using these constants avoids relying on platform-dependent default encodings.

When converting text into bytes, Java uses encoding. When converting bytes into text, it uses decoding. Both operations must use compatible charsets to preserve the original text. If the wrong charset is selected, characters may become corrupted or appear as replacement characters.

For file operations, `Files.readString()` and `Files.writeString()` accept a `Charset` argument. For byte arrays, `String.getBytes(Charset)` converts text into bytes, while `new String(byte[], Charset)` reconstructs text. Explicitly specifying UTF-8 makes the intended encoding clear and improves portability.

Java's charset API also supports error handling through `CharsetEncoder` and `CharsetDecoder`. Their configuration can determine whether malformed or unmappable input is replaced or reported as an error. This is useful when processing data that must meet strict encoding requirements.

Key points

• A charset defines how characters map to bytes and how bytes are decoded into text.
• UTF-8 supports Unicode and uses between one and four bytes per Unicode code point.
• Use `StandardCharsets.UTF_8` instead of relying on the platform's default charset.
• Use the same compatible charset when encoding and decoding data.
• Use `CharsetEncoder` and `CharsetDecoder` when explicit error handling is required.

Interview questions

What is the difference between UTF-8 and UTF-16?

UTF-8 and UTF-16 are both Unicode encoding formats. UTF-8 uses one to four bytes for each Unicode code point, while UTF-16 uses one or two 16-bit code units. UTF-8 is often convenient for text interchange because ASCII characters retain their single-byte representation. UTF-16 can be useful in environments that work extensively with 16-bit code units, but neither encoding guarantees one unit per character.

What is a character encoding, and why does Java need it?

A character encoding defines how text is represented as bytes. Java Strings represent text, but files and network protocols commonly exchange bytes. Encoding converts text into bytes, and decoding converts bytes back into text. Explicitly choosing a charset prevents inconsistent behavior across different systems.

How do you convert a Java String into UTF-8 bytes and back?

Use `String.getBytes(StandardCharsets.UTF_8)` to encode a String. To decode the resulting bytes, pass them to the String constructor with the same charset. This preserves text when the bytes are valid UTF-8. Using an explicit charset avoids depending on the machine's default encoding.

Why should Java backend developers avoid the default charset?

The default charset may vary depending on the Java runtime and environment. Code that relies on it can behave differently when deployed on another machine. This can lead to corrupted file contents or incorrect network payloads. Using `StandardCharsets.UTF_8` makes the expected encoding explicit and predictable.

How does Java handle malformed or unmappable character sequences?

Convenience methods for converting between Strings and byte arrays generally use replacement behavior when an input cannot be represented or decoded. For stricter validation, Java provides `CharsetEncoder` and `CharsetDecoder`. Their `onMalformedInput()` and `onUnmappableCharacter()` methods allow applications to choose whether to report, ignore, or replace problematic input. Reporting errors is useful when silently changing data would be unacceptable.

How would you ensure consistent character encoding when reading and writing files?

Use Java's `Files` methods with an explicitly selected charset, such as `StandardCharsets.UTF_8`. Specify the same encoding when reading and writing unless the file format defines another one. For external files, validate assumptions about their encoding rather than blindly treating every file as UTF-8. When strict correctness is necessary, configure a decoder to report malformed input.