Jump to content

CESU-8: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous edit
Content deleted Content added
VisualWikitext
Correct encoding of supplementary characters
 
(37 intermediate revisions by 24 users not shown)
Line 1: Line 1:
{{Short description|Encoding scheme for Unicode}}
The '''Compatibility Encoding Scheme for UTF-16: 8-Bit''' ('''CESU-8''') is a variant of [[UTF-8]] that is described in [[Unicode]] Technical Report #26 [http://www.unicode.org/reports/tr26/]. A Unicode code point from the [[Basic Multilingual Plane]] (BMP), i.e. a code point in the range U+0000 to U+FFFF, is encoded in the same way as in UTF-8. A Unicode supplementary character, i.e. a code point in the range U+10000 to U+10FFFF, is first represented as a surrogate pair, like in [[UTF-16]], and then each surrogate code point is encoded in UTF-8. Therefore, CESU-8 needs six bytes (3 bytes per surrogate) for each Unicode supplementary character while UTF-8 needs only four. Each CESU-8 character code (1, 2, or 3 bytes) can be converted to exactly one UTF-16 code (2 bytes).
The '''Compatibility Encoding Scheme for UTF-16: 8-Bit''' ('''CESU-8''') is a variant of [[UTF-8]] that is described in [[Unicode]] Technical Report #26.<ref>{{Cite web |url=https://www.unicode.org/reports/tr26/ |title=Unicode Technical Report #26 - Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8) |last=McGowan |first=Rick |publisher=Unicode Consortium}}</ref> A Unicode code point from the [[Basic Multilingual Plane]] (BMP), i.e. a code point in the range U+0000 to U+FFFF, is encoded in the same way as in UTF-8. A Unicode supplementary character, i.e. a code point in the range U+10000 to U+10FFFF, is first represented as a surrogate pair, like in [[UTF-16]], and then each surrogate code point is encoded in UTF-8. Therefore, CESU-8 needs six bytes (3 bytes per surrogate) for each Unicode supplementary character while UTF-8 needs only four. Though not specified in the technical report, ''unpaired'' surrogates are also encoded as 3 bytes each, and CESU-8 is exactly the same as applying an older [[UCS-2]] to UTF-8 converter to UTF-16 data.


⚫
The encoding of Unicode non-BMP characters works out to <code>11101101 1010yyyy 10xxxxxx 11101101 1011xxxx 10xxxxxx</code> (yyyy represents the top five bits of the character minus one). The byte values 0xF0—0xF4 will not appear in CESU-8, as they start the 4-byte encodings used by UTF-8.
⚫
CESU-8 is not an official part of the Unicode Standard, because Unicode Technical Reports are informative documents only. It should be used exclusively for internal processing and never for external data exchange.


⚫
CESU-8 is not an official part of the Unicode Standard, because Unicode Technical Reports are informative documents only.<ref>{{Cite web |url=https://www.unicode.org/reports/about-reports.html#Types |title=About Unicode Technical Reports - Types of Unicode Technical Reports: UAX, UTS, UTR |publisher=Unicode Consortium}}</ref> It should be used exclusively for internal processing and never for external data exchange.
CESU-8 is similar to Java's [[UTF-8#Modified UTF-8|Modified UTF-8]] but does not have the special encoding of the NUL character (U+0000).


Supporting CESU-8 in [[HTML]] documents is prohibited by the [[W3C]]<ref>{{Cite web |url=https://www.w3.org/TR/html51/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5.1 Standard |publisher=W3C}}</ref><ref>{{Cite web |url=https://www.w3.org/TR/html5/syntax.html#character-encodings |title=8.2.2.3. Character encodings |website=HTML 5 Standard |publisher=W3C}}</ref> and [[WHATWG]]<ref>{{Cite web |url=https://html.spec.whatwg.org/multipage/parsing.html#character-encodings |title=12.2.3.3 Character encodings |website=HTML Living Standard |publisher=WHATWG}}</ref> HTML standards, as it would present a [[cross-site scripting]] vulnerability.<ref>{{Cite web |url=https://developer.mozilla.org/en-US/docs/Web/HTML/Element/meta |title=&lt;meta> - HTML |website=MDN Web Docs |publisher=Mozilla}}</ref>
The CESU-8 encoding form is used in the [[Oracle database]] software. Oracle's UTF8 character set (unfortunately, a misnomer), available since version 8.0 of the database, is actually CESU-8. The character set AL32UTF8, introduced in version 9.0, is UTF-8 compliant.


Java's [[UTF-8#Modified UTF-8|Modified UTF-8]] is CESU-8 with a special overlong encoding of the NUL character (U+0000) as the two-byte sequence <code>C0 80</code>.<ref>{{cite web |title=Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8 |url=https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8 |year=2015 |publisher=[[Oracle Corporation]] |access-date=2021-04-30}}</ref>
⚫
The encoding of unicode supplementary characters works out to <code>11101101 1010yyyy 10xxxxxx 11101101 1011xxxx 10xxxxxx</code> (yyyy represents the top five bits of the character minus one i.e. U+10**** becomes 1111, U+01**** becomes 0000, x represents the remaining bits of the character).{{Clarify|date=August 2010}}

The [[Oracle database]] uses CESU-8 for its "UTF8" character set. Standard UTF-8 can be obtained using the character set "[[AL32UTF8]]" (since Oracle version 9.0).


==Examples==
==Examples==


{| class="wikitable" border="1" cellpadding="5" cellspacing="0" style="font-size: 95%;"
{| class="wikitable"
|-----
! rowspan="3" | Encoding
! colspan="19" | Unicode code point
|-----
|-----
! Code point
| U+0045 || colspan="2" | U+0205 || colspan="12" | U+10400
| U+0045 || colspan="2" | U+0205 || colspan="12" | U+10400
|-----
|-----
! Character
| E || colspan="2" | ȅ || colspan="12" | &#x10400;
| E || colspan="2" | ȅ || colspan="12" | 𐐀
|-----
|-----
! UTF-8
! UTF-8
Line 44: Line 46:
| colspan="2" style="border-left: 1px #dddddd solid" | 80
| colspan="2" style="border-left: 1px #dddddd solid" | 80
|}
|}

==References==
{{Reflist}}


==External links==
==External links==
* [http://www.unicode.org/reports/tr26/ Unicode Technical Report #26]
* [https://www.unicode.org/reports/tr26/ Unicode Technical Report #26]
* [http://java.sun.com/j2se/1.5.0/docs/guide/jni/spec/types.html Modified UTF-8 overview]
* [https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#modified_utf_8_strings Modified UTF-8 definition]
* [http://demo.icu-project.org/icu-bin/convexp?conv=CESU-8 Graphical View of CESU-8 in ICU's Converter Explorer]
* [http://demo.icu-project.org/icu-bin/convexp?conv=CESU-8 Graphical View of CESU-8 in ICU's Converter Explorer]


Line 56: Line 61:
[[Category:Unicode Transformation Formats]]
[[Category:Unicode Transformation Formats]]
[[Category:Character encoding]]
[[Category:Character encoding]]

[[cs:CESU-8]]
[[de:CESU-8]]
[[fr:CESU-8]]

Latest revision as of 21:47, 17 April 2022

The Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8) is a variant of UTF-8 that is described in Unicode Technical Report #26.[1] A Unicode code point from the Basic Multilingual Plane (BMP), i.e. a code point in the range U+0000 to U+FFFF, is encoded in the same way as in UTF-8. A Unicode supplementary character, i.e. a code point in the range U+10000 to U+10FFFF, is first represented as a surrogate pair, like in UTF-16, and then each surrogate code point is encoded in UTF-8. Therefore, CESU-8 needs six bytes (3 bytes per surrogate) for each Unicode supplementary character while UTF-8 needs only four. Though not specified in the technical report, unpaired surrogates are also encoded as 3 bytes each, and CESU-8 is exactly the same as applying an older UCS-2 to UTF-8 converter to UTF-16 data.

The encoding of Unicode non-BMP characters works out to 11101101 1010yyyy 10xxxxxx 11101101 1011xxxx 10xxxxxx (yyyy represents the top five bits of the character minus one). The byte values 0xF0—0xF4 will not appear in CESU-8, as they start the 4-byte encodings used by UTF-8.

CESU-8 is not an official part of the Unicode Standard, because Unicode Technical Reports are informative documents only.[2] It should be used exclusively for internal processing and never for external data exchange.

Supporting CESU-8 in HTML documents is prohibited by the W3C[3][4] and WHATWG[5] HTML standards, as it would present a cross-site scripting vulnerability.[6]

Java's Modified UTF-8 is CESU-8 with a special overlong encoding of the NUL character (U+0000) as the two-byte sequence C0 80.[7]

The Oracle database uses CESU-8 for its "UTF8" character set. Standard UTF-8 can be obtained using the character set "AL32UTF8" (since Oracle version 9.0).

Examples

[edit]
Code point U+0045 U+0205 U+10400
Character E ȅ 𐐀
UTF-8 45 C8 85 F0 90 90 80
UTF-16 0045 0205 D801 DC00
CESU-8 45 C8 85 ED A0 81 ED B0 80

References

[edit]
  1. ^ McGowan, Rick. "Unicode Technical Report #26 - Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8)". Unicode Consortium.
  2. ^ "About Unicode Technical Reports - Types of Unicode Technical Reports: UAX, UTS, UTR". Unicode Consortium.
  3. ^ "8.2.2.3. Character encodings". HTML 5.1 Standard. W3C.
  4. ^ "8.2.2.3. Character encodings". HTML 5 Standard. W3C.
  5. ^ "12.2.3.3 Character encodings". HTML Living Standard. WHATWG.
  6. ^ "<meta> - HTML". MDN Web Docs. Mozilla.
  7. ^ "Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8". Oracle Corporation. 2015. Retrieved 2021-04-30.
[edit]
Retrieved from "https://en.wikipedia.org/w/index.php?title=CESU-8&oldid=1083247588"