UTF-7: Difference between revisions

<!-- Not | extends = [[US-ASCII]] as even plain ASCII text can be reinterpreted. -->

| encodes = [[ISO/IEC 10646]] ([[Unicode]])

'''UTF-7''' (7-[[bit]] [[Unicode Transformation Format]]) is an obsolete variable-length character encoding for representing [[Unicode]] text using a stream of [[ASCII]] characters. It was originally intended to provide a means of encoding [[Unicode]] text for use in [[Internet]] [[E-mail]] messages that was more efficient than the combination of [[UTF-8]] with [[quoted-printable]].

UTF-7 (according to its RFC) isn't a "[[Unicode Transformation Format]]", as the definition can only encode code points in the [[Basic Multilingual Plane|BMP]] (the first 65536 Unicode code points, which does not include [[emojis]] and many other characters). However if a UTF-7 translator is to/from [[UTF-16]] then it can (and probably does){{citation needed|date=August 2023}} encode each surrogate half as though it was a 16-bit code point, and thus can encode all code points. It is unclear if other UTF-7 software (such as translators to UTF-32 or UTF-8) support this.

UTF-7 has never has been an official standard of the [[Unicode Consortium]]. It is known to have security issues, which is why software has been changed to disable its use.<ref name="dotnet5"/> It is prohibited in [[HTML 5]].<ref>{{CitationCite web needed|dateurl=Decemberhttps://www.w3.org/TR/html51/syntax.html#character-encodings 2020|title=8.2.2.3. Character encodings |website=HTML 5.1 Standard |publisher=W3C}}</ref><ref>{{Cite web |url=https://html.spec.whatwg.org/multipage/parsing.html#character-encodings |title=12.2.3.3 Character encodings |website=HTML Living Standard |publisher=WHATWG}}</ref>

[[MIME]], the modern standard offor Ee-mail formatformats, forbids encoding of [[header (computing)|headers]] using byte values above the ASCII range. Although MIME allows encoding the message body in various [[character set]]s (broader than ASCII), the underlying transmission infrastructure ([[Simple Mail Transfer Protocol|SMTP]], the main E-mail transfer standard) is still not guaranteed to be [[8-bit clean]]. Therefore, a non-trivial content transfer encoding has to be applied in case of doubt. Unfortunately [[base64Base64]] has a disadvantage of making even [[US-ASCII]] characters unreadable in non-MIME clients. On the other hand, UTF-8 combined with [[quoted-printable]] produces a very size-inefficient format requiring 6&ndash;9 bytes for non-ASCII characters from the [[Basic Multilingual Plane|BMP]] and 12 bytes for characters outside the BMP.

Provided certain rules are followed during encoding, UTF-7 can be sent in e-mail without using an underlying MIME [[MIME#Content-Transfer-Encoding|transfer encoding]], but still must be explicitly identified as the text character set. In addition, if used within e-mail headers such as "Subject:", UTF-7 must be contained in MIME [[MIME#Encoded-Word|encoded word]]s identifying the character set. Since encoded words force use of either [[quoted-printable]] or [[base64Base64]], UTF-7 was designed to avoid using the = sign as an escape character to avoid double escaping when it is combined with quoted-printable (or its variant, the RFC 2047/1522 ?"Q?"-encoding of headers).

UTF-7 is generally not used as a native representation within applications as it is very awkward to process. Despite its size advantage over the combination of UTF-8 with either quoted-printable or base64Base64, the now defunct [[Internet Mail Consortium]] recommended against its use.<ref>{{Cite web|url=https://www.imc.org/imcr-010.html|title=Using International Characters in Internet Mail |work=Internet Mail Consortium |date=1 August 1998 |archive-url=https://web.archive.org/web/20150907234243/https://www.imc.org/imcr-010.html |archive-date=2015-09-07}}</ref>

A modified form of UTF-7 (sometimes dubbed 'mUTF-7'{{Citation needed|date=December 2015}}) is currently used in the [[IMAP]] e-mail retrieval protocol for mailbox names.<ref>RFC{{cite 3501 section 5.1.3</ref>web

Some characters can be represented directly as single ASCII bytes. The first group is known as "direct characters" and contains 62 alphanumeric characters and 9 symbols: <code>' ( ) , - . / : ?</code>. The direct characters are safe to include literally. The other main group, known as "optional direct characters", contains all other printable characters in the range {{U+|0020}}&ndash;U+007E except <code>~ \ +</code> and space (the characters {{code|\}} and {{code|~}} being excluded due to being redefined in "variants of ASCII" such as [[JIS-Roman]]). Using the optional direct characters reduces size and enhances human readability but also increases the chance of breakage by things like badly designed mail gateways and may require extra escaping when used in encoded words for header fields.

Other characters must be encoded in [[UTF-16]] (hence U+10000 and higher would be encoded into two surrogates), and then in [[base64Base64#UTF-7|modified Base64]]. The start of these blocks of modified Base64 -encoded UTF-16 is indicated by a <code>+</code> sign. The end is indicated by any character not in the modified Base64 set. If the character after the modified Base64 is a <code>-</code> (ASCII [[hyphen-minus]]) then it is consumed by the decoder and decoding resumes with the next character. Otherwise decoding resumes with the character after the base64Base64.

First, an encoder must decide which characters to represent directly in ASCII form, which <code>+</code> hashave to be escaped as <code>+-</code>, and which to place in blocks of Unicode characters. AThe simpleexpansion encodercost mayof encodeUTF-7 allcan charactersbe it considers safehigh: for directexample, encodingthe directly.character Howeversequence theU+10FFFF costU+0077 ofU+10FFFF endingis a9 Unicodebytes sequencein UTF-8, outputtingbut a17 singlebytes characterin directlyUTF-7. in(At ASCIIworst, andtreating thenevery startingcodepoint anotheras Unicodea sequence isin 3its toown {{frac|3|2|3}}right bytes. This is more thanproduces the {{frac|2|2|3}}maximum bytesexpansion neededof to5x, represente.g. thewhen characterencoding <code>@@</code> as a part of a Unicode sequence<code>+AEA-+AEA-</code>.) Each Unicode sequence must be encoded using the following procedure, then surrounded by the appropriate delimiters.

|1= Express the character's Unicode numbers (UTF-16) in Binarybinary:{{unbulleted list

|2= Concatenate the binary sequences:<br />

|3= Regroup the binary into groups of six bits, starting from the left:<br />

|4= If the last group has fewer than six bits, add trailing zeros:<br />

|5= Replace each group of six bits with a respective Base64 code:<br />

#Each group of 16 bits is a character's Unicode (UTF-16) number and can be expressed in other forms:<br /><samp>0000 0000 1010 0011 ≡ 0x00A3 ≡ 163₁₀</samp>

==UnicodeByte signatureorder mark==

A Unicodebyte signatureorder mark (often loosely called a "BOM") is an optional special byte sequence at the very start of a stream or file that, without being data itself, indicates the encoding used for the data that follows; ait signaturecan isbe used<!--it often isn't--> in the absence of metadata that denotes the encoding. For a given encoding scheme, the signature isit's that scheme's representation of Unicode code point <code>U+FEFF</code>,.<ref>{{Cite theweb|url=https://unicode.org/faq/utf_bom.html|title=FAQ so– UTF-called8, ''BOM''UTF-16, (byteUTF-order32 mark)& [character].BOM}}</ref>

While a Unicode signature isit's typically a single, fixed byte sequence, the nature ofin UTF-7 necessitates 5four variations: Themay appear, because the last 2 bits of the 4th byte of the UTF-7 encoding of <code>U+FEFF</code> belong to the ''following'' character, resulting in 4 possible bit patterns and therefore 4 different possible bytes in the 4th position. The 5th variation is needed to disambiguate the case where no characters at all follow the signature. See the UTF-7 entry in the [[Byte order mark#Byte order marks by encoding|table of Unicode signaturesbyte order marks]].<ref>{{cite web| url=https://unicode.org/L2/L2021/21038-bom-guidance.pdf | title=Clarify guidance for use of a BOM as a UTF-8 encoding signature | access-date=2024-01-17}}</ref>

Older versions of [[Internet Explorer]] can be tricked into interpreting the page as UTF-7. This can be used for a [[cross-site scripting]] attack as the <code>&lt;</code> and <code>&gt;</code> marks can be encoded as <code>+ADw-</code> and <code>+AD4-</code> in UTF-7, which most validators let through as simple text.<ref>{{cite web|url=https://code.google.com/p/doctype-mirror/wiki/ArticleUtf7 |title=ArticleUtf7 - doctype-mirror - UTF-7: the case of the missing charset - Mirror of Google Doctype - Google Project Hosting |publisher=Code.google.com |date=2011-10-14 |access-date=2012-06-29}}</ref>

UTF-7 is considered obsolete, at least for Microsoft software (.NET), with code paths previously supporting it intentionally broken (to prevent security issues) in .NET 5, in 2020.<ref name="dotnet5">{{Cite web|last=gewarren|title=BaseBreaking classchange: libraryUTF-7 breakingcode changespaths -are .NET Coreobsolete|url=https://docs.microsoft.com/en-us/dotnet/core/compatibility/corefxcore-libraries/5.0/utf-7-code-paths-obsolete|access-date=20202021-1101-1108|website=docs.microsoft.com|language=en-us}}</ref>