UTS #39: Unicode Security Mechanisms
Unicode® Technical Standard #39
Version 18.0.0 Editors Josh Hadley, Roozbeh Pournader Date 2026-08-27 This Versionhttps://www.unicode.org/reports/tr39/tr39-34.html Previous Versionhttps://www.unicode.org/reports/tr39/tr39-32.html Latest Versionhttps://www.unicode.org/reports/tr39/ Latest Proposed Updatehttps://www.unicode.org/reports/tr39/proposed.html Revision34
_Summary_
_Because Unicode contains such a large number of characters and incorporates the varied writing systems of the world, incorrect usage can expose programs or systems to possible security attacks. This document specifies mechanisms that can be used to detect possible security problems._
_Status_
_This document has been reviewed by Unicode members and other interested parties, and has been approved for publication by the Unicode Consortium. This is a stable document and may be used as reference material or cited as a normative reference by other specifications._
> _**A Unicode Technical Standard (UTS)** is an independent specification. Conformance to the Unicode Standard does not imply conformance to any UTS._
_Please submit corrigenda and other comments with the online reporting form [Feedback]. Related information that is useful in understanding this document is found in the References. For the latest version of the Unicode Standard, see [Unicode]. For a list of current Unicode Technical Reports, see [Reports]. For more information about versions of the Unicode Standard, see [Versions]._
_Contents_
- 1 Introduction
- 2 Conformance
- 3.1 General Security Profile for Identifiers
- Table 1. Identifier_Status and Identifier_Type
- 3.1.1 Joining Controls
- 3.1.1.1 Limited Contexts for Joining Controls
- 3.1.1.2 Limitations
- 3.1.2 Choosing Identifier_Type Values
- 3.2 IDN Security Profiles for Identifiers
- 3.3 Email Security Profiles for Identifiers
- 4.1 Whole-Script Confusables
- 4.2 Mixed-Script Confusables
- 5.1 Mixed-Script Detection
- Table 1a. Mixed Script Examples
- 5.2 Restriction-Level Detection
- 5.3 Mixed-Number Detection
- 5.4 Optional Detection
- 6.1 Confusables Data Collection
- 6.2 Identifier Modification Data Collection
- 7 Data Files
- Table 2. Data File List
- Table 3. Version Correspondence
- * *
1 Introduction
_Unicode Technical Report #36, "Unicode Security Considerations"_ [UTR36] provides guidelines for detecting and avoiding security problems connected with the use of Unicode. This document specifies mechanisms that are used in that document, and can be used elsewhere. Readers should be familiar with [UTR36] before continuing. See also the Unicode FAQ on _Security Issues_ [FAQSec].
2 Conformance
An implementation claiming conformance to this specification must do so in conformance to the following clauses:
**UTS-39-C1[](https://unicode.org/reports/tr39/)**. _An implementation claiming to implement the **General Profile for Identifiers** shall do so by conforming to either UTS-39-C1-1 or UTS-39-C1-2._
> **UTS-39-C1-1**. _The Implementation shall be in accordance with the specifications in Section 3.1, General Security Profile for Identifiers, without change._ > >
> **UTS-39-C1-2**. _The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.1, General Security Profile for Identifiers._
**UTS-39-C1.1[](https://unicode.org/reports/tr39/)**. _An implementation claiming to implement the **IDN Security Profiles for Identifiers** shall do so by conforming to either UTS-39-C1.1-1 or UTS-39-C1.1-2._
> **UTS-39-C1.1-1**. _The implementation shall be in accordance with the specifications in Section 3.2, IDN Security Profiles for Identifiers for Identifiers, without change._ > >
> **UTS-39-C1.1-2**. _The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.2, IDN Security Profiles for Identifiers._
**UTS-39-C1.2[](https://unicode.org/reports/tr39/)**. _An implementation claiming to implement the **Email Security Profiles for Identifiers** shall do so by conforming to either UTS-39-C1.2-1 or UTS-39-C1.2-2._
> **UTS-39-C1.2-1**. _The implementation shall be in accordance with the specifications in Section 3.3, Email Security Profiles for Identifiers, without change._ > >
> **UTS-39-C1.2-2**. _The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.3, Email Security Profiles for Identifiers._
**UTS-39-C2[](https://unicode.org/reports/tr39/)**. _An implementation claiming to implement any of the following confusable-detection functions for Identifiers defined in Section 4, Confusable Detection shall do so by conforming to either UTS-39-C2-1 or UTS-39-C2-2_.
1. X and Y are single-script confusables 2. X and Y are mixed-script confusables 3. X and Y are whole-script confusables 4. X has whole-script confusables in set of scripts S
> **UTS-39-C2-1**. _The implementation of the function shall be in accordance with the specifications in Section 4, Confusable Detection, without change._ > >
> **UTS-39-C2-2**. _The implementation shall provide a precise list of character mappings that are added to or removed from those provided, but otherwise be in accordance with the specifications in Section 4, Confusable Detection._
**UTS-39-C3[](https://unicode.org/reports/tr39/)**. _An implementation claiming to detect mixed scripts shall do so by conforming to either UTS-39-C3-1 or UTS-39-C3-2._
> **UTS-39-C3-1**. _The implementation shall be in accordance with the specifications in Section 5.1, Mixed-script Detection, without change._ > >
> **UTS-39-C3-2**. _The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.1, Mixed-Script Detection._
**UTS-39-C4[](https://unicode.org/reports/tr39/)**. _An implementation claiming to detect Restriction-Levels shall do so by conforming to either UTS-39-C4-1 or UTS-39-C4-2._
> **UTS-39-C4-1**. _The implementation shall be in accordance with the specifications in Section 5.2, Restriction-Level Detection, without change._ > >
> **UTS-39-C4-2**. _The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.2, Restriction-Level Detection._
**UTS-39-C5[](https://unicode.org/reports/tr39/)**. _An implementation claiming to detect mixed numbers shall do so by conforming to either UTS-39-C5-1 or UTS-39-C5-2._
> **UTS-39-C5-1**. _The implementation shall be in accordance with the specifications in Section 5.3, Mixed-Number Detection, without change._ > >
> **UTS-39-C5-2**. _The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.3, Mixed-Number Detection._
3 Identifier Characters
Identifiers ("IDs") are strings used in application contexts to refer to specific entities of certain significance in the given application. In a given application, an identifier will map to at most one specific entity. Many applications have security requirements related to identifiers. A common example is URLs referring to pages or other resources on the Internet: when a user wishes to access a resource, it is important that the user can be certain what resource they are interacting with. For example, they need to know that they are interacting with a particular financial service and not some other entity that is spoofing the intended service for malicious purposes. This illustrates a general security concern for identifiers: potential ambiguity of strings. While a machine has no difficulty distinguishing between any two different character sequences, it could be very difficult for humans to recognize and distinguish identifiers if an application did not limit which Unicode characters could be in identifiers. The focus of this specification is mitigation of such issues related to the security of identifiers.
Deliberately restricting the characters that can be used in identifiers is an important security technique. The exclusion of characters from identifiers does not affect the general use of those characters for other purposes, such as for general text in documents. Unicode Standard Annex #31, "Unicode Identifiers and Syntax" [UAX31] provides a recommended method of determining which strings should qualify as identifiers. The UAX #31 specification extends the common practice of defining identifiers in terms of letters and numbers to the Unicode repertoire.
That specification also permits other protocols to use that method as a base, and to define a _profile_ that adds or removes characters. For example, identifiers for specific programming languages typically add some characters like "$", and remove others like "-" (because of the use as _minus_), while IDNA removes "_" (among others)—see Unicode Technical Standard #46, "Unicode IDNA Compatibility Processing" [UTS46], as well as [IDNA2003], and [IDNA2008].
This document provides for additional identifier profiles for environments where security is an issue. These are profiles of the extended identifiers based on properties and specifications of the Unicode Standard [Unicode], including:
- The XID_Start and XID_Continue properties defined in the Unicode Character Database (see [DCore])
- The toCasefold(X) operation defined in _Chapter 3, Conformance_ of [Unicode]
- The NFKC and NFKD normalizations defined in _Chapter 3, Conformance_ of [Unicode]
The data files used in defining these profiles follow the UCD File Format, which has a semicolon-delimited list of data fields associated with given characters, with each field referenced by number. For more details, see [UCDFormat].
3.1 General Security Profile for Identifiers
The files under [idmod] provide data for a profile of identifiers in environments where security is at issue. The files contain a set of characters recommended to be restricted from use. They also contain a small set of characters that are recommended as additions to the list of characters defined by the XID_Start and XID_Continue properties, because they may be used in identifiers in a broader context than programming identifiers.
The Restricted characters are characters not in common use, and they can be blocked to further reduce the possibilities for visual confusion. They include the following:
- characters not in modern use
- characters only used in specialized fields, such as liturgical characters, phonetic letters, and mathematical letter-like symbols
- characters in limited use by very small communities
The choice of which characters to specify as Restricted starts conservatively, but allows additions in the future as requirements for characters are refined. For information on handling modifications over time, see _Section 2.10.1, Backward Compatibility_ in _Unicode Technical Report #36, "Unicode Security Considerations"_ [UTR36].
An implementation following the General Security Profile does not permit any characters in \p{Identifier_Status=Restricted}, unless it documents the additional characters that it does allow. Such documentation can specify characters via properties, such as \p{Identifier_Type=Technical}, or by explicit lists, or by combinations of these. Implementations may also specify that fewer characters are allowed than implied by \p{Identifier_Status=Allowed}; for example, they can allow only characters permitted by [IDNA2008].
Common candidates for such additions include characters for scripts listed in _Table 7, Limited Use Scripts_ of [UAX31]. However, characters from these scripts have not been a priority for examination for confusables or to determine specialized, non-modern, or uncommon-use characters.
Canonical equivalence is applied when testing candidate identifiers for inclusion of _Allowed_ characters. For example, suppose the candidate string is the sequence
<u, _combining-diaeresis_>
The target string would be Allowed in _either_ of the following 2 situations:
1. u is Allowed and ¨ is Allowed, or 2. ü is Allowed
For details of the format for the [idmod] files, see _Section 7, Data Files_.
Table 1. Identifier_Status and Identifier_Type
| Identifier_Status | Identifier_Type | Description | | --- | --- | --- | | Restricted | Not_Character | Unassigned characters, private use characters, surrogates, non-whitespace control characters. | | Deprecated | Characters with the Unicode property _Deprecated=Yes_. | | Default_Ignorable | Characters with the Unicode property _Default\_Ignorable\_Code\_Point=Yes_. | | Not_NFKC | Characters that cannot occur in strings normalized to NFKC. | | Not_XID | Characters that do not qualify as default Unicode identifiers; that is, they do not have the Unicode property _XID\_Continue=True_. |
| Exclusion | Characters with Script_Extensions values containing a script in _Table 4, Excluded Scripts_ from [UAX31], and no script from _Table 7, Limited Use Scripts_ or _Table 5, Recommended Scripts_, other than “Common” or “Inherited”. | | Obsolete | Characters that are no longer in modern use, or that are not commonly used in modern text. | | Technical | Specialized usage: technical, liturgical, etc. |
| Uncommon_Use | Characters that are uncommon, or are limited in use (even though they are in scripts that are not "Limited_Use"), or whose usage is uncertain. May be combined with Exclusion or Limited_Use for characters that are less common than the main characters of their scripts. |
| Limited_Use | Characters from scripts that are in limited use: with Script_Extensions values containing a script in _Table 7, Limited Use Scripts_ in [UAX31], and no script from _Table 5, Recommended Scripts_, other than “Common” or “Inherited”. |
| Allowed | **Inclusion** | Exceptionally allowed characters, including _Table 3a, Optional Characters for Medial_ and _Table 3b, Optional Characters for Continue_ in [UAX31], and some characters for [IDNA2008], except for certain characters that are Restricted above. |
| **Recommended** | Characters from scripts that are in widespread everyday common use: with Script_Extensions values containing a script in _Table 5, Recommended Scripts_ in [UAX31], except for those characters that are Restricted above. |
> **Note:** In Unicode 15.0, the Join_Control characters (ZWJ/ZWNJ) have been removed from Identifier_Type=Inclusion. They thereby have the properties Identifier_Type=Default_Ignorable and Identifier_Status=Restricted. Their inclusion in programming language identifier profiles has usability and security implications. > >
> Implementations of the General Profile for Identifiers that wish to retain ZWJ and ZWNJ should declare that they use a modification of the profile per _Section 2, Conformance_, and should ensure that they implement the restrictions described in _Section 3.1.1, Joining Controls_.
Identifier_Status and Identifier_Type are properties of characters (code points). See _UTS #18: Unicode Regular Expressions_ [UTS18] and _UTR #23: The Unicode Character Property Model_ [UTR23] for more discussion. For the purpose of regular expressions, the long and short names of these properties and their values are documented in their respective data files; see _Section 7, Data Files_.
For stability considerations, see Migrating Persistent Data.
There may be multiple reasons for restricting a character; therefore, the Identifier_Type property allows multiple values that correspond with Restricted. For example, some characters have Identifier_Type values of Limited_Use and Technical. Multiple values are not assigned to characters with strong restrictions: Not_Character, Deprecated, Default_Ignorable, Not_NFKC. For example, if a character is Deprecated, there is little value in also marking it as Uncommon_Use. For the qualifiers on usage, Obsolete, Uncommon_Use and Technical, the distinctions among the Identifier_Type values is not strict and only one might be given. The important characteristic is the Identifier_Status: whether or not the character is Restricted.
The default Identifier_Type property value should be Uncommon_Use if no other categories apply. See _Section 3.1.2, Choosing Identifier\_Type Values_.
_As more information is gathered about characters, this data may change in successive versions._ That can cause either the Identifier_Status or Identifier_Type to change for a particular character. Thus users of this data should be prepared for changes in successive versions, such as by having a backward compatibility policy in place for previously supported characters or registrations. Both Identifier_Status and Identifier_Type values are to be compared case-insensitively and ignoring hyphens and underbars.
Restricted characters should be treated with caution when considering possible use in identifiers, and should be disallowed unless there is good reason to allow them in the environment in question. However, the set of Identifier_Status=Allowed characters are not typically used as is by implementations. Instead, they are applied as filters to the set of characters C that are supported by the identifier syntax, generating a new set C′. Typically there are also particular characters or classes of characters from C that are retained as **Exception** characters.
C′ = (C ∩ {Identifier_Status=Allowed}) ∪ **Exception**
The implementation may simply restrict use of new identifiers to C′, or may apply some other strategy. For example, there might be an appeal process for registrations of ids that contain characters outside of C′ (but still inside of C), or in user interfaces for lookup of identifiers, warnings of some kind may be appropriate. For more information, see [UTR36].
The **Exception** characters would be implementation-specific. For example, a particular implementation might extend the default Unicode identifier syntax by adding **Exception** characters with the Unicode property _XID\_Continue=False_, such as “$”, “-”, and “.”. Those characters are specific to that identifier syntax, and would be retained even though they are not in the Identifier_Status=Allowed set. Some implementations may also wish to add some [CLDR] exemplar characters for particular supported languages that have unusual characters.
The Identifier_Type=Inclusion characters already contain some characters that are not letters or numbers, but that are used within words in some languages. For example, it is recommended that U+00B7 (·) MIDDLE DOT be allowed in identifiers, because it is required for Catalan.
The implementation may also apply other restrictions discussed in this document, such as checking for confusable characters or doing mixed-script detection.
3.1.1 Joining Controls
Visible distinctions created by certain characters excluded by the General Security Profile because their Identifier_Type is Default_Ignorable (particularly the _Join\_Control characters_) are necessary in certain languages. A blanket exclusion of these characters makes it impossible to create identifiers with the correct visual appearance for common words or phrases in those languages.
Identifier systems that attempt to provide more natural representations of terms in "modern, customary usage" should allow these characters in input and display, but limit them to contexts in which they are necessary. The term _modern customary usage_ includes characters that are in common use in newspapers, journals, lay publications; on street signs; in commercial signage; and as part of common geographic names and company names, and so on. It does not include technical or academic usage such as in mathematical expressions, using archaic scripts or words, or pedagogical use (such as illustration of half-forms or joining forms in isolation), or liturgical use.
The goals for such a restriction of format characters to particular contexts are to:
- Allow the use of these characters where required in normal text
- Exclude as many cases as possible where no visible distinction results
- Be simple enough to be easily implemented with standard mechanisms such as regular expressions
An implementation following the General Security Profile that allows the additional characters ZWJ and ZWNJ shall only permit them where they satisfy the conditions A1, A2, and B in _Section 3.1.1.1, Limited Contexts for Joiner Controls_, unless it documents the additional contexts where it allows them.
More advanced implementations may use script-specific information for more detailed testing. In particular, they can:
1. _Disallow joining controls_ in sequences that meet the conditions of A1, A2, and B, where in common fonts the resulting appearance of the sequence is normally not distinct from appearance in the same sequences with the joining controls removed.
2. _Allow joining controls_ in sequences that don't meet the conditions of A1, A2, and B, where in common fonts the resulting appearance of the sequence is normally distinct from the appearance in the same sequences with the joining controls removed. The following regular expressions describe sequences that typically result in distinct rendering. They use the notation explained below in A1.
> /$L ZWNJ $V $L/ > > > /$L ZWJ $V $L/
#### 3.1.1.1 Limited Contexts for Joining Controls
An implementation that attempts to provide more natural representations of terms in "modern, customary usage" should allow the following Join_Control characters in the limited contexts specified in A1, A2, and B below.
> U+200C ZERO WIDTH NON-JOINER (ZWNJ) > U+200D ZERO WIDTH JOINER (ZWJ)
There are also two global conditions incorporated in each of A1, A2, and B:
- **Script Restriction.** In each of the following cases, the specified sequence must only consist of characters from a single script (after ignoring _Common_ and _Inherited_ script characters).
- **Normalization.**In each of the following cases, the specified sequence must be in NFC format. (To test an identifier that is not required to be in NFC, first transform into NFC format and then test the condition.)
Implementations may also impose tighter restrictions than provided below, in order to eliminate some other circumstances where the characters either have no visual effect or the effect has no semantic importance.
**A1. Allow ZWNJ in the following context:**
**Breaking a cursive connection.** That is, in the context based on the Joining_Type property, consisting of:
- A Left-Joining or Dual-Joining character, followed by zero or more Transparent characters, followed by a ZWNJ, followed by a Right-Joining or Dual-Joining character
This corresponds to the following regular expression (in Perl-style syntax): **/$LJ $T* ZWNJ $RJ/**
Where the character classes like $T could be defined with Unicode properties (similar to UnicodeSet notation) like this:
> $T = \p{Joining_Type=Transparent} > $RJ = [\p{Joining_Type=Dual_Joining}\p{Joining_Type=Right_Joining}] > $LJ = [\p{Joining_Type=Dual_Joining}\p{Joining_Type=Left_Joining}]
For example, consider the Persian word <_Noon, Alef, Meem, Heh, Alef, Farsi Yeh_>. Without a ZWNJ, it translates to "names of", as shown in the first row; with a ZWNJ between _Heh_ and _Alef_, it means "a letter", as shown in the second row of _Figure 1_.
Figure 1. Persian Example with ZWNJ
| Appearance | Code Points | Abbreviated Names | | --- | --- | --- | | !Image 1: diagram1 | 0646 + 0627 + 0645 + 0647 + 0627 + 06CC | NOON + ALEF + MEEM + HEH + ALEF + FARSI YEH | | !Image 2: diagram2 | 0646 + 0627 + 0645 + 0647 + 200C + 0627 + 06CC | NOON + ALEF + MEEM + HEH + ZWNJ + ALEF + FARSI YEH |
**A2. Allow ZWNJ in the following context:**
**In a conjunct context.** That is, a sequence of the form:
- A Letter, followed by a Virama, followed by a ZWNJ (optionally preceded or followed by certain nonspacing marks), followed by a Letter.
This corresponds to the following regular expression (in Perl-style syntax): **/$L $M* $V $M₁* ZWNJ $M₁* $L/**
Where:
> $L = \p{General_Category=Letter} > $V = \p{Canonical_Combining_Class=Virama} > $M = \p{General_Category=Mn} > $M₁ = [\p{General_Category=Mn}&\p{CCC≠0}]
For example, the Malayalam word for _eyewitness_ is shown in _Figure 2_. The form without the ZWNJ in the second row is incorrect in this case.
Figure 2. Malayalam Example with ZWNJ
| Appearance | Code Points | Abbreviated Names | | --- | --- | --- |
| !Image 3: diagram3 | 0D26 + 0D43 + 0D15 + 0D4D + 200C + 0D38 + 0D3E + 0D15 + 0D4D + 0D37 + 0D3F | DA + VOWEL SIGN VOCALIC R + KA + VIRAMA + ZWNJ + SA + VOWEL SIGN AA + KA + VIRAMA + SSA + VOWEL SIGN I |
| !Image 4: diagram4 | 0D26 + 0D43 + 0D15 + 0D4D + 0D38 + 0D3E + 0D15 + 0D4D + 0D37 + 0D3F | DA + VOWEL SIGN VOCALIC R + KA + VIRAMA + SA + VOWEL SIGN AA + KA + VIRAMA + SSA + VOWEL SIGN I |
**B. Allow ZWJ in the following context:**
**In a conjunct context.**That is, a sequence of the form:
- A Letter, followed by a Virama, followed by a ZWJ (optionally preceded or followed by certain nonspacing marks), and not followed by a character of type Indic_Syllabic_Category=Vowel_Dependent
This corresponds to the following regular expression (in Perl-style syntax): **/$L $M* $V $M₁* ZWJ (?!$D)/**
Where:
> $L= \p{General_Category=Letter} > $V = \p{Canonical_Combining_Class=Virama} > $M = \p{General_Category=Mn} > $M₁ = [\p{General_Category=Mn}&\p{CCC≠0}] > $D = \p{Indic_Syllabic_Category=Vowel_Dependent}
For example, the Sinhala word for the country 'Sri Lanka' is shown in the first row of _Figure 3_, which uses both a space character and a ZWJ. Removing the space results in the text shown in the second row of _Figure 3_, which is still legible, but removing the ZWJ completely modifies the appearance of the 'Sri' cluster and results in the unacceptable text appearance shown in the third row of _Figure 3_.
Figure 3. Sinhala Example with ZWJ
| Appearance | Code Points | Abbreviated Names | | --- | --- | --- |
| !Image 5: diagram5 | 0DC1 + 0DCA + 200D + 0DBB + 0DD3 + 0020 + 0DBD + 0D82 + 0D9A + 0DCF | SHA + VIRAMA + ZWJ + RA + VOWEL SIGN II + SPACE + LA + ANUSVARA + KA + VOWEL SIGN AA |
| !Image 6: diagram6 | 0DC1 + 0DCA + 200D + 0DBB + 0DD3 + 0DBD + 0D82 + 0D9A + 0DCF | SHA + VIRAMA + ZWJ + RA + VOWEL SIGN II + LA + ANUSVARA + KA + VOWEL SIGN AA |
| !Image 7: diagram7 | 0DC1 + 0DCA + 0DBB + 0DD3 + 0020 + 0DBD + 0D82 + 0D9A + 0DCF | SHA + VIRAMA + RA + VOWEL SIGN II + SPACE + LA + ANUSVARA + KA + VOWEL SIGN AA |
> **Note:**The restrictions in A1, A2, and Bare more restrictive than the CONTEXTJ rules defined in _Appendix A, Contextual Rules Registry_, in _The Unicode Code Points and Internationalized Domain Names for Applications (IDNA)_[IDNA2008]. In particular, A1does not allow transparent characters after the ZWNJ, while A2and Brequire an extended context indicating an actual conjunct cluster rather than merely a single preceding virama. Consequently, any identifier that satisfies these constraints also satisfies the CONTEXTJ rules for joining controls in IDNs. Note that many registries impose additional restrictions on permissible IDNs beyond those required by CONTEXTJ.
#### 3.1.1.2 Limitations
While the restrictions in A1, A2, and B greatly limit visual confusability, they do not prevent it. For example, because Tamil only uses a Join_Control character in one specific case, most of the sequences these rules allow in Tamil are, in fact, visually confusable. Therefore based on their knowledge of the script concerned, implementations may choose to have tighter restrictions than specified in _Section 3.1.1.1, Limited Contexts for Joining Controls_—for example, by explicitly providing for the exceptional sequence, while otherwise disallowing the joiner in context.
There are also cases where a joiner preceding a virama makes a visual distinction in some scripts. It is currently unclear whether this distinction is important enough in identifiers to warrant retention of a joiner. For more information, see UTR #36: _Unicode Security Considerations_ [UTR36].
**_Performance._** Parsing identifiers can be a performance-sensitive task. However, these characters are quite rare in practice, thus the regular expressions (or equivalent processing) only rarely would need to be invoked. Thus these tests should not add any significant performance cost overall.
3.1.2 Choosing Identifier_Type Values
The following identifier types may be assigned singly or in combination. The values are based on best available information and may be updated when new information becomes available. Multiple classifications may be possible, particularly where a character is of one type in a commonly used writing system and of different type in another context.
- **Uncommon_Use** focuses on usage of the character in orthographies for living languages. Uncommon_Use can also represent the absence of confirmed or credible data on a level of usage that would correspond to “common everyday use” for an orthography in widespread use. Uncommon_Use is the default for newly encoded characters for all scripts other than Excluded scripts, unless a sufficient level of usage can be confirmed for one or more specific orthographies in common modern use.
- Where combining marks, such as U+3099 COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK, are only needed for the NFD form of recommended characters, they have been given the Identifier_Type Uncommon_Use. Identifier systems that work with unnormalized text, or text in NFD, may wish to add the full set of characters required in canonical decompositions.
- **Obsolete** focuses on the degree to which a character is in common modern use. If a writing system, or an orthography using a character, has fallen out of use, or a character is no longer used in a given context, it is marked Obsolete. A character can become obsolete in the context of a writing system; it is not required that the entire writing system has fallen out of use. A character may be Obsolete in a widely used writing system, but also part of an orthography where it is in Uncommon_Use.
- **Technical** focuses on the purpose of use. If a character is limited to particular types of texts or forms part of a notation that is not in everyday use, then it would be appropriate to categorize it as Technical. Technical uses can include liturgical texts, poetry, phonetic notation, and so on. A character may have a common technical use, but may also be used in one or more orthographies at a level that is marked by Uncommon_Use, or it could be Obsolete as a character for general use.
- **Inclusion** focuses on Allowed punctuation and characters that look like punctuation. For each identifier environment they should be carefully reviewed, as some or all of them may not be suitable for that environment: For example, when a character is, or can be confused with, a syntax character in the given environment.
The other identifier types are largely, if not fully, determined by a character’s other property values and are therefore automatically assigned.
3.2 IDN Security Profiles for Identifiers
Version 1 of this document defined operations and data that apply to [IDNA2003], which has been superseded by [IDNA2008] and Unicode Technical Standard #46, "Unicode IDNA Compatibility Processing" [UTS46]. The identifier modification data can be applied to whichever specification of IDNA is being used. For more information, see the [IDN FAQ].
However, implementations can claim conformance to other features of this document as applied to domain names, such as Restriction Levels.
In addition, there are other specifications that are extremely useful for IDNs. Notably, RFC7940 specifies a machine-readable format for expressing profiles for IDNs with specific features that support security against spoofing. Such profiles are known as “Label Generation Rules” (LGR) and are typically defined for a script or language, with features that allow support of multiple LGRs for the same “DNS zone”.
Using the LGR format, implementations can:
1. **Select a repertoire of characters or enumerated character sequences.** The selection of the set of characters can take into account the intersection between IDNA2008 allowed values and Identifier_Status resp. Identifier_Type. 2. **Limit certain characters to one or more of an enumerated set of sequences.** That’s useful to limit diacritics to contexts where they are expected, or to exclude standalone use of some characters only found in certain sequences.
3. **Provide a context rule.** A context rule is an anchored regular expression describing either a required or prohibited context for a character or sequence inside a label. This is useful for restrictions on Indic scripts, so you can stay within safe boundaries of what rendering engines can render in a distinct way. One application is to implement Limited Contexts for Joining Controls. 4. **Make two characters / sequences “blocked” variants of each other.** Either one is allowed, but the first label registered blocks the label differing only by a variant character or sequence at that position.
5. **Use a “whole label evaluation” rule to validate an identifier.** A WLE rule is a regular expression describing a pattern either for a valid or an invalid label. This can be used, for example, to prevent digit set mixing in the same label, or making sure a label is consistent in the choice of a regional alternative for a character.
The LGR mechanism, as defined in RFC 7940, allows you to present all of these restrictions in a machine-readable form which you can then convert into whatever works best for validating and resolving competing identifier registrations in your system (or symbol table). The format was designed with IDNA2008 in mind and is used in the normative definition of IDN profiles for the DNS Root Zone.
In RFC7940, the regular expressions can be specified in terms of Unicode properties or custom attributes assigned to the characters in the LGR. The same mechanisms can be applied to other types of identifiers. More information can be found at [ICANN].
3.3 Email Security Profiles for Identifiers
The _SMTP Extension for Internationalized Email_ provides for specifications of internationalized email addresses [EAI]. However, it does not provide for testing those addresses for security issues. This section provides an email security profile that may be used for that. It can be applied for different purposes, such as:
1. When an email address is registered, flag anything that does not meet the profile:
- Either forbid the registration, or
- Allow for an appeals process.
2. When an email address is detected in linkification of plain text:
- Do not linkify if the identifier does not meet the profile.
3. When an email address is displayed in incoming email:
- Flag it as suspicious with a wavy underline, if it does not meet the profile.
- Filter characters from the quoted-string-part to prevent display problems.
This profile does not exclude characters from EAI. Instead, it provides a profile that can be used for registration, linkification, and notification. The goal is to flag addresses that are structurally unsound or contain unexpected detritus.
An email address is formed from three main parts. (There are more elements of an email address, but these are the ones for which Unicode security is important.) For example:
> "Joey" <joe31834@gmail.com> > > > * The **domain-part** is "gmail.com" > * The **local-part** is "joe31834" > * The **quoted-string-part** is "Joey"
To meet the requirements of the **Email Security Profiles for Identifiers** section of this specification, an identifier must satisfy the following conditions for the specified <restriction level>.
#### Domain-Part
The domain-part of an email address must satisfy _Section 3.2, IDN Security Profiles for Identifiers_, and satisfy the conformance clauses of [UTS46].
#### Local-Part
The local-part of an email address must satisfy all the following conditions:
1. It must be in NFKC format 2. It must have level = <restriction level> or less, from Restriction_Level_Detection 3. It must not have mixed number systems according to Mixed_Number_Detection 4. It must satisfy _dot-atom-text_ from RFC 5322 §3.2.3, where _atext_ is extended as follows:
> Where C ≤ U+007F, C is defined as in §3.2.3. (That is, C ∈ [!#-'*+\-/-9=?A-Z\^-~]. This list copies what is already in §3.2.3, and follows HTML5) for ASCII.) > > > Where C > U+007F, both of the following conditions are true: > > > 1. C has Identifier_Status=Allowed from General Security Profile > 2. If C is the first character, it must be XID_Start from Default Identifier_Syntax in [UAX31]
Note that in RFC 5322 §3.2.3:
> ``` > dot-atom-text = > 1*atext *("." 1*atext) > ```
That is, dots can also occur in the local-part, but not leading, trailing, or two in a row. In more conventional regex syntax, this would be:
> ``` > dot-atom-text = > atext+ ("." atext+)* > ```
Note that bidirectional controls and other format characters are specifically disallowed in the local-part, according to the above.
#### Quoted-String-Part
The quoted-string-part of an email address must satisfy the following conditions:
1. It must be in NFC. 2. It must not contain any stateful bidirectional format characters.
- That is, no [:bidicontrol:] except for the LRM, RLM, and ALM, since the bidirectional controls could influence the ordering of characters outside the quotes.
3. It must not contain more than four nonspacing marks in a row, and no sequence of two of the same nonspacing marks. 4. It may contain mixed scripts, symbols (including emoji), and so on.
#### Other Issues
The restrictions above are insufficient to prevent bidirectional-reordering that could intermix the quoted-string-part with the local-part or the domain-part in display. To prevent that, implementations could use bidirectional isolates (or equivalent) around each of these parts in display.
Implementations may also want to use other checks, such as for confusability, or services such as Safe Browsing.
A serious practical issue is that clients do not know what the identity rules are for any particular email server: that is, when two email addresses are considered equivalent. For example, are _mark@macchiato.com_ and _Mark@macchiato.com_ treated the same by the server? Unfortunately, there is no way to query a server to see what identity rules it follows. One of the techniques used to deal with this problem is having whitelists of email providers indicating which of them are case-insensitive, dot-insensitive, or both.
4 Confusable Detection
The data in [confusables] provide a mechanism for determining when two strings are visually confusable. The data in these files may be refined and extended over time. For information on handling modifications over time, see the Migration section of this document.
The data provides a mapping from source characters to their prototypes. A prototype should be thought of as a sequence of one or more classes of symbols, where each class has an exemplar character. For example, the character U+0153 (œ), LATIN SMALL LIGATURE OE, has a prototype consisting of two symbol classes: the one with exemplar character U+006F (o), and the one with exemplar character U+0065 (e). If an input character does not have a prototype explicitly defined in the data file, the prototype is assumed to consist of the class of symbols with the input character as the exemplar character.
For an input string X, define internalSkeleton(X) to be the following transformation on the string:
1. Convert X to NFD format, as described in [UAX15]. 2. Remove any characters in X that have the property Default_Ignorable_Code_Point. 3. Concatenate the prototypes for each character in X according to the specified data, producing a string of exemplar characters. 4. Reapply NFD.
For an input string X and a direction 𝑑 ∈ {RTL, LTR, FS}, define bidiSkeleton(𝑑, X) to be the following transformation on the string:
1. Reorder the code points in X for display by applying the rules of the Unicode Bidirectional Algorithm [UAX9] up to and including L2, treating X in isolation; if 𝑑≠FS, apply protocol HL1 to set the paragraph level to 1 if 𝑑=RTL, and to 0 if 𝑑=LTR; this yields the reordered sequence of characters R. 2. Apply rule L3 of the UBA: move combining marks after their base in R; this yields the sequence R′. 3. Replace any character whose glyph would be mirrored by rule L4 of the UBA by the value of its Bidi_Mirroring_Glyph property, yielding R″. 4. bidiSkeleton(𝑑, X) is then internalSkeleton(R″).
The strings X and Y are defined to be 𝑑-confusable if and only if bidiSkeleton(𝑑, X) = bidiSkeleton(𝑑, Y). This is abbreviated as X ≒ Y (𝑑).
This mechanism imposes transitivity on the data, so if X ≒ Y (𝑑) and Y ≒ Z (𝑑), then X ≒ Z (𝑑). It is possible to provide a more sophisticated confusable detection, by providing a metric between given characters, indicating their "closeness." However, that is computationally much more expensive, and requires more sophisticated data, so at this point in time the simpler mechanism has been chosen. That means that in some cases the test may be overly inclusive.
> **Note:**The operation _internalSkeleton_may change the Bidi_Class of characters, so it does not commute with the reordering and mirroring steps, and needs to be performed after them.
> **Example:** The sequences of code points S₁ and S₂ are LTR-confusable: > > > > S₁ ≔ "A1<שׂ" = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT) > > S₂ ≔ "Αשֺ>1" = (GREEK CAPITAL LETTER ALPHA, HEBREW LETTER SHIN, HEBREW POINT HOLAM HASER FOR VAV, GREATER-THAN SIGN, DIGIT ONE) > > > Computation of bidiSkeleton(LTR, S₁): > > R₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW POINT SIN DOT, HEBREW LETTER SHIN) > R′₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT) > R″₁ = (LATIN CAPITAL LETTER A, DIGIT ONE, LESS-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT SIN DOT) > bidiSkeleton(LTR, S₁) = internalSkeleton(R″₁) = (LATIN CAPITAL LETTER A, LATIN SMALL LETTER L, LESS-THAN SIGN, HEBREW LETTER SHIN, COMBINING DOT ABOVE) > Computation of bidiSkeleton(LTR, S₂): > > R₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE, GREATER-THAN SIGN, HEBREW POINT HOLAM HASER FOR VAV, HEBREW LETTER SHIN) > R′₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE, GREATER-THAN SIGN, HEBREW LETTER SHIN, HEBREW POINT HOLAM HASER FOR VAV) > R″₂ = (GREEK CAPITAL LETTER ALPHA, DIGIT ONE,