GSM 03.38

The standard encoding for GSM messages is the 7-bit default alphabet as defined in the 23.038 recommendation. with no other language indicator but only the standard class prefix, can transport up to (140*8)/7=160, that is 160 GSM 7-bit characters (but note that the ESC code counts for one of them, if characters in the high part of the table are used). Longer messages may be sent, but will require a continuation prefix and a sequence number on subsequent SMS messages (these prefix bytes and sequence number are counted within the maximum length of the 140-octet payload of the envelope format). When there are 1 to 6 spare bits in the last octet of a message, these bits are set to zero (these bits do not count as a character but only as a filler). When there are 7 spare bits in the last octet of a message, these bits are set to the 7-bit code of the CR control (also used as a padding filler) instead of being set to zero (where they would be confused with the 7-bit code of an '@' character). This 7-bit encoding allows the transport of texts consisting of printable characters from Basic Latin (Unicode block) (with the exception of the grave accent/backtick), as well as some characters of the ISO Latin 1 character set. It also allows the encoding of texts written in the Greek script, but only capitals; for such use in Greek, the Latin capital letters that look like the Greek letters are reused with the same code, so that the above character set is complete only for modern monotonic Greek restricted to capital letters. A complete support for the Greek alphabet (including small letters) requires a national version of the shifted 7-bit table (using the ESC code for each national character encoded in this shifted table), or an unspecified proprietary 8-bit encoding, or the use of the UCS-2 encoding (see below). Note that the special code marked SS2 in the table above has also been assigned (and encoded as 0x1B,0x1B) to allow using another alternate 7-bit shift table. But this mechanism has never been used and the UCS-2 encoding has been preferred. Note that the character 0x09 (Ç, capital C with cedilla) should instead be replaced by ç (small c with cedilla) in modern implementation, as recommended by Unicode, since the uppercase version is of little use. == GSM 8-bit data encoding ==

8-bit data encoding mode treats the information as raw data. According to the standard, the alphabet for this encoding is user-specific. == UCS-2 Encoding ==

This encoding allows use of a greater range of characters and languages. UCS-2 can represent the most commonly used Latin and eastern characters at the cost of a greater space expense. Strictly speaking, UCS-2 is limited to characters in the Basic Multilingual Plane. However, since modern programming environments do not provide encoders or decoders for UCS-2, some cell phones (e.g. iPhones) use UTF-16 instead of UCS-2. This works, because for characters in the Basic Multilingual Plane (including full alphabets of most modern human languages) UCS-2 and UTF-16 encodings are identical. To encode characters outside of the BMP (unreachable in plain UCS-2), such as Emoji, UTF-16 uses surrogate pairs, which when decoded with UCS-2 would appear as two valid but unmapped code points. A single SMS GSM message using this encoding can have at most 70 characters (140 octets). Note that on many GSM cell phones, there's no specific preselection of the UCS-2 encoding. The default is to use the 7-bit encoding described above, until one enters a character that is not present in the GSM 7-bit table (for example the lowercase 'a' with acute: 'á'). In that case, the whole message gets reencoded using the UCS-2 encoding, and the maximum length of the message sent in a single SMS is immediately reduced to 70 characters, instead of 160. Others vary based on the choice and configuration of SMS application, and the length of the message. To avoid unexpected costs for senders that have a subscription for a limited pack of sent SMS, applications should display the number of character used and the maximum number of characters in the composed SMS. When a message exceeds this maximum, the message will be sent as multiple successive SMS containing parts of the message (each one containing a sequence number, which also uses a few leading characters in each part); these parts are intended to be reassembled later by the recipient. Some applications alert the user when a message will require splitting, or even send a longer message as a multimedia message (MMS). == National language shift tables ==

Since release 8 of the 3GPP 23.038 standard of March 2008, additional characters sets can be accessed through the use of a National Language Shift Tables. These tables allow using of different character sets according to the language the text is going to be written. The choice of table for a given message is selected in the User Data Header section of an SMS message and can be specified for the whole text (a Locking shift table replacing standard GSM 7-bit default alphabet table) or a single character (Single shift table replacing the GSM 7-bit default alphabet extension table). Locking and Single shift tables together in the same message are possible, if both standard default alphabet table and default alphabet extension table are to be replaced. Using a shift table, a message can still use 7-bit encoding for the characters, but a different set can be chosen to correctly show accented and language specific characters. This allows up to 155 characters, encoded in 136 octets (140 octets, minus the 4-octets of User Data Header required to indicate the use of a shift table and the language code). With both Locking and Single shift tables, up to 152 characters are allowed, encoded in 133 octets (140 octets, minus 7-octets User Data Header). Characters from any locking shift table take one septet, characters from the single shift table (or Basic Character Set Extension table) take two septets. Initially, shift tables only for Turkish were specified; Spanish and Portuguese were added in later revisions of release 8. Release 9 introduced 10 languages used in India written with a Brahmic scripts (Bengali, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, Telugu) and Urdu. There is still no defined national language shift table for French, Greek, Russian, Bulgarian, Arabic, Hebrew and most Central European languages that need a better coverage than the default 7-bit standard character set and its default 7-bit extension character set: if ever any character is composed that cannot be represented in those default GSM 7-bit sets, the message will be automatically reencoded using UCS-2, with the effect of dividing by more than two the maximum length in characters of messages that can be sent at the price of a single SMS (when a message is split in multiple parts, a few other octets are needed in the User Data Header to indicate the sequence number of each part). Although a revision of GSM 03.38 (as early as in version 4.0.1 of September 1994) has defined Data Coding Scheme values for Cell Broadcast System (CBS) for German, English, Italian, French, Spanish, Dutch, Swedish, Danish, Finnish, Norwegian, Greek and Turkish; with Hungarian, Polish, Czech, Hebrew, Arabic, Russian and Icelandic added in later revisions, no coding tables were defined for these languages. The purpose of this field was purely to identify the language of the message. There's also no language shift table for Japanese written in basic kanas, or for Korean written in Hangul jamos, or for Chinese written in the Han script. This is often not a problem in Japan, because it uses other standards than GSM and WAP for messaging. The two other languages also have too many distinct characters to fit into a 7-bit shift table. Spanish language (Latin script) There's no specific Locking Shift Character Set for the Spanish language. Uses the default Basic Character Set. • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   ||title="Latin capital letter I with acute"| Í ||style="background:#CCC"|   ||title="Latin small letter i with acute"| í ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Portuguese language (Latin script) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   ||title="Latin capital letter I with acute"| Í ||style="background:#CCC"|   ||title="Latin small letter i with acute"| í ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Turkish language (Latin script) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   ||title="Latin capital letter dotted I"| İ ||style="background:#CCC"|   ||title="Latin small letter dotless i"| ı ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Urdu language (Arabic and basic Latin scripts) It may also be used for the Sindhi language also written in the Arabic script. Sometimes it may be used for Arabic language as well, but the Eastern digits (encoded here in their Persian-Hindu variant) won't be used in that case because standard Arabic prefer its traditional Eastern Arabic digits, and will frequently be replaced by Western Arabic digits (encoded in the locking shift character set in column 0x30) which are also used now frequently in Urdu as well. However, in India, phones recognizing the Arabic language indication may substitute the Persian-Hindu variants of the Eastern Arabic digits by the traditional Eastern Arabic digits. • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. || ٲ || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Hindi language (Devanagari and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. || ॰ || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Bengali and Assamese languages (Bengali and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Punjabi language (Gurmukhī and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Gujarati language (Gujarati and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Oriya language (Oriya and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Tamil language (Tamil and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Telugu language (Telugu and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. ||style="background:#CCC"|   || I || Y ||style="background:#CCC"|   ||style="background:#CCC"|   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Kannada language (Kannada and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. || style="background:#CCC" |   || I || Y || style="background:#CCC" |   || style="background:#CCC" |   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. Malayalam language (Malayalam and basic Latin scripts) • LF is a Line Feed control. • CR is a Carriage Return control, or filler. • ESC is an Escape control. • SP is a Space character. || style="background:#CCC" |   || I || Y || style="background:#CCC" |   || style="background:#CCC" |   • FF is a Page Break control. If not recognized, it shall be treated like LF. • CR2 is a control character. No language specific character shall be encoded at this position. • SS2 is a second Single Shift Escape control reserved for future extensions. == See also ==