Wide Character support

Across all unicode capabilities tested, Wide character support is best. This is likely attributed to the widespread adoption of emojis, which are treated as wide characters, generating interest across developers and users of all languages.

While all tested terminals demonstrate support for wide characters, there are variations in the Unicode versions they support. Notably, Konsole, iTerm2, and Kovid Goyal's kitty support wide characters up to Unicode release version 15.0.0 (2022). In contrast, Hyper and Visual Studio Code, both built on xterm.js, provide support only up to Unicode release 12.1.0 (2019).

This means that these wide characters take up 1 cell instead of 2, often occluded by the next character.

Pictured here in Hyper terminal, the wcwidth developer tool wcwidth-browser.py shows several Wide Emoji mistakenly displayed as Narrow instead of Wide, due to out-of-date code tables in xterm.js, causing some to be partially occluded by the Pipe character (|).

The wcwidth project Specification describes Wide characters as:

> Any character defined by East Asian Fullwidth (F)
> or Wide (W) properties in EastAsianWidth txt
> files, except those that are defined by the
> Category codes of Nonspacing Mark (Mn) and
> Spacing Mark (Mc).

The "except" clarification is needed, as there are several characters officially categorized as Wide or Fullwidth, but contradictory definitions of Zero by other data files!

The definition continues:

> Any characters of Modifier Symbol category,
> 'Sk' where 'FULLWIDTH' is present in comment
> of unicode data file, aprox. 3 characters.

The definition is further expanded to include any characters falling within the Modifier Symbol 'Sk' category, specifically those with 'FULLWIDTH' mentioned in the comment field of the Unicode data file—approximately three characters in total.

This clause is crucial for a small set of characters from the modifier symbol category that, while not officially designated as Fullwidth or Wide, indeed exhibit these properties. Detecting these characters necessitates parsing the comment field of the data files.

The "Modifier Symbol" category is a strange category. It is a set of combining characters that do not act as combining characters, they are for lone display, except for the Emoji Modifier Fitzpatrick codepoints, which modify the skin tone of the preceding Emoji in sequence, making it a kind of combining character unlike all other characters of this category.

How difficult! The Unicode.org data files present contradictory categorizations. It's no wonder that developers, even those that strive for full compliance, can still encounter difficulty in accurately categorizing a small percentage of characters.

Zero Width

Testing support for Zero Width characters poses a particular challenge. While it may be possible to combine some combining characters with any other Unicode characters, like U+0309 "Combining Hook Above" with box drawing character U+2532:

>          ┲̉

Hoever, this is not the case for most combining characters, which can only combine with specific characters. For instance, U+094D "Devanagari Sign Virama" successfully combines with an appropriate Devanagari letter, like U+0915 "Devanagari Letter Ka":

>           क्

However, it fails to combine for non-Devanagari letters, such as U+0061 "Latin Small Letter A":

>           a्

The "dotted donut" depicted after "Latin Small Letter A" is used as a placeholder for these illegal combinations.

Depicted here in iTerm2 are several combining characters after U+0007 "Latin Small Letter O", where many fail to combine, resulting in the display of a "dotted donut".

To explore and visualize combining characters in a naive manner, you can use the developer tool wcwidth-browser.py from the wcwidth repository. Press 'c' after launch or use the CLI argument --combining. However, this tool serves primarily to demonstrate that naive combining is not feasible for a vast number of characters.

> Implementations that determine the boundaries
> between characters of given scripts should never
> break between a combining mark (a character with
> General_Category value of Mc, Mn or Me)

> The CM line break class includes all combining
> characters with General_Category Mc, Me, and Mn,
> unless listed explicitly elsewhere. This includes
> viramas that don’t have line break class VI or VF.

Variation Selector-16

U+FE0F "Variation Selector-16" is peculiar.

I suspect it is some kind of "fixup" or compatibility sequence for the earliest emojis. These emojis may be displayed in either "text" or "emoji" style, and default to "text" style. In "text" style, emojis should appear without color in a single cell (Narrow), while in "emoji" style, they should display in color and occupy two cells (Wide).

Despite this distinction, very few fonts effectively differentiate between the two styles, often rendering both types in color. When not in sequence with U+FE0F "Variation Selector-16", they are occluded by any next character.

For example, U+23F1 "Stopwatch":

Depicted here in iTerm2 is a single U+23F1 "Stopwatch" character partially occluded by any next character. Surprisingly, this is the correct behavior of a terminal when U+FE0F "Variation Selector-16" is not in sequence.

From python wcwidth Specification on Wide characters:

> Any character in sequence with `U+FE0F`_
> (Variation Selector 16) defined by Emoji
> Variation Sequences txt as ``emoji style``.

A list of such characters is found in emoji-variation-sequence.txt.

VS-16 Results

Out of the 23 terminals subjected to testing, only 7 demonstrated correct behavior by displaying these emojis as "Wide" characters when combined with VS-16 in sequence.

Remarkably, I found scarce documentation, if any, about VS-16 and its effects in terminals. The absence of documentation on this matter was the primary motivation for writing this article.

Wezterm, for example, excels in complying with all other Unicode specifications outlined in this article and tested by ucs-detect. However, like 16 other terminals tested, it falls short in supporting VS-16. These emojis are consistently occluded by the next character, even when in sequence with VS-16.

Depicted here in Wezterm is U+23F1 "Stopwatch" followed in sequence by U+FE0F "Variation Selector-16". However, the stopwatch is displayed as Narrow. Wezterm does however do a good job of scaling the font to fit within a single cell, while most other terminals cause it to be partially occluded by any next character.

Emoji ZWJ

U+200D "Zero Width Joiner" is a special character facilitating the reduction of multiple emojis into a single representation that embodies their combination. This feature resembles a special case of combining, but it is encoded in a completely different manner.

The python wcwidth Specification on "Width of 0" reads:

> Any character following a ZWJ (U+200D) when
> in sequence by function wcwidth.wcswidth().

An instance of a terminal lacking ZWJ support is Kovid Goyal’s kitty. It's important to note that this terminal should not be confused with KiTTY, another terminal emulator sharing a similar name but predating it by 14 years. Mr. Goyal expresses particular hostility about this naming conflict.

• python wcwidth.wcswidth() measures width 2, while Kovid Goyal's kitty measures width 6.

In this kitty example, the depicted sequence is expected to measure a width of 2. However, kitty measures it as 6 because it does not interpret the Zero Width Joiner character to reduce the three wide characters into one.

Concluding remarks

I intend to use this article as a reference when filing bug reports in open source projects. I hope you appreciate the effort invested in writing a clear Specification within the python wcwidth library and the ucs-detect tool, systematically testing terminals for compliance with the specification.

Additionally, it is worth nothing that the python wcwidth project systematically generates code lookup tables for Wide, Zero-Width, and VS-16 sequences. These tables are created using update-tables.py, which fetches the latest data from unicode.org. The project utilizes jinja2 templates to transform that data into Python code.

This can be easily extended for languages like C/C++, Rust, Ruby, Go, or any other. Feel free to contribute new code templates to wcwidth project for seamless integration with your preferred language.

Finally, I strongly advocate for Python to internally implement some version of wcwidth. Functions like str.ljust(), textwrap.wrap(), or format strings such as f'{my_string:<{width}}' should inherently account for the width of non-ascii characters when formatting strings. Presently, these functions rely solely on the count of characters without understanding their printed width. I believe this adversely affects many developers who discover 'the hard way' that an external library is necessary. Given that wcwidth is downloaded over 50 million times per month, incorporating this functionality into Python should be a sound and economically sensible decision.

I've discovered a Draft standard for C++, P1868R0 that proposes adding this support, and I wholeheartedly endorse this direction. While I'm unsure of its acceptance, I'm inclined to submit a similar proposal for the Python language (Issue #94). Equipped with a concise Specification, I encourage fellow developers to embark on similar initiatives for all modern programming languages.