Decode 64 byte blocks of valid UTF-8 containing no ESC using the
approach of simdutf: blocks are validated with the lookup tables of
Keiser and Lemire and converted twelve bytes at a time using shuffles
looked up from a mask of the bytes that end code points, with fast
paths for runs of ASCII, two byte and three byte sequences. Anything
else, ESC, invalid UTF-8 and the end of the input, is decoded as
before.
The shuffle tables are generated by gen/utf8_decode_tables.py.
This decodes the text of the unicode benchmark 2.6 times faster, faster
than simdutf's own conversion with error checking, and speeds up the
unicode benchmark by ~8%.
The new AVX2 decoder is faster than the AVX-512 one, so remove the
AVX-512 decoder and use the AVX2 one on CPUs with AVX-512 as well.
Also add a test comparing the SIMD decoders with the scalar one on long
inputs, which exercise the decoding of blocks, as the existing tests use
short inputs.
- Check for multicell chars in the erased range before looking for them
cell by cell, with a branch free check that reads only the word of each
cell containing the multicell flag. The common case is a line with no
multicell chars.
- Add a fast path that parses CSI codes of the common form, an optional
private marker, numeric params separated by ; or : and the final byte,
in a single pass, falling back to the generic CSI parser for anything
else.
- Do not call the UTF-8 decoder for no text when an escape code directly
follows another one, and start parsing CSI codes right after ESC [
rather than in the next iteration of the parse loop.
This speeds up the CSI benchmark by 25% to 30% and the truecolor
benchmarks by ~10%.
A differential test compares the fast path and the generic CSI parser on
random input, and a new test covers erasing parts of multicell chars,
which no existing test did.
Implement revision 0.4 of the Program Status Protocol, which lets a
program report whether it is idle, working, done, blocked on the user
or failed, and why.
- Parse and validate reports per the spec: limits, base64 text without
control characters, malformed ids discard the report instead of
falling back to the root record, unknown states are ignored
- Keep up to 256 records per window (LRU eviction), with clear removing
a record and its descendants and app inherited from ancestors
- Reply to the feature detection query with supported states and kinds
- Lifetime: working/blocked/idle records are dropped at a new shell
prompt (OSC 133;A), done/error are dropped when the window gets focus,
a full reset (RIS) removes every record
- Expose records via tab.program_status in tab_title_template, the
output of kitten @ ls and a new on_program_status_change watcher
Spec: https://www.superlogical.com/rex/docs/build/program-status
Ref: ghostty-org/ghostty#14560, ghostty-org/ghostty#14630
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix dangling text pointer in buffered key events and test CSI param overflow
Key events buffered while waiting for an overlay to become ready stored
the text pointer from the GLFWkeyevent, which is only valid for the
duration of the key callback. Make the buffered events own a copy of
the text and free it in a single place.
Also add a test for the out of bounds write of is_sub_param when a
separator follows the last allowed CSI parameter.
Alpha blending of graphics protocol images and animation frames is
2-3.5x faster and glyph alpha masks are composited onto canvases using
the same vectorized primitives
Optimise the AVX 2 implementation reducing number of operations per block
and also add an AVX-512 implementation used when the CPU supports it at
runtime.
Allow terminal applications to detect when their view is not observable so they can avoid unnecessary rendering work.
Implement mode 2033 and the visibility status query, and report changes from tab, pane, window, and platform occlusion state.
Change the graphics protocol N key from a boolean into a usage-hints
bitmask. Define the first bit as a transient hint, allowing the terminal
to treat the image data as short-lived and apply optimizations such as
skipping disk cache writes.
Propagate the transient hint through frame coalescing and composition, so
a composed frame is transient if any contributing frame is transient.
Modified `start_classification` in `utf8_decode_to_esc` in `simd-string-impl.h`, which now:
Rejects `0xC0`, `0xC1` and `0xF5..0xFF` lead bytes in UTF-8 subsequences.
Enforces special ranges for the second subsequence bytes after `0xE0`, `0xED`, `0xF0` and `0xF4` bytes to prevent overlong sequences, surrogates, and code points above U+10FFFF.
Accumulates UTF-8 validation errors in a single vector to avoid many conditional branches.
Worsens unicode benchmark performance by about 4%.
Needed by the execrable ncurses. Adds an extra branch in the hot path,
sigh. Thanks to branch prediction it doesnt have any measurable impact
on the benchmark, thankfully.