I have been working on a library called Bunsho and experimenting with ways to handle text parsing generically.
Bunsho has an interface called Moji. The interface handles classification and transformation of individual runes, represented as u21.
I usually use char to mean u8 and rune to mean u21, similar to Go. I think Zig adopting similar terminology could be useful, but that is a separate discussion.
The idea behind Moji is to let text-processing code operate on runes without hard-coding one particular character repertoire.
A Moji implementation defines things such as whether a rune is alphabetic, numeric, whitespace, a symbol, or a control character. It also provides casing and optional conversions such as ASCII representations and identifiers.
Code consuming the interface does not need to know whether the implementation only supports ASCII or supports a larger subset of Unicode.
This is somewhat similar in spirit to interfaces such as Allocator or Io. The caller supplies an implementation defining the policy, while generic code works through the common interface.
An interesting consequence is that the implementation can also be changed while parsing. For example, a parser could use one Moji implementation while parsing source code and another while parsing a comment or another embedded language.
Bunsho currently provides two implementations: Ascii and UnicodeSubset.
The purpose of UnicodeSubset is not to provide full Unicode support. It intentionally implements a small set of useful rune-level properties without depending on the full Unicode database.
The current table-based sets are:
cyrillic
greek
european_latin
hiragana
katakana
old_futhark
Some other character groups can mostly be represented using ranges rather than explicit tables, such as Chinese characters and Japanese kana.
The current tables used by Bunsho are only around 352 lines, so the amount of data remains relatively small.
Because Moji operates on a single u21, it intentionally does not attempt to handle higher-level Unicode concepts such as grapheme clusters. Full Unicode normalization, grapheme segmentation, and similar functionality would still require a more complete Unicode implementation.
This made me wonder whether an abstraction similar to Moji could be useful in Zig’s standard library.
Zig already provides low-level Unicode encoding and decoding support through std.unicode, as well as ASCII-specific helpers through std.ascii, but it does not attempt to provide the large Unicode property tables required for complete Unicode text semantics.
A small interface could provide a middle ground.
The standard library could define the interface and perhaps provide implementations for ASCII and a deliberately limited Unicode subset. Applications requiring full Unicode behavior could provide their own implementation backed by a separate Unicode library.
If standard-library functions accepted such an interface, they could provide some generic character classification and transformation functionality without requiring std itself to contain the complete Unicode database.
The important part would be documenting very clearly that the provided Unicode implementation is only a strict subset, and that applications requiring full Unicode behavior must provide their own implementation.
The current interface in Bunsho looks roughly like this:
//! Interface for handling Unicode subset identification.
//! Only handles single runes (u21).
//! Moji base coverage is limited to basic symbols, without variations or formatted versions.
//! The interface is meant to provide a common way to implement rune handling.
//! Wider coverage should be implemented by the user when needed.
const Moji = @This();
pub const Case = enum {
unsupported,
none,
uncased,
upper,
lower,
};
vtable: *const VTable,
pub const VTable = struct {
is_alpha: *const fn (*const Moji, r: u21) bool,
is_digit: *const fn (*const Moji, r: u21) bool,
is_symbol: *const fn (*const Moji, r: u21) bool,
is_whitespace: *const fn (*const Moji, r: u21) bool,
is_control: *const fn (*const Moji, r: u21) bool,
casing: *const fn (*const Moji, r: u21) Case,
to_lower: *const fn (*const Moji, r: u21) u21,
to_upper: *const fn (*const Moji, r: u21) u21,
to_ascii: *const fn (*const Moji, r: u21) []const u8,
from_ascii: *const fn (*const Moji, cs: []const u8) ?u21,
to_ident: *const fn (*const Moji, r: u21) ?[]const u8,
from_ident: *const fn (*const Moji, cs: []const u8) ?u21,
};
/// Is rune alphabetic or letter
pub fn isAlpha(self: *const Moji, r: u21) bool {
return self.vtable.is_alpha(self, r);
}
/// Is rune digit
pub fn isDigit(self: *const Moji, r: u21) bool {
return self.vtable.is_digit(self, r);
}
/// Is rune symbol
pub fn isSymbol(self: *const Moji, r: u21) bool {
return self.vtable.is_symbol(self, r);
}
/// Is rune whitespace
pub fn isWhitespace(self: *const Moji, r: u21) bool {
return self.vtable.is_whitespace(self, r);
}
/// Is rune a control symbol
pub fn isControl(self: *const Moji, r: u21) bool {
return self.vtable.is_control(self, r);
}
/// Runes casing type
pub fn casing(self: *const Moji, r: u21) Case {
return self.vtable.casing(self, r);
}
/// Format rune to small case
pub fn toLower(self: *const Moji, r: u21) u21 {
return self.vtable.to_lower(self, r);
}
/// Format rune to big case
pub fn toUpper(self: *const Moji, r: u21) u21 {
return self.vtable.to_upper(self, r);
}
/// Format rune to ascii representation
pub fn toAscii(self: *const Moji, r: u21) []const u8 {
return self.vtable.to_ascii(self, r);
}
/// Format ascii representation to rune, first found is returned
pub fn fromAscii(self: *const Moji, cs: []const u8) ?u21 {
return self.vtable.from_ascii(self, cs);
}
/// Returns the identifier for the rune, or null if no identifier is available.
pub fn toIdent(self: *const Moji, r: u21) ?[]const u8 {
return self.vtable.to_ident(self, r);
}
/// Returns the rune matching the identifier, or null if no match is found.
pub fn fromIdent(self: *const Moji, cs: []const u8) ?u21 {
return self.vtable.from_ident(self, cs);
}
For example, the same interface can be used like this:
test "UnicodeSubset implements Moji interface" {
const subset = UnicodeSubset{};
const m = subset.moji();
try std.testing.expect(m.isAlpha('É'));
try std.testing.expect(m.isAlpha('Ж'));
try std.testing.expect(m.isAlpha('カ'));
try std.testing.expect(m.isDigit('7'));
try std.testing.expect(m.isSymbol('?'));
try std.testing.expect(m.isControl('\n'));
try std.testing.expectEqual(Moji.Case.upper, m.casing('É'));
try std.testing.expectEqual(Moji.Case.lower, m.casing('é'));
try std.testing.expectEqual(Moji.Case.uncased, m.casing('カ'));
try std.testing.expectEqual(@as(u21, 'é'), m.toLower('É'));
try std.testing.expectEqual(@as(u21, 'É'), m.toUpper('é'));
try std.testing.expectEqual(@as(u21, 'カ'), m.toLower('カ'));
try std.testing.expectEqual(@as(u21, 'カ'), m.toUpper('カ'));
try std.testing.expectEqualStrings("e", m.toAscii('É'));
try std.testing.expectEqualStrings("ka", m.toAscii('カ'));
try std.testing.expectEqual(@as(?u21, 'A'), m.fromAscii("A"));
try std.testing.expectEqual(@as(?u21, 'か'), m.fromAscii("ka"));
try std.testing.expectEqualStrings("Alpha", m.toIdent('Α').?);
try std.testing.expectEqual(@as(?u21, 'α'), m.fromIdent("Alpha"));
}
I am interested in whether this kind of interface makes sense as a general abstraction, and particularly whether something similar could be useful in std while keeping the actual Unicode data deliberately small.
I am aware that this interface has problems and is not a perfect solution, but I think it could be a better middle ground than what std provides today.