Accessible immediate-mode GUIs

Subclassing is needed if you don’t have control over the window at all, or you for whatever reason cannot override the message loop (e.g., with SDL3 and an ordinary SDL_CreateWindow call). Someone may for example wish to have SDL3 handle all the windowing/graphics for them, and just have the UI be a visual overlay within the window. In that context, you can’t override the window procedure because that breaks SDL, but you also can’t intercept window events either without subclassing the window first.

With respect to testing, I’d be happy to (dvui doesn’t build under Zig 0.16.0-dev.747+493ad58ff right now though). I’m blind myself.

As for tree views being supported, that must’ve been a relatively new addition; Godot also uses AccessKit and I noticed that tree views functioned like tables and node information (collapsed, expanded, node levels) weren’t communicated to NVDA last time I gave it’s beta a try.

2 Likes

Will definitely take you up on testing, but maybe more useful currently would be helping me understand from a user’s standpoint how this is used. Can you speak to these questions:

What is a “normal” (if any) workflow for screen reader navigation? Is it to read the whole window first? Do you use keyboard navigation (tabbing) or use the accessibility navigation stuff (like next section/heading, next paragraph/word, etc.)?

I’ve been reading accessibility information, but still don’t know how it works in practice. I’m particularly unsure how the alternate input stuff works.

Any good intro information or pointers about this would be useful - thanks!

Yes, I second that. Any help will be most welcome. Thank you!

So, typically what happens depends on the type of dialog:

  • If the dialog is, say, an install wizard dialog, or a message box or similar, the flow is simple: the screen reader will read the title bar, the main text (if any), and then the currently focused widget will be read after that. Tab/shift-tab is used to navigate among controls in their appropriate tab order.
  • If the dialog is a web view, the navigation is a lot more complicated. Screen readers (at least on Windows) use an internal virtual “buffer” of sorts to understand the construction of a web page (landmarks, headings, sections, lists, links, etc.). As such, far more navigational options are made available to you: the b key navigates among buttons; the h key goes by heading; the t key goes to the next table; etc etc etc. I however get the feeling that you won’t be writing a full webview implementation so this isn’t much of a concern.

The “main text” of a dialog is a bit subjective and hard to nail down, and screen readers have different ideas as to what this actually is. Something similar happens with labels for controls: if a control (say, an edit box) has a label above it, the distance between the labelled control and it’s label have to be close together (I think the max is 10-15 coordinates), otherwise the screen reader won’t be able to automatically associate the two. I’m not honestly sure why this problem exists, or what it’s parameter space is. However, if you don’t want to deal with this problem, set the accessible name of the control to whatever the label is. This is one of the accessibility properties which you should generally expose to consumers of the library.

The accessible name is the name of the control which will be exposed to accessibility clients. When set, this overrides any label detection logic the screen reader might ordinarily employ. Thus, the most ideal is to set this to either the label of the control as it would visually appear to end-users, or to set it to something which is equally as descriptive.

The next property is the accessible description. This usually isn’t set by many applications, but the general idea is to supplement the label. As an example, this is used in Reaper, in it’s preferences dialog: if I go to the audio settings and tab to the audio system combo box, NVDA will say “Audio system: combo box WASAPI collapsed Choose your desired audio system here. By default, REAPER uses WaveOut for compatibility, but ASIO is recommended for best performance.” instead of just “Audio system: combo box WASAPI collapsed” as it would were an accessibility description not set.

The last property in this trio is the accessible role. Client applications (applications consuming dvui or similar) should rarely, if ever, set this property, and I strongly discourage anyone from doing it without an extremely good reason. UI frameworks such as .NET’s Windows Forms allow you to do this for any control, as an example. .But you shouldn’t ever do this beyond what the UI framework does, because this changes how the screen reader reports the control to the user. But it also changes how the screen reader interacts with the control. For example, if you have a button but you tell the screen reader that it’s a spin button, the screen reader will treat it like a spin button: it has a value associated with it, you can use the arrow keys to change the value, or the left/right arrows to navigate by character, or you can enter a custom value. Naturally, a button won’t exactly react well to this kind of interaction. You should expose it but put all kinds of red flags around it, since this property should only be used if there is no other possible way of getting accessibility clients to properly interact with the control the way you’d like. Otherwise, it should remain unchanged.

I know this is significantly more than you asked for, but I figured the more information the better.

Edit:
I also want to note that there are controls that the UI framework should NOT expose in tab order. Labels, for example. This usually ends up confusing people instead of being helpful. So in those instances, clear the focusable attribute on those widgets.

The NVDA screen reader has a way of allowing you to hear the accessibility tree, at least sort of. In any application, a feature called object navigation can be used to navigate any window, all the way up to the DWM/compositor. Orca on Linux also has a similar feature. If you ever want to test NVDA with dvui, you can use object nav in both laptop or desktop mode:

  • In desktop mode, you use the numpad. Numpad 8, 4, 6, and 2 navigate via object nav.
  • On laptop, caps-lock + shift + arrow keys. is used to navigate among objects. In an edit field, caps-lock and up/down/left/right is used to navigate the field even when focus is not set to it using what is known as the review cursor.

I hope all this helped! :slight_smile:

4 Likes

This is amazing thank you so much! This is exactly the kind of overview I was missing.

I’m going to have to go over this a few times while testing stuff to make sure I’m understanding what you wrote. Especially thanks for the pointer about the “object navigation” of the accessibility tree, will check that out. Great info about guiding users towards setting the right name/label, but guiding them towards not overriding the role.

Thank you again!

Of course! NVDA has a bunch of keyboard shortcuts like that to help you either figure out what a window has (in the case of something not presenting stuff via the keyboard as it normally should) or, in your case, for debugging/diagnostic purposes. You can find all of the keyboard shortcuts in the NVDA documentation, but I’ll reproduce all the ones you’ll need here in case it helps. In the below, the NVDA key is either caps lock on the laptop keyboard layout or insert on the desktop one. You can mix and match both, since the desktop ones are always available.

Command Desktop keyboard shortcut Laptop keyboard shortcut Touch gesture
Moves the navigator object to the first object inside it NVDA+numpad 2 NVDA+shift+down arrow flick down
Moves the navigator object to the next object NVDA+numpad 6 NVDA+shift+right arrow 2 finger flick right
Moves the navigator object to the object containing it NVDA+numpad 8 NVDA+shift+up arrow flick up
Moves the navigator object to the previous object NVDA+numpad 4 NVDA+shift+left arrow 2 finger flick left
Performs the default action on the current navigator object (example: presses it if it is a button) NVDA+numpad enter NVDA+enter double tap
Pressed once sets the keyboard focus to the navigator object, pressed twice sets the system caret to the position of the review cursor NVDA+shift+numpad minus NVDA+shift+backspace
Sets the navigator object to the current focus, and the review cursor to the position of the caret inside it, if possible NVDA+numpad minus NVDA+backspace

There are a bunch more that you probably won’t need. The last two are particularly useful if you find you’ve strayed out of the window and can’t find your way back (this happens to me a lot, and I’ve been using NVDA for years lol).

5 Likes