Usability testing with disabled participants

I recently organized and conducted a series of usability tests with disabled participants for WordPress.com. This was the first time such tests were conducted at Automattic, which meant I had a lot of new ground to cover, but fortunately I didn't have to start from scratch.

The Web Accessibility community is an incredibly generous one, and there is a wealth of information and knowledge at your fingertips. In that same spirit, I'd like to share here what the process was like for me, and what I learned from it.

Recruitment

Having just returned from AccessU, an accessibility-focused conference organized by Knowbility, I reached out to them to use their recruitment services. The folks at Knowbility were great to work with and the whole process was very smooth and professional.

One massive benefit of working with them is that you can get as granular and specific as you need when screening participants. They help you recruit for a variety of disabilities, assistive technologies (screen readers, etc.), device settings, and even diversity in gender, age and location. Some of the specifics are:

• Low vision

  • Screen magnification

  • Inverted colors

  • High contrast

  • Bigger text size

• Blind

  • AWS

  • NVDA

  • VoiceOver

  • Braille

• Mobility impaired

  • Keyboard with mouse

  • Keyboard with alternative pointing device

  • Keyboard-only

  • Speech input

• Cognitive disability

  • Attention

  • Memory

  • Processing speed

  • Time management

  • Letters and language

  • Numbers, symbols, or math

  • Making/understanding choices

• Deaf or hard of hearing

  • ASL services can be provided

Two of the five participants from our pool were very technical and worked in one way or another as accessibility consultants. There was nothing wrong with that, as those sessions proved to be very informative, but still something to keep in mind depending on what you’re trying to achieve with your tests.

Conducting the tests

As with any usability test, you’ll want to have a plan and be fully prepared and unless requested, your participants should not be treated any different. That being said there’s a few things to keep in mind.

Time of task completion is going to vary widely

Just because someone has an impairment does not mean they will be slow. I found that some were very experienced with their devices and tools and they got past the flow we were testing in a breeze. So be prepared and plan ahead for both scenarios where some participants will complete the task quickly and others will need more time.

Let them use the device, assistive technology and settings they’re most comfortable with

It is extremely important that your participants are able to use their own devices, assistive technologies and settings they’re most comfortable with. Do not ask them to change any settings (zoom level, contrast, etc.) or turn off their assistive technology.

If doing a remote session, you won’t be able to see and experience what’s being tested like they are

If you’re using Zoom or any other video conferencing service for a remote usability test, it’s very likely that what you see in the video feed is not what they are seeing. Assistive technology, such as ZoomText, and the operating system’s accessibility settings add a layer that does not get piped to the video feed. It’s important that you get as much detail as possible from your participants.

Takeaways

I'll start by saying that I highly encourage everyone working on digital products, and in particular on the Web, to include disabled participants in your usability tests. Good accessibility is good usability and what we learn from these tests will help us make our products better for everyone.

Untested assumptions are dangerous

The way we all experience the Web is vastly different. Just because someone has an impairment doesn't necessarily mean that they will use assistive technology, or that they even know how to or want to change and adapt their device settings. We can't make any assumptions and instead we need to test and learn.

Unnecessary challenges

Some folk face big challenges every single day while trying to use the products and services that we create. It doesn't have to be this way, we make it be this way. We are creating these barriers and obstacles by not designing for everyone.

So I hope this article helps you get started if you haven't yet. I'm confident that you'll find that the value that you get from these tests will be priceless.