Comparing sites for accessibility
During a discussion of our findings, we talked about how we could compare the Salsa website to other websites to compare our accessibility. We came to the conclusion that it wasn’t possible, even though there are open-source and proprietary products that claim they can tell you how accessible your site is compared to another.
The problem with automated accessibility testing
It all comes down to where that score comes from — which is automated testing tools. This data creates a problem for four reasons:
Automated testing tools only uncover roughly 15-25% of the violations on your site.
It’s fairly easy to manipulate your code, so that an automated testing tool thinks a site is accessible when it’s not.
Depending on the automated tool used, it can be “all or nothing.” If your site was evaluated across 10 pages and you had 1-2 violations across a possible 20 violations, your score wouldn’t reflect that you actually had 18 instances where you had correct code for a guideline.
It doesn’t take into account the complexity of your site components or theme compared to the other sites that you’re being ranked against.
Unfortunately, these issues can lead to either a false confidence for accessibility or can lead you to thinking your site is not as accessible as it is. For example, your site could score a 95% but have a long list of violations that weren’t discovered (but would have been by doing manual testing). Or your site could score a 75% but you could have very few accessibility issues from manual testing, which means that your compliance percentage score is actually higher.
We wanted to make sure that if you see an accessibility compliance percentage score for your site, that it includes all guidelines, all components, and that we’re evaluating all possible violations and giving you “credit” for all items that are compliant. And so the Salsa Scoring System was born!
How does the Salsa Accessibility Score work?
Our use pass, fail, does not apply, or partially tested criteria to evaluate your site components and theme to ensure that you know when a page, or group of pages, does or does not meet accessibility compliance for a guideline. The definitions for these criteria are as follows:
Pass: The site is fully compliant with all tests associated with this standard.
Fail: One or more components or pages on the site failed to comply with one or more tests associated with this standard.
Does not apply: Content or functionality, such as audio, video and sensory characteristics do not exist on any pages of the site, so tests cannot be run.
Partially tested: Automated tools do not completely cover the testing criteria for this standard.
To calculate your Salsa Accessibility Score, we use the following metrics:
Number of tests run
Number of pages and components we audited
Number of possible violations
Number of actual violations
To find the total number of violations, we count how many issues there were and how many pages those violations appear on. For example, if there is one violation of a standard, but that violation occurs on 10 separate pages, that would equal 10 violations.
In order to calculate the Salsa Accessibility Score for WCAG 2.0 AA, we do the following:
|51 tests x number of pages tested = |
number of possible violations
|51 tests x 15 pages tested =|
765 possible violations
|Number of possible violations - number of actual violations = number of items that passed||765 - 220 = 545 items passed|
(Number of items that passed/number of possible violations) x 100 =
WCAG 2.0 AA Salsa Score
(545/765) * 100 =
71.2% WCAG 2.0 AA Salsa Score
Your Salsa accessibility score is a compilation of all 38 WCAG 2.0 AA guidelines or 50 WCAG 2.1 AA guidelines, depending on what set of guidelines we’re auditing against. Several guidelines have multiple items associated with them, so we use a larger set of tests to audit your site and account for this difference in your Salsa Accessibility Score, since some tests within a guideline may pass while others do not.
Your score will be weighted against others based on tests run, pages and components audited, and number of violations, so it’s a true comparison of how accessible your site is compared to others.