20 January 2021
Danielle Sheffler

The new Salsa website was recently released, and we decided to run automated and manual accessibility audits on it to demonstrate to our clients and to ourselves that we understand what it means to build an accessible site.

Comparing sites for accessibility

During a discussion of our findings, we talked about how we could compare the Salsa website to other websites to compare our accessibility. We came to the conclusion that it wasn’t possible, even though there are open-source and proprietary products that claim they can tell you how accessible your site is compared to another.

The problem with automated accessibility testing 

It all comes down to where that score comes from — which is automated testing tools. This data creates a problem for four reasons:

  1. Automated testing tools only uncover roughly 15-25% of the violations on your site.

  2. It’s fairly easy to manipulate your code, so that an automated testing tool thinks a site is accessible when it’s not.

  3. Depending on the automated tool used, it can be “all or nothing.” If your site was evaluated across 10 pages and you had 1-2 violations across a possible 20 violations, your score wouldn’t reflect that you actually had 18 instances where you had correct code for a guideline.

  4. It doesn’t take into account the complexity of your site components or theme compared to the other sites that you’re being ranked against.

Unfortunately, these issues can lead to either a false confidence for accessibility or can lead you to thinking your site is not as accessible as it is. For example, your site could score a 95% but have a long list of violations that weren’t discovered (but would have been by doing manual testing). Or your site could score a 75% but you could have very few accessibility issues from manual testing, which means that your compliance percentage score is actually higher.

We wanted to make sure that if you see an accessibility compliance percentage score for your site, that it includes all guidelines, all components, and that we’re evaluating all possible violations and giving you “credit” for all items that are compliant. And so the Salsa Scoring System was born!

Your Salsa Accessibility Score determines how compliant your website is against all WCAG 2.0 Level AA standards or WCAG 2.1 AA standards and can be compared to other sites we have audited.

How does the Salsa Accessibility Score work?

Our accessibility audits use pass, fail, does not apply, or partially tested criteria to evaluate your site components and theme to ensure that you know when a page, or group of pages, does or does not meet accessibility compliance for a guideline. The definitions for these criteria are as follows:

  • Pass: The site is fully compliant with all tests associated with this standard.

  • Fail:  One or more components or pages on the site failed to comply with one or more tests associated with this standard.

  • Does not apply: Content or functionality, such as audio, video and sensory characteristics do not exist on any pages of the site, so tests cannot be run.

  • Partially tested: Automated tools do not completely cover the testing criteria for this standard.

To calculate your Salsa Accessibility Score, we use the following metrics:

  • Number of tests run

  • Number of pages and components we audited 

  • Number of possible violations

  • Number of actual violations

To find the total number of violations, we count how many issues there were and how many pages those violations appear on. For example, if there is one violation of a standard, but that violation occurs on 10 separate pages, that would equal 10 violations.

In order to calculate the Salsa Accessibility Score for WCAG 2.0 AA, we do the following:

Explanation Example
51 tests x number of pages tested =
number of possible violations
51 tests x 15 pages tested =
765 possible violations
Number of possible violations - number of actual violations = number of items that passed 765 - 220 = 545 items passed

(Number of items that passed/number of possible violations) x 100 = 

WCAG 2.0 AA Salsa Score

(545/765) * 100 = 

71.2% WCAG 2.0 AA Salsa Score

Your Salsa accessibility score is a compilation of all 38 WCAG 2.0 AA guidelines or 50 WCAG 2.1 AA guidelines, depending on what set of guidelines we’re auditing against. Several guidelines have multiple items associated with them, so we use a larger set of tests to audit your site and account for this difference in your Salsa Accessibility Score, since some tests within a guideline may pass while others do not.

Your score will be weighted against others based on tests run, pages and components audited, and number of violations, so it’s a true comparison of how accessible your site is compared to others.

Download a copy of our extensive accessibility audit of the Salsa website to see the Salsa Scoring System in action!

Download our Accessibility Starter Kit

Download a copy of our Salsa Scoring Spreadsheet, which gives you all of the tools to determine your site’s accessibility compliance score against either WCAG 2.0 AA or WCAG 2.1 AA standards.

Download accessibility starter kit