Assignment
Enable remote access to job candidate assessment test materials, update platform front-end technology, identify opportunities for product improvement
Clients
Aon Assessment Solutions for FBI, US Marshal Service, TSA, Bank of America, CVS, Honeywell, and Ford
Responsibilities
UX research and design, front-end development, management of content designers and front-end developer
Collaborators
Full stack developers, IT project manager, business stakeholders
*Some images are intentionally altered or blurred to protect intellectual property.
Existing Product
Aon business leaders approached my team to update an existing Assessment Solutions product: a telephone role-play assessment in which job candidates study scenario-based test materials then demonstrate their knowledge in phone calls with various role-play assessors. At the time of the request, the product was available as a remote test using Adobe Flash-based study materials or in person using printed study materials.
Business Needs
Business stakeholders brought three key points to our attention:
1. Pandemic restrictions required all further testing to be conducted online.
2. Adobe Flash was approaching end of life, warranting an update to HTML and CSS.
3. Clients were providing feedback we could leverage for iterative product improvement.
Problem Statements and OKRs
In collaboration with business stakeholders, we identified four main problems to address. We formulated statements that focus on the needs of our clients and their job candidates. For each problem statement, we wrote a corresponding objective and key result, later used to measure the success of the solution.
Restructure and Redesign
We deconstructed the test materials for all existing clients and spoke with client consultants to understand the requirements for the redesign:
• Render on all desktop screen sizes
• Simulate emails with attachments, memos, and voicemail transcripts
• Include welcome screens, role-play agenda, charts, tables, and other reference materials
• Easy to navigate and find specific content
• Include an onboarding sequence/tutorial
• Easy for candidates to get help
• Support client theming/branding
We designed client themes with front-end development efficiency in mind. Colors, fonts, and images are all SCSS variable-based. New themes can be generated within a matter of minutes by using entering two colors, one font, and image sources for logo and header into one SCSS variable file.
Usability Testing Feedback
During product validation, employees currently holding the same or similar position as the role being applied for are asked to participate in a test run of the assessment to measure the psychometric validity of test materials. During this phase, we surveyed participants to collect feedback on the usability of the redesigned UI.
One of the features we improved based on feedback was the way email attachments open in the remote testing environment. Our clients observed that participants were attempting to open attachments by clicking on the icon next to the attachment title. To meet users' expectations for the behavior of UI elements, we added a modal click trigger to the icon. Also, based on feedback about ease of consuming information within attachments, we added additional modal window sizes to accommodate different types of content so candidates could more easily read information.
Feature Prioritization
Survey feedback and client meetings yielded edits and features our clients wanted us to implement before bringing testing back online. For every day testing was on pause, we were losing revenue. Collaborative feature prioritization enabled us to narrow scope so we could edit the redesign and complete implementation within a few weeks, bringing testing back online earlier than business leaders expected. By project management calculations, we saved ~280 hours of design, development, and QA time.
Measuring Success
After launching the redesign, we referenced our original OKRs (objectives and key results) to measure the success of the implemented solution.
Remote Test Materials
We finished out 2020 recovering $420,000 of revenue from the FBI that would have otherwise been lost to the pandemic downtime. In addition we gained roughly $48,000 over 2019 FBI numbers, bringing our total 2020 FBI earnings to about $653,000.
Technology Update
All updates were made in between job candidate access times so that none of our clients experienced downtime nor incurred any additional expenses. Our updated security features actually provided a cost savings by removing the need for clients to obtain proctoring resources for onsite testing.
Content Restructuring
Role-play assessors and client testing coordinators provided feedback that candidates have continued to demonstrate knowledge of test materials. Additionally, we surveyed 12 validation test participants who all found the content easy to navigate.
Visual Appeal
Our redesign caught the attention of senior consultants who requested an off-the-shelf version of the product they could easily sell to multiple clients. Ford and Honeywell were the first adopters of this version. Additionally, an existing client we would have otherwise lost to a competitor agreed to renew their contract with us upon viewing the redesign.
Next Steps
After the initial redesign, we used additional feedback to collaborate with stakeholders on new features that helped to offer our clients new kinds of testing solutions.
We continue to iterate on the product by gathering feedback, prioritizing potential features, and proposing updates to business stakeholders and clients.
