Measuring citizen satisfaction

It's important we're able to show our department clients and citizens how we improve digital services over time. To do this, we have 4 key performance indicators to measure their performance. These are: number of completed transactions, completion rate, digital update and citizen satisfaction.

The citizen sentiment has been a challenging metric to nail down, but we've made headway on a few fronts this past year.

Revisiting our existing feedback form design pattern

We have a feedback design pattern departments should apply to their websites and digital services. A person can select a thumbs up or thumbs down to indicate their satisfaction and then they can tell us what worked well or what we can improve.

This has worked really well and the data has been helpful, but hasn't been the best at getting insights into citizen satisfaction.

Testing a new feedback form design pattern

We looked at how we could improve the quality of user feedback data we receive so we redesigned the feedback form and we've started to test it out. There are 6 questions in the form and it takes people less than a minute to complete on average. The data we've collected so far is providing us with a much clearer understanding and metric of levels of satisfaction. Most questions use a 5-point Likert scale and there are two optional, open-ended questions where people can elaborate on their experience if they like.

We seem to be on the right track with this and will continue testing it before we add it to the feedback form onlone design pattern.

Looking at staff's capacity to review, analyze and action user feedback

Every team we've partnered with agrees on the importance delivering an excellent digital service experience. We all agree we on the importance of reviewing user feedback, analyzing it and implementing solutions to address issues. The main issue in accomplishing this is most teams manage the digital service in addition to the in-person service and their other job duties. 

How much time does it take to complete this work?

To answer this question, I looked at:

• Yukon.ca and the two-person team that manage it;
• the Yukon.ca assessment work completed earlier this year;
• people who manage the day-day oversight of a typical digital service; and
• the work I do in my role as the user experience manager.

Yukon.ca team

The team that oversee Yukon.ca content do a great job managing the user feedback. Each month they review it, remove spam and any submissions with PI and they share it with departments and provide advice on how to resolve issues. To do this can take one person between 2 and 3 days per month. 

Yukon.ca assessment work

In this instance we hired a vendor to complete an assessment of the website. One piece of this user research included a review of user feedback from 2023. We tasked them with identifying themes so we could get their assessment and prioritize future improvements to the website. This was rolled into an analytics review as well, but both tasks took their specialized team 1 week to complete.

An average employee who manages a digital service

For other websites and services, the person in charge of the day-to-day collects user feedback. They often manage the online service as well as the in-person service. It can be challenging for them to find the time needed to consistently review feedback, identify themes, determine appropriate solutions to address issues improve the user experience. For these people the main issues are finding the time, and accessing the expertise to know what to do with the information. 

eServices UX manager

As the sole UX manager I do make time to review user feedback, but as the government delivers more and more digital services to meet citizens' expectations - it's challenging to find the time to do this work. My need is to have a way be able to identify if a service is underperforming so I can work with that team to address the issues. This will allow me to be more strategic about where I spend my time. 

Finding a solution to address our business needs

As a government we need solutions that will:

• reduce the time spent on tasks like manual review of data;
• be accurate, reliable and repeatable; and
• allow us to look at user feedback through multiple lenses.

Connecting with the institute for Citizen-Centric Services (ICCS)

The Institute for Citizen-Centric Services (ICCS) conduct a Citizens First survey in Canadian jurisdictions every few years. This helps us understand what people's expectations are when it comes to accessing and using the government's digital services. 

Through our calls with the ICCS we met a team who work in the machine learning, data analytics and natural language processing fields. We partnered with them to see if we could apply LDA and LDI (keyword topics) to parse Yukon.ca user feedback and generate feedback themes. From there we created a custom text model for the Government of Yukon and we've recently started using BERT + sentences (Google natural language processing) to review the feedback, generate themes and nail down a citizen satisfaction metric.

Over the course of the last couple of years we learned the following.

• Using these methodologies did save a lot of time - especially for services like Yukon.ca that see hundreds of pieces of feedback each month. 
• The results are accurate and consistent with manual analysis completed by government staff and vendors.
• Cleaning up data to remove spam and personal information still takes time. We're looking at solutions to reduce this.
• One we upload the data - we can use it to run countless reports. For example I can create a report to flag themes like content quality, accessibility and ease of use. The people who manage emergency information about topics like wildfires and floods can create reports to look at those themes. 
• We do get a better picture of citizen satisfaction. With a new user feedback form this will be even more accurate.
• This creates capacity for the user experience manager to strategically target working with teams to improve services that need it the most.

All of this work has culminated into formalizing a process for service owners to track and report on their user feedback. We are still in the early stages, but we've started to bring on other department clients who've already launched services and collect this data. You'll see guidance in the Digital Service Delivery Guide in the coming months.