Sensitivity analysis

A key skill in data is being able to simplify analysis and know when to stop digging. This is particularly true in the data team at Monzo, where we’re trying to move fast and prioritise having impact over perfection. We often have to strike a balance between precision and speed when looking into something. Simplifying things is useful for you (helping you prioritise what to dig into and include, reducing your workload), and for your stakeholders (helping them understand your conclusions and key caveats or risks to your analysis).

Like many data people, I have a tendency to get trapped in the details trying to analyse every single thing that might impact a situation - commonly known as “getting stuck down a rabbit hole”. One tool I find really useful to help me simplify things is sensitivity analysis, which is where you quantitatively assess how much impact changing each variable has on the final outcome.

Applying sensitivity analysis to workforce allocation

Let’s say your business stakeholder brings you a proposal to move manual work that’s currently done by one team to another. The new team is cheaper, but for regulatory and training reasons they won’t be able to do all the tasks the current team can do, so they’ll have to transfer some of them back to the original team.

Any change in workforce allocation is full of complex moving parts. The new team could be faster or slower than the original team, they could transfer more or less than we expect, the amount of time they spend on the transferred tasks could be higher or lower than other tasks, the quality of the work could change, they might have more breaks or need more training . . .

A blonde lady is looking very confused with mathematical equations and drawings superimposed over the image

How do you prioritise which factors to report on or include? Which are most important?

You check which ones matter!

Identifying factors that could affect the plan

I was presented with a case just like this in my role in the Customer Operations data team at Monzo. I started out by drawing some pictures of simple scenarios I could imagine happening and calculating the cost impact of each of these. This gave me a list of factors that I thought could affect the cost-effectiveness of this plan:

average handling time of the new team vs the old
transfer rates
average handling time of the new team on transferred tasks
average handling time of the old team on transferred tasks
occupancy of the new and old teams (how much of their scheduled time they spend handling tasks).

In this case we actually ignored quality as we would monitor the output of the new team very closely. So our role was to focus on examining the cost-effectiveness of the move.

Imagine we have 4 tasks that normally take 1 hour each. In the baseline, team A does all of the work, taking 4 hours. They have an occupancy of 80% meaning that translates to 5 hours of Team A’s scheduled time, costing £128. In the ideal scenario for moving the work, team B do it all with the same handling time and occupancy, but cost £72 leading to a £56 saving. However, what if team B can only complete two of the tasks but have to send the others back to team A (after spending half an hour identifying that they can’t finish them). In this case, if both teams have 80% occupancy, we need 3.75 team B hours but also 2 team A hours, leading to a total cost of £105 - still a saving of £23, but only 41% of the total opportunity. And that’s not the only thing that can affect cost. For example, if team B are twice as slow as team A at doing these tasks, then we will need 10 team B hours costing £144 - a £16 loss vs the baseline. Or if team B have much lower occupancy (e.g. if they are scheduled less efficiently) - if team B have an occupancy of 50% instead of 80%, we need 8 team B hours to do the 4 1-hour tasks, and this costs us £115. We only save £13 vs the baseline, 23% of the amount we save if team B have the same occupancy as team A.

Calculating the cost of changing variables

This gave me enough understanding to write out an equation for the change in cost of moving the work with each of these variables. This helped me understand the role of each variable and quickly realise that some of them weren’t linearly related to the outcome, which is always fun.

The baseline cost is the cost of an hour of team A time multiplied by the number of tasks times the AHT of those tasks, and then divided by the occupancy of team A. If we know the transfer rate of team B, and how long it takes both team A and team B to work tasks that get transferred, we can write out the cost of the team B scenario too. The Team B part of the cost is cost of team B hours divided by team B occupancy, times by the number of hours team B need to handle. We can calculate the handling time of team B by multiplying their handling time for completed tasks by the number of completed tasks, and their handling time for transferred tasks by the number of transferred tasks. Team A also have to handle the transferred tasks, so we get this element of cost by multiplying the number of transferred tasks by how long it takes team A to do them, then by the cost of team A, and dividing by team A’s occupancy to account for scheduling inefficiency.

From here I could have gone on to try and remember how to differentiate, which would give me a mathematical equation of how each variable affects the cost. But it was pretty clear that not only was this already hard for me to understand, it would be really difficult to explain to stakeholders and make it tangible to the business.

Instead, I popped together a quick Google Sheets model of the cost impact, with each variable included separately. I could then tweak each variable manually to get a feel for the changes, and even plot the outcome vs the variables in different scenarios.

This helped me narrow down the list of variables to report on to:

average handling time of the new team vs the old (AHT)
occupancy of the new and old teams
transfer rates

And made it clear that the distribution of time spent on transferred vs non-transferred tasks mattered a lot less than AHT (Average handling time) or Occupancy, particularly at the levels of AHT (Average handling time) and Occupancy we were seeing.

Meaningful results for reporting

As a nice side effect of this process, we were able to report the impact of each metric on cost in a way stakeholders could easily grasp; a simple line graph with the metric on one axis and cost on the other. We were also able to keep reporting in terms of cost throughout the project so it was clear which metric was having the biggest impact (and therefore had the biggest opportunity for improvement).

If you’re not sure if a variable is important to the end outcome, check! A simple equation or spreadsheet model can help you answer the question of whether you should care about this and simplify your analysis. Opening a quick Google Sheet and doing some calculations is a go-to for me when I’m trying to understand a new problem, or when I’m in an analysis paralysis hole.

If you're interested in this blog you should check out our Data blogs on how we experiment to measure incrementality and using topic modelling to understand customer saving goals.

And whilst you're here, check out our open Data roles 👇