A key skill in data is being able to simplify analysis and know when to stop digging. This is particularly true in the data team at Monzo, where we’re trying to move fast and prioritise having impact over perfection. We often have to strike a balance between precision and speed when looking into something. Simplifying things is useful for you (helping you prioritise what to dig into and include, reducing your workload), and for your stakeholders (helping them understand your conclusions and key caveats or risks to your analysis).
Like many data people, I have a tendency to get trapped in the details trying to analyse every single thing that might impact a situation - commonly known as “getting stuck down a rabbit hole”. One tool I find really useful to help me simplify things is sensitivity analysis, which is where you quantitatively assess how much impact changing each variable has on the final outcome.
Applying sensitivity analysis to workforce allocation
Let’s say your business stakeholder brings you a proposal to move manual work that’s currently done by one team to another. The new team is cheaper, but for regulatory and training reasons they won’t be able to do all the tasks the current team can do, so they’ll have to transfer some of them back to the original team.
Any change in workforce allocation is full of complex moving parts. The new team could be faster or slower than the original team, they could transfer more or less than we expect, the amount of time they spend on the transferred tasks could be higher or lower than other tasks, the quality of the work could change, they might have more breaks or need more training . . .
How do you prioritise which factors to report on or include? Which are most important?
You check which ones matter!
Identifying factors that could affect the plan
I was presented with a case just like this in my role in the Customer Operations data team at Monzo. I started out by drawing some pictures of simple scenarios I could imagine happening and calculating the cost impact of each of these. This gave me a list of factors that I thought could affect the cost-effectiveness of this plan:
average handling time of the new team vs the old
average handling time of the new team on transferred tasks
average handling time of the old team on transferred tasks
occupancy of the new and old teams (how much of their scheduled time they spend handling tasks).
In this case we actually ignored quality as we would monitor the output of the new team very closely. So our role was to focus on examining the cost-effectiveness of the move.
Calculating the cost of changing variables
This gave me enough understanding to write out an equation for the change in cost of moving the work with each of these variables. This helped me understand the role of each variable and quickly realise that some of them weren’t linearly related to the outcome, which is always fun.
From here I could have gone on to try and remember how to differentiate, which would give me a mathematical equation of how each variable affects the cost. But it was pretty clear that not only was this already hard for me to understand, it would be really difficult to explain to stakeholders and make it tangible to the business.
Instead, I popped together a quick Google Sheets model of the cost impact, with each variable included separately. I could then tweak each variable manually to get a feel for the changes, and even plot the outcome vs the variables in different scenarios.
This helped me narrow down the list of variables to report on to:
average handling time of the new team vs the old (AHT)
occupancy of the new and old teams
And made it clear that the distribution of time spent on transferred vs non-transferred tasks mattered a lot less than AHT (Average handling time) or Occupancy, particularly at the levels of AHT (Average handling time) and Occupancy we were seeing.
Meaningful results for reporting
As a nice side effect of this process, we were able to report the impact of each metric on cost in a way stakeholders could easily grasp; a simple line graph with the metric on one axis and cost on the other. We were also able to keep reporting in terms of cost throughout the project so it was clear which metric was having the biggest impact (and therefore had the biggest opportunity for improvement).
If you’re not sure if a variable is important to the end outcome, check! A simple equation or spreadsheet model can help you answer the question of whether you should care about this and simplify your analysis. Opening a quick Google Sheet and doing some calculations is a go-to for me when I’m trying to understand a new problem, or when I’m in an analysis paralysis hole.
If you're interested in this blog you should check out our Data blogs on how we experiment to measure incrementality and using topic modelling to understand customer saving goals.
And whilst you're here, check out our open Data roles 👇