Processing payments in Monzo Stand-in

Monzo Stand-in is our backup bank that we can use in rare platform outages. An important function of a bank is our ability to process customer payments. We send and receive these customer payments through different payment schemes including:

Monzo Stand-in is able to process these types of payments. This means that in the event of a major outage of the Primary Platform, customers can continue to use their card and send or receive bank transfers as normal.

To understand how we receive payments in Monzo Stand-in, let’s first cover how we receive them in the Primary Platform. 

We connect directly to payment schemes through software and hardware hosted in our data centres. We then run edge services in the Primary Platform which run in the cloud, whose only responsibility is to connect to the software in the data centres.

These edge services are stateless and have very few dependencies on shared platform components. When an edge service receives a payment request, it then forwards it to payment scheme specific processing logic. These payment processing services do depend on other shared platform services, in order to make decisions on processing a payment. Once the payment request is processed, we return a payment response back to the scheme. 

Now let’s look at how we route payments to Monzo Stand-in. We've designed Monzo Stand-in to accept payment requests directly from our data centres or via the Primary Platform. 

Given Monzo Stand-in can connect directly to the data centres, it may seem unnecessary to also implement routing via the Primary Platform. However, applying routing changes in the data centres is not as quick as applying them in the Primary Platform. We also have a lot more fine-grained control of routing in the Primary Platform, where we can, for example, control routing of inbound and outbound payments separately. 

Specifically, we have added the ability to route payment requests to Monzo Stand-in within the edge services. We’ve decided to do this as the edge services are the earliest service within the payment processing path and also the most resilient to platform failures. Therefore, during an outage we can be quite confident in our ability to quickly and reliably route payments to Monzo Stand-in, even if the main payments processing services or shared platform components are degraded.

If the platform degradation is too severe for even the edge services to run, or if AWS itself is unavailable, Monzo Stand-in can connect to the data centre services directly, bypassing the Primary Platform entirely. 

Being able to route payments to Monzo Stand-in in these two different ways means we can use Monzo Stand-in in a wider variety of situations, not just during a complete platform outage. Most outages won’t be severe enough to affect the edge services, so having an easy way to route traffic to Monzo Stand-in in those cases helps us mitigate any customer impact. 

For each of the payment schemes that we support, we’ve built new payments processing logic in Monzo Stand-in. Building new logic ensures that we don’t share any common points of failure between Monzo Stand-in and the Primary Platform. 

We have many years of experience maintaining payments processing logic in the Primary Platform. The processing of a payment universally involves making a number of smaller decisions, the combination of which decides whether we approve or decline the payment. Once we’ve made a decision, we then need to apply the effects of that decision. Examples of effects are updating an account balance, showing a transaction in the app, and sending a response message back to the payment scheme.

When rebuilding the payments processing logic for Monzo Stand-in we followed a pattern of implementing these smaller decisions as “validators”. For example, when processing a card payment, there is a validator that will check whether the card is frozen, one that will confirm the account is open, and so on. These validators are ran in parallel, ensuring that Monzo Stand-in is able to process payments quickly. Different payment types require different sets of validators.

Although the implementation of payments processing logic is entirely distinct, we ensure that Monzo Stand-in’s payment decisions are aligned with those from the Primary Platform. We do this by shadow testing a proportion of production payments traffic across the supported payment schemes. This means we check the payment decisions being made by Monzo Stand-in against those made by the Primary Platform, and track the differences. We don’t expect the number of differences to be zero, but we do expect it to be inside a tolerable range. We also expect these differences to stay at an explainable level, and investigate any increases to ensure both platforms aren’t drifting apart.

For every payment that we process in Monzo Stand-in, we emit a Monzo Advice – a message that advises the Primary Platform of the inputs and outcomes of processing decisions. We’ve extended the Primary Platform’s payment processing services to understand these advice messages. This allows us to sync the outcomes of Monzo Stand-in payments processing back into the Primary Platform. The Primary Platform does not re-run any decisioning for the payments that we’ve processed in Monzo Stand-in. This means that every payment decision made by Monzo Stand-in is always applied in the Primary Platform. 

So far this sounds quite simple, but in reality payments can be a little more complicated. The payment lifecycle can include a number of different request types. For example, when you’re paying for something at a card terminal, we will first receive an authorization request. If the card terminal disconnects before we’re fully able to return the authorization response, then we’ll receive a reversal request from Mastercard. This reversal request tells us that we now need to undo the payment.

In short, a payment is not just a single request and we may process the same payment multiple times. Prior to the introduction of Monzo Stand-in, all of this processing happened only within the Primary Platform. Here we were guaranteed to have a complete picture of the lifecycle of the payment and all decisions that we have made about it. 

With Monzo Stand-in however, this is no longer the case, as we can switch processing over to Monzo Stand-in at any point during a payment’s lifecycle. Our advice syncing system therefore needs to be able to correctly compute a payment’s final state, using information from both platforms. To achieve this we’ve built our advice syncing system to be tolerant of receiving advice messages in any order. It can also deal with payments where we’ve made decisions in both the Primary Platform and in Monzo Stand-in, and figure out what the correct final state of the payment is.

This way, once we move processing back to the Primary Platform after running in Monzo Stand-in, all payments will be correctly reflected in customers’ accounts

If our Primary Platform has an outage, even a very severe one, we can process customer payments with Monzo Stand-in. 

Once we have restored the Primary Platform, we will sync all payments processed in Monzo Stand-in back to the Primary Platform, ensuring everything is in the correct state.

If you want to join us to build reliable and resilient systems that serves millions of customers, view our open roles in engineering!