Measuring, learning and keeping up with AI

Article 3 from the Series: Winning in the data economy

Topics covered:

1. The MVNO as an AI first business
2. Measuring success (and failure)
3. The importance of Experimentation
4. Cultural change: From Knowing to Learning
5. Conclusion

More than ever the DNA of businesses is being scrutinised. In the AI era what separates success and respect from failure and obscurity? What are the indicators that your MVNO is going to succeed in their competitive market place? How do you stack the cards and keep on stacking them in your favour to give you an advantage? And, at a time where it’s hard to know what’s real and what’s fake how do you know that you’re even measuring the real indicators of success?

We are in the AI era, to be a winner in this era one thing is as certain as taxes, and that is that AI needs to be part of your business or these kinds of challenges will be impossible to solve.

The MVNO as an AI first business

An AI first business has a different DNA from traditional and digital businesses. It’s not just a business that has an AI strategy, it’s a business that has prioritised AI as a core component of everything: it’s workflow and processes, its services and its products. 

The DNA of successful AI first businesses also have another common trait and that is that they know the compound business case of AI. They are deploying AI for cost saving and value creation, for efficiency and quality, for profit and sustainability. In other words, they have a business plan and they are clear throughout that business plan how AI is going to enable their business across short term and longer term horizons to support all it’s operational metrics.

To make the point for an MVNO this means you have a business plan that describes proposition, product, distribution, pricing, marketing, the service experience, wholesale commercials etc but interlaced into each dimension is where AI sits – there isn’t a section called AI in your plan it’s throughout your plan.

Your competitive advantage depends on your leverage of AI in creative and commercial ways aligned to your brand.

DNA: An AI Strategy vs An AI first business

Measuring success (and failure)

So how do you know what good looks like? 

When we deploy predictive models we use a variety of metrics: Accuracy (we use scores like AUC – Area under Curve), Precision and Recall. These measures are used to monitor the accuracy of a model and they should be part of your vocabulary if you work in BI or Data Science but they are a technical evaluation of a models performance.

To measure the impact of AI you need a critical eye on your business performance metrics, your team metrics and your financial metrics. That’s because once you have implemented predictive models the real-world impact of the model is the single most important success criteria. It no longer matters what the accuracy score is; it matters that it is helping you to save customers or increase revenue or similar.

We strongly advise rethinking KPIs when you’re moving to an AI first approach. The reason for that is that AI should bring efficiency improvements, quality improvements and greater insights which need more dynamic KPIs, more agile delivery and more alignment across teams.

When we observe the challenge that Opcos have had moving from legacy operating models to AI first operating models it’s very obvious that many of their problems come back to measurement and accountability. One of the classic examples in Opcos is the vanity metric of subscriber base which can be a primary churn driver when the quality of the acquisition is poor.  MVNOs, being agile and more focused should have a natural advantage!

As an MVNO we’d recommend moving to dynamic KPIs that express desired changes and better measure outcomes over time.  The more the KPI creates collaboration across business units and breaks organisational silos the better – without this you’ll end up with ‘data hoarders’ who aren’t acting as great corporate citizens for your business.

The importance of experimentation

Experimentation goes hand in hand with data science. There are several reasons for this:

• More data leads to more insights means more problems to fix
• More change means more uncertainty around cause and effect
• It just makes sense – if you could be confident that you were acting on a killer causal insight and could be confident that your intervention worked wouldn’t you want to be making confident decisions; conversely if your intervention bombs wouldn’t you rather know before you spent $$$$ on putting that into your TV commercial??

Experimentation is so important that major brands that are AI first (Netflix is a great example – interesting article here) they treat their entire product experience and marketing offer catalogue as a continuous stream of experiments. They continue to run experiments and promote new versions of models, UX etc based on an experiment out-performing a status quo version.

The other important aspect of experimentation is that if you are breathing data into everything you do then there is no better way to get more useful data than to try new / more things. Experimentation is not a nice to have; it’s a necessity. And, I’d go as far as saying ‘if you aren’t experimenting then you’re probably not data-lead’ and you are certainly not setting yourself up for success in the AI Era.

Cultural change: From Knowing to Learning

As leaders, it is drilled into us that we should have answers. We should know best because we’ve worked up the ranks to become a leader. But that’s not how AI works; you have to be ok not to understand the algorithm; ok that you can’t describe how it works or how it was built; and you have to happy not to have the answer.  The DNA of AI first businesses that are incredibly successful (putting the tech behemoths like Musk, Bezos and  the Zuck aside) is that their leaders are embracing ‘learning the answer’ not knowing it. That’s the movement from gut (‘I know best’) to data lead decision making that will signal you’re in a data lead business.

Making this change means creating cultural change in your business:

• Embracing imperfection
• Being comfortable when something doesn’t work first time – or the second time
• Embracing experimentation, iteration and continuous change
• Having people that are comfortable with constant change
• Being the master of manageable scope – starting small

This is a profound change for businesses but an equally significant change for leaders in your business. If you can, invest in upskilling yourself and your leaders so that they can lead in the complex era of AI and turn your business into an AI lead business that can win through innovation and agility in the data economy.

Conclusion

This three part blog series shared my experience working with MVNOs and Operators that are winning in the AI era. There are fundamental things that separate the winners. The winners have found ways to use data as a differentiator – even a revenue stream. They have foundational capability around telco data platforms and nurture their data. They leverage their data comprehensively across their product and value chain so that their investment delivers them compound benefit and they continue to invest and reinvest. Importantly, those winning in the AI era have one other commonality – they have created the right culture and operational model coalescing around performance metrics that allows them to turn data into an advantage.

Winning in the data economy

Thanks for reading this three part blog series. Feel free to be in touch if I can help you transform your business and build your competitive advantage with AI.