As public services tried to wrest control during the pandemic, local authorities in Essex were forced to think on their feet. To deal with the new airborne virus, they redeployed a data tool they had previously created to spot flood risk areas in Jaywick Sands and mark them for evacuation. This emergency services tool – VIPER – could create a cordon around the epicentre of an outbreak allowing local authorities and responders to view real-time data to spot vulnerable groups and residents who would be directly impacted. “VIPER actually came about partly as a result of Essex area Jaywick Sands’ coastal flooding evacuation experiences,” Ian Davidson, Tendring CEO. 

Likewise, Camden Council – who manage around 600 types of services – drew on preexisting population-level data related to demographics, housing, education, crime, poverty and employment to determine where demand was likely to be greatest during the pandemic, says Tariq Khan, chief digital and information officer at Camden Council.

“These events have definitely reinforced just how transformational data can be,” says Khan. “As such, there feels like much more investment is being made across local government to ensure we have the requisite skills, tools and approach to start baking in data-informed decision-making at all levels.”

However, other parts of the public machine became ragged as Covid-19, Brexit and the fuel crisis bit hard. Everything from bin collections to NHS waiting lists were impacted as public services struggled to manage a shortage of resources and a change in customer base.

Getting a head for figures

The success of the above public initiatives worked by using data with laser-like focus, whereas often public services are hamstrung by how much information they have to collect. “Garbage” data can block access to the strands which could be transformative, says Roger Grimes, data-driven defence evangelist at KnowBe4. “Public services do eventually figure out solutions, but it's often two or three years after the problem first started. By the time they get involved, it’s at crisis level. It's very tough and slow to steer big ships in new directions.”

For instance, while government schools struggled to monitor pupils during online learning – with 100,000 at-risk pupils ‘lost’ during the pandemic and around a third of lessons lost overall – private companies, who were already entirely digital flourished. Quizlet – a free, digital revision cards system for shared learning – exploded in growth during the pandemic and was also able to continually survey their 60 million global users to gather instant information. This improved their own service as well as the lives of their pupils. They could acquire data as disparate as the impact that access to laptops was having on grade achievement, the rise in students wanting to pursue medicine during the pandemic, and – when they returned to schools – the 60% spike in students studying at the weekend to catch up on studies. 

Other private companies were showing how data could improve services. While the UK government version cost £37bn and stalled badly, Irish firm Nearform managed to create a track and trace app in just a few weeks, which became the world's most effectively deployed contact tracing app reaching 55 million people. The same firm is also producing a data-driven system to eradicate kidney disease, combining diagnostics, automated analysis and reporting to identify patients at most risk of developing chronic kidney disease. 

“Bringing a diagnostics-based product to the healthcare market is hard,” says Larry Breen, head of health and life sciences at Nearform. “It means combining strong diagnostics data, automated analysis and reporting with a robust technical solution, while making sure the whole service is accessible and useful at the same time,”

Adaptability is also important. Adam Hurst, chief technical officer at Log My Care, says private companies are not restricted by archaic systems that reduce the sharing of data. Log My Care is a digital care management platform that removes the need for paper records and, during the pandemic, was able to add Covid dashboards so users could monitor and report everything from coughs to PCR test results. “(In the future,) social care providers will be able to analyse the risks for each of their service users, allowing for early health interventions which could reduce the need for unexpected hospital visits or expensive, reactive medical treatments,” says Hurst.

With data, a problem shared is a problem solved

But while the private sector was able to move quickly, local councils are also gaining traction by sharing their data. As profiled in a report by the Centre For Data Ethics And Innovation (CDEI) on how data was shared by local councils during the pandemic, Hackney Council combined internal and external datasets for the first time to help them identify residents who are particularly vulnerable to Covid-19. Authorities across London also shared data about children in receipt of free school meals, allowing them to be better supported while schools were closed.

There were also shifts in sharing data across partners. Central government gave local authorities access to the NHS shielding patients database. This allowed authorities to better target support, including food parcels and pharmacy deliveries, to those individuals who are particularly vulnerable to Covid-19.

“Faced with a once in a generation public health crisis, local authorities have sought novel ways of keeping their residents safe while continuing to deliver public services at a distance – and data-driven interventions have played a key role in these efforts,” says the DEI report.

Camden is one such council now seeing that managing data across its hundreds of different types of services, from property management to parking and schools, is not just a challenge but an incredible opportunity for learning across many departments. “Our investment in data and product disciplines should help us work towards synthesising data across services and unearthing new insights which we can test, learn and build from,” says Tariq Khan of Camden Council.

It is this ingenuity and shared purpose that will help public services help citizens through this age of increased demand, high cost of living and reduced physical access to services.

Whether it’s council tax details, information on the frequency of bin collections or outcomes of cancer treatments, the public sector is swimming in data. Struggling already to steer their ships in difficult waters, decision-makers in local and regional government are often too busy to capitalise on the potential of big data to innovate. 

How can you break out of the silo and smash down those obstacles, bringing about data-led innovation and decision-making? Here are five key barriers to overcome.

1. Not knowing where to look for key skills

One issue common to all public sector projects is finding the right people to carry out the work. A survey of public sector chief information officers (CIOs) conducted by software company Couchbase found that 17% of organisations pinpointed a lack of skilled staff to delayed or downsized digital transformation projects.

That means outsourcing is vital. But for time-poor public sector leaders, even identifying the problem and triaging the solution can be a hurdle too difficult to surmount. Half of cities surveyed by ESI ThoughtLab said identifying innovative IT suppliers was one of the biggest problems they faced. 

“Globally, thousands of ‘B2G’ (business to government) startups are developing high-quality, contextually sensitive products and services to help to address the pressing public needs that cities face, from sustainable micro-mobility to overcoming property tax evasion, all of which require a sophisticated use of data,” says Tanya Filer, research lead at Cambridge University’s Bennett Institute for Public Policy. “This suggests that often the problem is not supply, but knowledge and relationships.”

2. Government silos

The fear of mishandling data can lead government departments at national, regional and local levels to cordon off their data and not let anyone else have access to it – even at the expense of innovation. It’s a well-founded worry, given the risks and fines that can come with data breaches and leaks. But it shouldn’t put a brake on innovation.

“Data stored by public sector bodies can be highly valuable when it can be shared with other departments,” says Pritesh Patel, public sector lead at Databricks, who has worked with the Ordnance Survey on their data innovation. The importance of not withholding data and sharing it across departments has become obvious during the coronavirus pandemic. Patel advises moving away from proprietary legacy data storage tools to more modern ones – such as data lakehouses – can unlock barriers.

3. Tight budgets

A quarter of those in the public sector point to constrained budgets as the reason why they can’t deliver the digital projects they want, according to Couchbase. Many of the leading customer relationship management (CRM) tools are shunned by public sector organisations because they require costly long-term commitments to subscriptions.

There “may be a reluctance to pay the ongoing license fees, however, the commercial provision is often best in class and takes the pain away from upgrades and potential security issues,” says David Mitton, government senior relationship director at Dun & Bradstreet. What seems a short-term economic saving can actually result in long-term losses and inefficiencies when trying to wrestle with lesser-quality services and apps.

4. Lingering legacy tech

Those tight budgets and strict rules around how public money can be spent mean that public sector organisations are often left to fend with the fast-moving digital revolution at a much more leisurely pace. One in five CIOs surveyed by Couchbase said a reliance on legacy technology hindered the development of their digital projects.

Refusing to move with the times doesn’t just cause a slowdown in advancement. Legacy technology comes with its own risks. “Without a strong data governance foundation, inaccurate data sets with duplicated or outdated information can be stored and shared, hence multiplying the problems,” says Patel.

5. Non-uniform frameworks

Ask 100 different government bodies and you’ll likely discover 100 different ways of recording information. “Without a uniform, standardised, yet flexible approach, it can be really hard for government bodies to access data and to then trust it,” says Patel. “This, in turn, can drastically slow progress when it comes to rolling out new policy, adapting to volatile changes in the political or citizen environment, or even responding to individual or collective citizen requests.” 

Some headway has been made with the government’s 2020 National Data Strategy, but standardising data formats and frameworks needs continual work to improve.

Data is a tricky beast. It’s one of the most valuable assets to organisations, and yet one of the hardest to make proper (and safe) use of. While there’s rarely any shortage of the stuff, raw data is of little use without actionable insights.

Local authorities, which face barriers due to budgetary restraints and skills shortages, particularly struggle to make sense of their data. “Our need for data is stretching local authorities beyond what we've traditionally been able to fund them to do”, says Ron Mackintosh, director at Differentis, a business transformation consultancy and partner of TrueCue. “What councils need is ways to cut through this.” Crucial to this is investing carefully in the right technology and expertise to build an affordable, sustainable and scalable data and analytics system. This avoids wasting public money on overly ambitious digital transformation projects.

One of the biggest challenges for leaders is having access to simple, actionable insights when they need them. Most leaders just need the basic key performance indicators (KPIs), and the ability to dig into the detail where necessary. As Max Kenney, director of analytics at TrueCue, a data and analytics solutions provider, explains, “It's moving people away from their day to day activities where they're wrestling with data, so they can actually focus on the analytics and then make those decisions in light of the findings. If you don't have the information readily available to you, you can't make a quick decision.”

It's moving people away from their day to day activities where they're wrestling with data, so they can actually focus on the analytics and then make those decisions in light of the findings.

Kenney continues: “For example, if you're going to a board meeting and someone asks a question and you don't have the data there, you have to wait a whole month until the next board meeting to make that decision. It would probably be the same approach for a local council. If you don't have all the information readily available, it just means that you go back to the drawing board and do more analytics.”

The lack of a coherent, centralised strategy often prevents these timely insights. Departments are working in isolation (and often manually extracting data), leading to data quality issues, lots of repeated effort, a lack of visibility and over-reliance on IT teams. This is where Differentis and TrueCue come in, combining TrueCue’s data warehouse automation platform with Differentis’ expertise to assess the data maturity of local authorities and help them build a solution. “It’s about providing honest, pragmatic advice, coupled with the tools at the right point in your maturity, to get the answers you need,” explains Mackintosh.

Being able to trust data is also a real challenge for councils. Mackintosh explains: “When you work in isolation, on one tool, you can kind of manage the data and understand it. But actually, you might not notice the data quality issues until you try and share it. Now, if you go into a big data warehouse project, what's likely to happen is that you set it up and the data quality just explodes. It's all over the place. But if you take a piece by piece approach, what you're doing is you're rapidly testing it. You figure out what's wrong with the data, and you start saying ‘how do we fix it so that when I add the next service on I’m only doing an incremental data change?’” 

Mackintosh says that this approach is met by a huge sigh of relief by councils. They know the data is not great, but there's no way they could fund a big data quality programme to fix everything. “If we can experiment quickly, stop the mistakes, tweak it, understand it and start to take it through, you can build an increasingly competent and high-quality data platform. And the important thing that you have to be able to do is to trust the output,” he explains. Once you have that high-quality platform, any analytics and algorithms run on top of that are using complete, trusted data and can be used to glean actionable insights. 

A user-friendly, centralised platform also eliminates the need for excessive spend on advanced and expensive data roles. TrueCue’s platform, for example, has a low-code/no-code approach intended for business users – so you don’t need to be a tech whiz. Once everything is up and running, it can largely be maintained and managed by the organisation. “At TrueCue, we provide training and enablement services with our platform, but also other technologies that we use, such as Tableau and Power BI, so that teams can become self sufficient,” Kenney adds.  

It’s also vital that citizens’ personal information is handled correctly. Without a smart, centralised system, data is at risk of leaking. Kenney explains: “It’s safer the way we’re going about it now. If you’ve got one single source of the truth, you can then manage who has access to that.” Before, you may have had data being thrown around over email or on shared file systems, which is neither secure nor easily traceable. 

An example of this all in action is TrueCue’s work integrating NHS care across North West London. The North West London health and care partnership is a collaboration of eight NHS Clinical Commissioning Groups, providing healthcare for more than two million people across eight boroughs. NHS NWL wanted to increase engagement with partners across all care settings, but a major barrier was that partners all had individual systems collecting data in silos. 

Over the last 10 years, TrueCue has been working with NHS NWL’s BI team, to build their  ‘Whole Systems Integrated Care’ solution. This has involved taking data from multiple systems and bringing it together into one central repository. Safely and securely consolidating this data has enabled a single patient view, which in turn, has helped improve care for millions of patients across the region.

This model could be similarly applied to different council levels, integrating data sets across town council, borough council and county council to achieve a single, complete view of citizens. With that information, the potential to transform communities is huge – whether that’s analysing footfall in town centres to help attract the right private investment in retail and hospitality, or understanding more about city traffic to pave the way for electric vehicles. The right insights exchanged between different organisations with common goals can be a powerful tool for change. Kenney and Mackintosh will be speaking about enhancing performance management in a local borough council at The Second Annual National Local Government Event 2022 on 30 March 2022. Find out more.

In the world of data, the tools get to blame bad workmen. As the public sector attempts to marry a smaller budget with a growing workload, the toolbox of shared data can lead the way to better services but only if used correctly by staff. Being data-driven goes deeper than bringing charts to meetings or ‘making decisions with numbers,’ it means daily decisions based on evidence even when that presents uncomfortable truths, says Scott Castle, senior Vice President of Sisence, a AI-driven platform for infusing analytics everywhere within businesses. “Don’t let your teams just search out favourable statistics – encourage them to look at the complete data picture and come to conclusions based on the preponderance of the evidence,” says Castle. “Success means implementing a hypothesis-driven culture which identifies theories, tests them, and rigorously seeks to disprove them while rapidly implementing those that show promise.”

Canadian airline Air Canada is one example where in-the-field workers were given mobile devices to access critical safety information without returning to their desks. This empowered gate agents, maintenance professionals, and other front-line employees to make critical business decisions with greater efficiency. 

Private companies, driven by profit and need for engagement, are often proactive out of necessity, whereas public firms are reactive says Roger Grimes, an expert in data-driven defence at KnowBe4. But the benefits of this proactive approach are clear. Grimes points to the example of how, during the pandemic, thermometer companies used the data of sudden increases in fevers to detect where Covid outbreaks were currently happening. “The best that governments, by themselves, could do was tell us where the outbreak did happen. (By being proactive) the private thermometer companies realised they were detecting Covid outbreaks in real time with high correlation.”

Allow data to make sure everything clicks

Most public service providers already own impressive data but don’t know the value of it, find it overwhelming, or the useful parts are logjammed in the C-suite. 

Clearing a pathway for data can be particularly hard when you are the CEO looking from the top down, says Monica Lagercrantz, the CEO and founder of Boardclic, which uses data analytics and best practice benchmarking to provide proof of any alignment issues between board and CEOs. “Once you are appointed to a board or become a chief executive the feedback often stops. Instead, allow data to provide that rare chance to get qualified feedback to improve performance at the top level. Boards must be proactively trained on how to use data, rather than believing that this should only be done in operations, or even worse, in the tech teams. Data is no longer a resource only for a few teams, everybody must embrace and work with it.”

Data is no longer a resource only for a few teams, everybody must embrace and work with it

Traditionally, a few key gatekeepers for data were asked to dig through dashboards, submit data requests, or wait for reports to make informed decisions. But this method has proven to be error-prone, distracting, and quite simply, slow, says Scott. “We must adopt new ways of delivering insights if we want to make quick decisions based on data at every turn. To do this, we have to bring the data to all staff rather than creating yet another destination for decision-making.”

Open the gates, control the flow

As if to demonstrate this, the Department for Work and Pensions (DWP) has kicked off a nine-month pilot project, the Labour Market Data Trust, which aims to create a cross-department data store for sharing labour data across various government departments. By making data available and easily discoverable, the goal of the pilot is to better address many departments' labour market needs as instantaneously as possible, preventing workflow failings within teams.

When introducing your staff to data – as well as ensuring they are meeting privacy standards – embolden them to see data management not as a storage task, but more like panning for gold. Find the kernels of information that define your business and don’t hold on to the rest. Otherwise, you can quickly fall victim to ‘death by data’ says Grimes. “A Fortune 500 retailer was getting 15,000 messages about a previously unknown programme being inserted all over their environment. This should have made the IT security team shut down the attack. But instead, the sheer amount of warnings were discounted by a single IT security employee who mistakenly identified it as a legitimate software update.”

Board level staff should be the ones initiating the change towards a data-centric culture by increasing data knowledge, offering training and creating rewards where those finding data points see the value of their efforts to the company. 

The benefits are that proper data management and assessment are something that brings tangible results, whether financial or in terms of focus, says Tomas Montvilas, chief commercial officer at Oxylabs, who help large companies like Trivago to use their own data to improve their business. “Any leadership, as long as it’s not forceful, from the highest levels of management is always beneficial. If the strategy for the change is headed from them, it’s much more likely to be seen as something that involves the entire organisation.”

When data involves the entire organisation, it can work better to help improve it.