Culture, not technology, the biggest challenge in creating data-driven organisations
The 2020annual survey by NewVantage Partners of senior executives about data and how organisations extract value from it reveals a worrisome trend of organisations unable to extract full value from their investments in AI, as they are yet to create a data-driven culture. At the same time, investments in new technologies around data science are increasing. Organisations are just not becoming data-driven and do not build data-focused cultures. This uncomfortable truth has been revealed in a survey of over 70 Fortune1000 or industry leading firms, with 98.8% of participants identifying themselves as C-executives or equivalent.
The survey results have been quite consistent; they portray a field that is struggling to succeed despite massive investments in technology and applications. New technologies excite executives and organisations, new positions are created to capitalize on data, and new applications appear to provide value. The principal challenges to becoming data-driven continue to be cultural – people and business process related – and not about technology (only 9.1%). The greatest barriers to success continue to be organisational.
This year’s results and findings paint a picture of businesses and industries in transition. C-executive decision-makers form an overwhelming majority of the group that was surveyed. This group was dominated by Chief Data Officer (CDO) and Chief Data & Analytics Officer (CDAO) respondents, the CDO and CDAO being a role and function which has emerged over the course of the past 8 years as the principal advocates for data within the corporate enterprise. Many organisations have merged the Chief Analytics Officer (CAO) role with the CDO role to create the combined CDAO function.A shrinking percentage of firms are represented in the survey by the CAO function.
Companies continue to focus on the supply side for data and technology, instead of increasing demand for them by business executives and employees. It’s a technology push rather than a pull from humans who want to make more data-based decisions, develop more intelligent business processes, or embed data and analytics into more products and services.
The survey has crisply summed up the scenario:
“More than two-thirds of these large organizations are investing $50MM or more in these technologies, and many spend half a billion or more. Yet minorities of those surveyed report that their business is data-driven, and even smaller minorities that their cultures emphasize data. About three quarters of respondents say that business adoption of big data and AI is a challenge. Over 90% report that the challenges to becoming data-driven are in people, process, and culture—not technology. Perhaps not surprisingly, only 15% have deployed AI broadly into production—because that is where people and process issues come into play.”
Continuing with the same supply-driven approach is not going to change this situation. What will change it is concerted efforts to educate senior and middle managers about the power of big data and AI, to provide experiences that make them more comfortable with technology, to redesign jobs to accommodate smart machines, and to upskill employees at every level to work with analytics and AI.
There is a growing concern that IT and data executives feel business leaders in their organisations are too old to learn new tricks. Indeed, in a recent global survey of executives sponsored by the analytics vendor Splunk, 73% felt that data skills were harder to learn than other business skills, and 53% believed they are too old to learn such skills. This speaks either to the need for a concerted effort to change their minds, or to a need for generational change.
The NewVantage survey comes up with some compelling percentages:
“Business adoption of Big Data continues to be a struggle, with 73.4% of firms citing this as an ongoing challenge. Only 37.8% report that they have thus far created a data-driven organisation. Even fewer – only 26.8% — report success at building a data culture within their firms. Firms are still struggling to compete on analytics – only 45.1%, a slight decline from 47.6% in 2019. Most firms cite people and process challenges – 90.9% – as the biggest barriers to becoming data driven organizations.”
Most companies will have to modernize their data architecture, ingest data from novel sources, design algorithms to model data and derive insights, and hire or train the talent to do it all. The price tag for these efforts can run from hundreds of millions of dollars for a midsized organisation to billions of dollars for the largest companies. Before the COVID-19 crisis, many organisations were projecting the need for more data investment, and the crisis has only increased this need. Businesses understand that being data-driven to drive decisions is no longer an option but has assumed mission criticality. The work must begin with a culture change.
Acknowledgement:Big Data and AI Executive Survey 2020 – sponsored by NewVantage Partners LLC