Big Data Big Problems

Corporate viability of Big Data is crucially dependent on certain problems being tackled head on

Given its primacy established during the COVID-19 pandemic, corporate leaders of big data will surely be rethinking their business objectives, looking to improve data quality and cause a major turnaround of several big data projects in 2021. This is, however, going to be a lot easier said than done. There are several aspects of big data that firms need to take care of going into the new year in order to make sure the full potential of big data is extrapolated into business usage.

Data Management and Monitoring

At IBM Switzerland, researchers developed an algorithm using AI and Machine Learning that would plough through reams of scientific research papers and journals in search of relevant information pertaining to molecular drug design. In the process, they realised that more data would be irrelevant to the problem at hand than relevant – and therefore, eliminating the need to parse such data would save the algorithm much time and reduce data storage wastage.

Although this sounds rather intuitive, this is a rather persistent problem worldwide. In fact, amidst the torrents of data available on the internet, most corporations are plagued with data that turns out to be more ‘bad’ (incomplete, incorrect or irrelevant, for example) than good. In fact, estimates have shown that poor data has been costing the US economy almost $3.1 trillion annually on its accrual and usage. This is, of course, a major burden that needs to be considerably eased off to ensure long-term sustainable profitability.

Stress needs to be placed on adequately screening the data at the point of entry, as well as in the proper cleaning and preparation of the data resources before it is added to corporate repositories. This includes checking for ‘incomplete, duplicate, and inaccurate data, and also normalizing data so it can be blended with other source data for analytics.’

Just setting up an effective process, however, is not enough. Continuous monitoring and optimisation of the process needs to be high on the agenda as well. In fact, many firms have already started using an iterative DevOps-style development approach to big data analytics. According to technology-blogging site TechRepublic: “In the age of constant security vulnerabilities and subsequent security patches, and the rapid and frequent addition of features to software, DevOps–a workflow that emphasizes communication between software developers and IT pros managing production environments–is at the forefront when considering how to shape an IT department to fit an organization’s internal needs and best serve its customers.”

Here, formalisation will be key. Data scientists need to make sure all big data analytics models which are deemed ‘ready’ for deployment into production are mature enough. This implies, that if the benchmark for corporate readiness is set at 95%, the model must accurately and consistently deliver the specified level of performance over a considerable period of time. This is, however, contrary to current standards, where more often than not, big data models fall off from their benchmark levels as time wears on. This means that IT and data science departments of corporations need to remeasure apps for accuracy annually, as well as consistently making sure levels of accuracy are maintained.

Formalisation of Hybrid Architectures for Big Data Analytics

Big data analytics applications are now being set up by most corporations in a way that seamlessly synthesises public and private cloud platforms. As this need for the coalescence of disparate data sources comes to fore, the establishment of an over-arching hybrid cloud architecture becomes crucial to long-term viability. For the aforementioned, a hybrid cloud and on-premises platform that brings together enterprise security and governance needs to be formalised throughout corporations.

TechRepublic writes: “As more vendors simplify AI solutions, there has been growth in citizen AI, where business units develop their own AI and big data applications. Later, when users want to train these apps and integrate them with other company data and platforms, they need IT and data science departments to help them.

If IT and data science professionals actively collaborate with business users early in their application processes, many of these follow-on integration difficulties can be avoided. Developing productive relationships with business units throughout the company should be a major big data and analytics goal for IT.”

An essential aspect of perfecting this architecture is going to require a major improvement in security, especially for IoT. Most corporate Internet of Things (IoT) devices usually do not meet company security and governance standards. With cybersecurity intrusions rampant, continuously reviewing the security of aforementioned devices whilst assuring security and governance standards are met is going to be paramount going ahead.