Never have computer and data science occupations experienced a pace of evolution faster than the past decade. Hand-in-hand, technological advances in computing speed, storage, and connectivity, combined with constantly updating, open-source platforms, have revolutionized analytic methodologies beyond what statisticians could have imagined a generation ago.
Plus, the widening adoption of machine learning, AI, and IoT foretells an accelerated pace of transformation ahead. The Global Big Data analytics market is projected to exceed $655 billion by 2029, up from $240 billion in 2021.
Not surprisingly, corporations are investing heavily in Big Data resources and talent. Job openings for data scientists have grown almost five-fold since 2016, and those trends are expected to continue for the foreseeable future. From an outside perspective, one might conclude that Big Data and analytics are fast becoming predominant components of business growth.
But do results match the pace of investment?
While excitement about the potential of Big Data receives wide publicity, some reports suggest the reality isn’t quite as rosy. Despite the billions invested, many companies are falling short. According to a 2021 survey of executives from big-name fortune 1000 firms, their results from Big Data efforts do not match the popular hype. For example,
- Only 48.5% are driving innovation with data
- Only 39.3% are managing data as a business asset
- Only 30.0% have a well-articulated data strategy for their company
- Only 29.2% are experiencing transformation business outcomes
- Only 24.0% have created a data-driven organization.
Other findings indicate that the failure rate of Big Data Projects is alarmingly high. Gartner, a $4B company that provides insights to corporate executives, projected that only 20% of big data analytic projects will deliver value to business through 2022.
Further, business journals highlight a pervasive level of disappointment with analytics: Why so many data science projects fail to deliver (MIT Sloan), Companies are failing in their efforts to become data-driven (Harvard Business Review), and 10 reasons why your organization still isn’t data-driven (Forbes).
Why are Big Data efforts falling short?
Reviews of what goes wrong emphasize that the problem is not technology. Workers have the tools, equipment, and resources to do analytic work. Neither is it a lack of analytic talent. Instead, reviewers conclude the primary culprits are organizational and cultural, a failure at the interface between data professionals and people who apply solutions to the business.
It should not surprise us that these two groups of professionals have a communication gap. They have different languages, styles, terminologies, training, and biases. They operate in separate departments. Their objectives are often misaligned, if not in direct conflict. Plus, neither profession places a high priority on communication training. As a result, as many as two-thirds of each group feel negatively about their interactions with the other.
If we have strained interactions between business and analytic teams now, we can only expect them to get worse. The pace of evolution in methodologies, and their complexity, will only pull analytic skills further into an area of rare expertise. Business leaders who did not understand p-values or regression coefficients will find themselves faced with black-box, machine learning models that most people cannot explain, let alone run.
Remember, only 6.7% of the workforce work in STEM fields. Meaning, more than nine out of ten working adults intentionally chose not to study science and math.
How do companies move forward?
To capitalize on Big Data opportunities, corporations will necessarily accumulate talent in …….