Denise Gosnell, chief data officer at DataStax, discussed how preparation, process and open source can help to ensure success from data science projects
It’s important to set out how your projects will support overall business goals.
For businesses, investment in machine learning, artificial intelligence (AI) and data science is growing. There is huge potential around data science to create new insights and services for internal and external customers. However, this investment can be wasted if data science projects don’t fulfil their promises. How can we make sure that these projects succeed?
Where we are today
According to McKinsey, around half of all the companies they served have adopted AI in at least one function, and there is already a small cohort of companies that can ascribe at least 20% of their earnings before interest and taxes to AI. Around $341.8 billion will be spent on AI solutions during 2021, a rise of 15.2 percent year over year, according to IDC.
IDC also found around 28% of AI and ML initiatives have failed so far. Based on the figure above, that would equate to $88.1 billion of spend on tooling associated with failed projects. The analyst firm identified reasons for this including the lack of staff with necessary expertise, and a lack of production-ready data as reasons for this. Alongside this, feeling unconnected and lacking an integrated development environment was another reason for projects not being successful.
To improve your chances of success around your projects, it is worth spending time to look at how data science works in practice, and how your organisation operates. While it includes the word ‘science’ in its title, in fact data science requires a blend of both art and science in order to produce the best results. Using this, it’s then possible to examine scaling up the results. This will help you successfully turn data science results into production operations for the business.
At the most simple level, data science involves coming up with ideas and then using data to test those theories. Using a mix of different algorithms, designs and approaches, data scientists can seek out new insights from the data that companies create. Based on trial, error and improvement, the teams involved can create a range of new insights and discoveries, which can then be used to inform decisions or create new products. This can then be used to develop machine learning (ML) algorithms and AI deployments.
Hot topics and emerging trends in data science
We gauged the perspectives of experts in data science, asking them about the biggest emerging trends in data science. Read here
Improvement #1 – know the expectations around business goals
The biggest risk around these projects is the gap between business expectations and reality. AI has received a huge amount of hype and attention over the past few years. This means that many projects have unrealistic expectations.
Unrealistic expectations can be in scope, speed, and/or technologies. Great project managers understand how to navigate challenges in scope and speed; it is the misinterpretation of the promises of AI technologies which have been causing the biggest problems for new projects. Rather than being focused on improving a process or delivering one insight, AI gets envisioned as changing how a company runs from top to bottom, or that a single project will deliver a change in profitability within months.
To prevent this problem, it’s important to set out how your projects will support overall business goals. …….