So you’re curious about the reasons why data science projects fail. In this article, I’ll explain 9 reasons why data science fail.
According to VentureBeat, fewer than 15% of Data Science projects actually make it into production. And the number of projects that actually add value is probably even lower. Before, we can resolve this problem, we need to understand the root causes of this problem. So let’s get to the content.
- Lack of Support from Top Leadership – If leadership at the very top of a company doesn’t prioritize data science initiatives, then middle management won’t have the time and incentive to prioritize those activities either. Before data science initiatives even start, the top leadership in a company needs to understand the specific and concrete advantages that data science can bring to their company and the approximate timelines for reaching related milestones.
- Inadequate Data Infrastructure – It may be the case that an organization still has an over-reliance on Excel sheets and doesn’t have a robust, enterprise-level database or cloud system. And the current infrastructure of a company may not yet allow for the versatile use of pipelines and reciprocal data systems. If a company hires a data scientist before hiring a team of data engineers to build and maintain the enterprise-level infrastructure, that’s usually not ideal.
- Low Analytics Maturity – Analytics maturity is determined by the average level of complexity related to how a business practices data exploration and data-based decision-making. There are 4 primary levels of analytics: descriptive, diagnostic, predictive, and prescriptive. Companies typically move up those levels gradually over time. In a nut shell, if a business isn’t even utilizing mathematically based forecasts at the predictive level on a regular basis, that may be a sign that it needs more time before it tries to gain acceptance of complex deliverables involving machine learning and deep learning among business stakeholders. To speed this along, It may help to gradually introduce data science and machine learning deliverables in conjunction with existing analytics deliverables in a company, so that business stakeholders can adjust their decision-making processes, departmental objectives, and expectations accordingly.
- Unclear Job Requirements – Unclear job requirements for data scientists seem to fall into two primary categories: The first category involves the job requirements that contain a laundry list of tools and technologies, many of which may not be used in the position on a normal basis. And in some cases, the years of experience required for the technologies exceed the number of years that those technologies existed. This type of job requirement may discourage an ideal candidate for a position from applying for that position. The second category of unclear job requirements involves requirements for several positions at once including data engineer, business analyst, Business Intelligence Developer, business intelligence analyst, and project coordinator. This type of job requirement may have qualified job candidates wondering what percentage of the job involves actual data science. That brings me to this next reason.
- Giving Data Scientists Several Jobs at One Time –If a data scientist is hired and that person is only able to spend less than 5% of their working hours on actual data scientist responsibilities, that may be a sign of inadequate personnel and program management planning within a department. The assumption that a company is saving money by hiring someone to be a generalist who does everything can backfire in the long and lead to an overall decrease in the quality of crucial deliverables.
- Lack of Clear Business Cases Communicated by Business Stakeholders – If there is no clear direction on what use cases to work on and what activities should be prioritized, the burden is placed on the data science team to think of use cases on its own. This can lead to a misalignment of goals and priorities across teams.
- No Clear Lines of Communication Between the Data Science Team and Business Stakeholders – Because business requirements can constantly evolve, a data science team can’t adequately serve a business if there isn’t a reciprocal relationship with business stakeholders.
- Lack of Emphasis on Business Impact – Sometimes a data science team can focus too much on the technology while neglecting to measure and maximize impact to the business. Having adequate business and product knowledge in a data science team can help alleviate that.
- Scattered Data in Silos Across Various Departments – This can lead to data scientists having access to data that is less comprehensive and lower quality than the data that is available to functional analysts throughout an organization. This can also lead to inconsistent measures such as revenues, profits, and order numbers throughout an organization, which can lead to ambiguity and doubt when data scientists present their findings.
___________________
REFERENCES:
“Why do 87% of data science projects never make it into production?” – VB Staff:
💻 https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/
“INFORMS Analytics Body of Knowledge” – James J. Cochran:
📚 https://amzn.to/3iN3EcP
“Data Science for Business” – Foster Provost:
📚 https://amzn.to/34JAVAM
___________________
Disclaimer: A few of the links in this post are affiliate links.
