The Data Science Bubble: What You Need to Know

The 2.5 quintillion bytes of data people generate every day are a valuable source of valuable insights. Organizations are investing accordingly, and IDC predicts the data science market will be worth $274 billion by the end of 2022. However, the buzz surrounding data science, artificial intelligence and predictive analytics has prompted some employment analysts and technologists to suggest the rapid growth in this relatively new field as too healthy, i.e., unsustainable over the long term in its current form.

That assertion is not unreasonable. According to Gartner, one of the most common failures in Big Data analytics in the last decade has involved organizations “setting overly optimistic expectations when a skilled team is not in place to deliver.” The idea that the data science market is a bubble waiting to burst stems partly from this overexcitement. Companies create data science teams and invest in AI infrastructure without understanding the applications of machine learning or deep learning, then do not receive the ROI they expected.

Professionals already pursuing data science careers may wonder what this means for the future of data scientists. And the idea that there is a looming data science bubble may give aspiring data scientists looking at programs such as the Charles V. Schaefer Jr. School of Engineering & Science‘s online Master of Science in Data Science (MSDS) pause. However, while data science is undoubtedly changing in several ways explored in more depth in this article, there is no objective evidence to suggest the Big Data market is in danger of collapsing.

Plenty of evidence suggests there is still room for the field to grow — and room for new data scientists, statistical analysis experts and data engineering professionals. According to research conducted by Seagate and IDC, organizations only use 32% of the data they collect. Nearly 99% of firms surveyed by NewVantage Partners want to leverage the power of data but cannot find the talent they need. MicroStrategy’s 2020 Global State of Enterprise Analytics Report found that only 57% of enterprise organizations currently use data to drive strategy or change. And according to some research, the global data science market will grow at a compound annual rate of nearly 27% from now until 2027.

That is good news for established data specialists and aspiring data scientists, but professionals must still consider how the field is changing. Automation, evolving employer expectations and skill set segmentation in data science are changing what it means to pursue a career in this field.

IS DATA SCIENCE OVERSATURATED?

Data science is not oversaturated. The myth that the field is saturated — or close to it — likely stems not from an abundance of advanced analytics talent but from the exponential growth in interest in the data science field.

Many businesses do not understand the technical aspects of data science or its potential to deliver value. The data science field evolved so quickly alongside advancements in computer science that organizations have struggled to keep up. Most know they should grow a data team to stay competitive but fewer understand what skills to look for in new hires or how to put those skills to work most effectively. The pressure is often on individual data scientists to grow a functional data strategy from the ground up and to find ways to integrate artificial intelligence and machine learning into existing business processes. To benefit from an abundance of data, organizations will need to hire not just data scientists but also data engineers, machine learning engineers, data visualization experts and other specialists in data science.

The shrinking disconnect between supply and demand in data science also contributes to the myth of oversaturation in data science. More highly trained data scientists exist today than four years ago when LinkedIn reported that 150,000 jobs for data scientists would go unfilled. Some employment analysts see this as evidence of a coming slump. However, there is still a significant disconnect between the technical skills data professionals possess and the skills necessary to excel in a changing technological landscape. Many data professionals do not have the skills needed to keep pace with automation, implement or work alongside AI technologies or effectively manage increasingly large and complex datasets.

And finally, titles are often misleading and may not align with work experience. Some professionals in data scientist roles have the skills to do what is colloquially known as “data janitor work” but not the credentials or knowledge to advance in this increasingly complex and specialized field. About half of all data science job postings across the U.S. require that applicants have an MSDS or have completed a similar graduate program in data analysis or business analytics. In-demand data science skills in 2022 include programming language skills (especially in Python and R), data visualization skills, Natural Language Processing skills, cloud computing skills, gradient boosting skills and deep learning skills.

While it’s true that there are more highly trained data scientists today than four years ago, there is still a global shortage of data talent when it comes to professionals with the right blend of advanced training and credentials. Therein lies the value of earning a master’s-level data science degree. Pursuing an MSDS is a straightforward way to gain leading-edge skills that separate applicants in data science hiring pools.

THE HYPE AROUND DATA SCIENCE IGNORES THE COMPLEXITY OF THE FIELD

The buzz generated by data science tends to overshadow the realities of data science as a profession. Technologists blithely assert that data science is an immature discipline concerned with chasing fads. Tom Davenport and DJ Patil provocatively called it sexy. Headlines proclaim that applied data science is alternately the future of everything or in danger of becoming obsolete. The reality falls safely in the middle.

Big Data now drives decision-making across industries, and many employment analysts imply that anyone with skills related to artificial intelligence, machine learning, neural nets or decision trees will be rejecting job offers left and right. But while there are talent gaps in data science, they are not equally distributed geographically. High-paying jobs in data science are typically clustered in established and emerging technology hubs, and the regions in which data scientists earn the most also attract significant data science talent. Consider, too, that 90% of data scientists have master’s degrees or other advanced degrees. In areas with no shortage of qualified data scientists, it may be difficult to find work with just a bachelor’s degree, online courses or a bootcamp certificate. Advanced education is essential in areas with the most data science jobs.

Big Data also is suffering from what might be called a hoarding effect that overlooks the importance of data science talent. “Compile enough data, the belief goes, and you no longer need to worry about statistics: once you have the totality of human existence encoded in your digital archives, truth emerges from the most basic of keyword searches with nothing further needed,” writes Kalev Leetaru for Forbes. Data science is about more than gathering and warehousing as much data as possible. Organizations equipped to profit from data understand that data is only useful in the hands of professionals who can leverage domain knowledge alongside technical skills to distinguish passing trends from actual insights.

Keep in mind that the field is growing more segmented, and the way people think about data science may be far too limiting. The Quant Crunch report compiled by IBM and the Business Higher Education Forum asserts that employers and higher education institutions think beyond the limitations of what we currently call a data scientist or data analyst “to develop talent for a variety of roles, such as data engineer, data governance and lifecycle and data privacy and security specialist and data product developer.” That speaks to a future with abundant opportunities for data scientists with the right skills.

AUTOMATION

As more data science processes are automated, some employment analysts have speculated that data scientists will become obsolete. However, more intelligent tools do not negate the need for human beings at the helm. Yes, AI and machine learning algorithms can dig through complex datasets much more quickly than humans and with fewer costly mistakes. But even the most sophisticated automated systems cannot replicate the critical thinking skills, emotional intelligence and reasoning skills of a single human data scientist. AI can spot trends, but only people can determine whether those trends are meaningful.

That’s because algorithms can’t see the bigger picture. Sales and customer service teams utilize data mining — a process that is increasingly powered by AI — to observe and predict consumer behavior and spending. Data mining can optimize marketing campaigns, increase ROI and provide analyses that drive reports for the company’s future. But relying too heavily on data mining without human oversight is risky. Consider the case of Zillow Offers, an arm of the familiar online real estate marketplace. It used Big Data to estimate how much homes were worth and how much Zillow could make flipping them but failed to adjust its estimates when the COVID-19 pandemic drove the cost of materials through the roof. A human data scientist would know that any real estate data gathered in or around 2020 would be rife with abnormalities.

As data scientist Michael Li, founder of The Data Incubator, wrote on TechCrunch, “real-world data are notoriously dirty and many assumptions have to be made to bridge the gap between the data we have and the business or policy questions we are seeking to address. These assumptions [are] highly dependent on real-world knowledge and business context.” In BDO Digital’s recent survey of middle-market executives, 42% of executives anticipated job roles would be re-imagined due to automation — not eliminated.

THE DATA SCIENCE BUBBLE MAY NEVER BURST

Headlines declaring that data science is dead or data science will die soon — or that data scientists will become obsolete — ignore the fact that this discipline is evolving. It is only natural that interest in data science is skyrocketing. Innovations in neural networks, Natural Language Processing, intelligent machines, supervised learning and Federated Learning are relatively recent advancements in a discipline defined by disruptive technology. Data science was once a murky discipline that even those working in it could not easily define. Modern data science is growing more segmented and specialization makes it easier to see how data scientists in different roles deliver value. Some data scientists handle model development. Others do analysis. Still others adapt AI for technical implementation. And many data scientists specialize in software engineering, deep learning, data mining, data visualization or data architecture.

According to the’ Quant Crunch report, “Technologies change quickly. This reality requires a new type of workforce and attitude from both employers and employees around continuous learning and mastering skills that will enable employees to prepare for not-yet-arrived jobs of the future.” Neither rapid change nor rapid growth correlates with technology bubbles today because this is no longer in a world in which entire fields blink out of existence. Technology changes, technology-focused jobs evolve and workers adapt by enrolling in data science master’s programs to stay competitive.

Cindi Howson, Chief Data Strategy Officer of ThoughtSpot, writes that “the mismatch of expectation and reality will cause data science to lose its luster.” This suggests the enthusiasm for data science and the demand for data scientists can continue to grow unabated, provided professionals in the field deliver insights that drive quantifiable enhancements. Pragmatism is clearly more useful than hype in data science, as are leading-edge data science skills, advanced mathematical skills and highly developed computational skills — everything covered in the coursework in degree programs such as Stevens’ 100% online M.S. in Data Science.

The graduate-level data science program is led by experts in subfields of data science and data engineering such as stochastic optimization and cryptography. It attracts data analysts and other professionals who feel that pursuing a part-time M.S. in Data Science online is the best way to advance in their careers or learn to leverage the data generated in their professional spheres more adroitly. They believe in the power of information to change the world, but they are ready to enter the field with their eyes wide open.