Subscribe to our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage.
There’s been a lot of talk recently about the AI revolution reducing the need for data engineers. I disagree — data expertise will be more important than ever. However, data professionals will need to learn new skills to maximize AI’s potential and enhance their career opportunities.
AI allows organizations to extract more value from their data efficiently, but this won’t happen on its own. Data engineers must learn how to apply the technology, which models to use, and in what situations.
Here are four areas where AI will transform data analytics in the coming year and the skills data engineers must develop to meet these needs:
Building Smarter Data Pipelines
Data pipelines integrate various sources of raw, unstructured, and disorganized data. Engineers then extract intelligence from these sources to provide valuable insights. AI is set to revolutionize this process.
Integrating AI into data pipelines can significantly speed up a data engineer’s ability to extract value and insights. For instance, a company with a database of customer service transcripts or other text documents can use AI to extract insights quickly with just a few lines of SQL, a task that would take many hours manually and may uncover insights only AI can find.
Data engineers who know how and where to apply AI models to extract maximum value from data pipelines will be highly valuable, but this requires new skills in choosing and applying the right models.
Less Data Mapping, More Data Strategy
Different data sources often store information differently. For example, one system might use “Massachusetts,” while another uses “MA.” Mapping data to ensure consistency and remove duplicates is an ideal task for AI. Engineers can prompt AI to merge various customer data sources into a single canonical database quickly.
This will require knowledge of how to write effective prompts, but it will also free up engineers’ time to focus on data strategy and architecture. The ultimate goal is to understand all available data sources and leverage them to meet business goals. Delegating data mapping to AI will free up time for higher-level tasks.
BI Analysts Must Up-Level Their Game
Business intelligence (BI) analysts currently spend much of their time creating static reports for business leaders. When leaders have follow-up questions, analysts must run new queries and create additional reports. Generative AI will change executives’ expectations.
As executives become more accustomed to AI-driven chatbots, they will expect to interact with business reports in a conversational manner. BI analysts will need to adapt by learning to provide these interactive capabilities, moving away from static charts to dynamic, interactive reports.
Cloud data platforms offer some of these capabilities in a low-code manner, giving BI analysts the chance to extend their skills. However, there is a learning curve, and acquiring these skills will be a key challenge in 2024.
Managing Third-Party AI Services
A decade ago, the rise of cloud computing shifted IT teams’ focus from building infrastructure to managing third-party cloud services. Data scientists are about to undergo a similar transition with the growth of generative AI.
Data scientists will need to work more with external vendors providing AI models, datasets, and other services. Familiarity with the options, selecting the right model for each task, and managing these third-party relationships will become crucial skills.
Looking Forward to a More Enjoyable Work Environment
Many data teams today feel stuck in a reactive mode, constantly addressing job requests or fixing broken applications. This is not enjoyable, but the influx of AI into data engineering will change that.
AI will enable engineers to automate the most tedious parts of their work, giving them time to focus on the bigger picture. This will require new skills but will allow them to engage in more strategic, proactive work, making data engineers even more valuable to their teams and making their work more enjoyable.
Jeff Hollan is the Director of Product Management at Snowflake.
DataDecisionMakers is a community where experts, including those doing technical data work, can share insights and innovations in the field. For cutting-edge ideas, best practices, and the latest information on data and technology, join us at DataDecisionMakers. You might even consider contributing your own article!