Industry Insights

Data engineering has never been more important or relevant than it is today. Many might say that data engineering as a profession has been around for well over a decade, or even several, since relational databases came to market in the 1970s and required extract, transform, and load (ETL) expertise, which was an early form of data engineering. Now data volumes, variety, and velocity are much greater than they used to be. This has led data engineering professionals away from using traditional ETL tools to developing and adopting new tools and processes to handle the data required to today’s data science and analytics processes.Burtch Works has experienced firsthand the increase in demand for data engineering driven by the data revolution. Since the inaugural data engineering report last year, many of the insights the report predicted have come to fruition. This includes high demand for data engineers, growing investment in data engineering teams, and an influx of early career engineers.This post, which is based on the newly-released 2022 Salary Report for Data Engineering Professionals (which can be downloaded for free here), shares more about the key insights and findings from this year’s data.

Astronomical Demand for Data Engineering Professionals

Many data teams had already resumed hiring last year (escalating in Q2), and so far in 2022, hiring has been incredibly competitive. In Q2 2022, the research found that 81% of data teams are planning to hire during Q3 or Q4 of 2022. The respondents of the survey spanned over 120 companies across the U.S. Of the total remaining respondents, only 14% of data teams indicated plans to hold steady, a mere 3% of teams revealed they are on a hiring freeze, while barely 2% of teams reported they are either cutting back on hiring or planning to do so. Several clients have shared their desire to fill all their open roles before the end of the fiscal year so they don’t jeopardize losing their budgets should the economy continue to slow down in 2023. Ultimately, permanent hiring seems to be the favored method of adding data engineering headcount, but as economic conditions shift contract and contract-to-hire demand is starting to increase.

The Great Resignation & Its Lasting Impact

The effects of the COVID-19 pandemic on the hiring market have shifted significantly over the past few years. During the immense disruption of 2020, many professionals chose to put planned job searches on hold, while some data teams were more impacted than others in terms of their hiring plans and response to the initial crisis. There simply wasn’t a lot of job movement.As the economic recovery picked up in 2021, we saw more data teams resuming hiring. There was also massive turnover in the labor market, dubbed the Great Resignation, as many professionals resumed their job searches or sought to change their working situation due to other pandemic-related factors. Unfortunately for those looking to hire, this has led to fewer professionals on the market in 2022, especially relative to the number of openings. Making things harder, many available candidates have increased work-life balance expectations. Many candidates are receiving numerous offers and are seeking job security, roles in specific industries, a meaningful increase in salary, and other benefits like remote work or work from anywhere scenarios. Below are the results from a flash survey we sent out in early August of this year to data engineering professionals.

Reassess Interview Processes

Given the competitive nature of today’s hiring market, it is crucial for organizations to implement a streamlined and transparent interview process. Utmost clarity as far as the role and responsibilities, tool usage, and day-to-day work will allow candidates to get a clear and accurate understanding of what the role entails. From recent conversations with clients and candidates, it is evident that the interview process has seen some immense shifts over the past few years as companies have realized the importance of a streamlined and effective process to vet candidates for open roles.It is also notable that long-form technical assessments and coding exams are used less frequently in the interview process and companies are beginning to consider a gap in skill as something that can be closed through training and mentorship after a candidate is onboarded. To remain competitive, it is vital to keep your interview process efficient. To meet this need, interviews with no more than two rounds that utilize a panel-style interview where the candidate meets with multiple team members at once is increasingly common.

Work From Home (WFH) Continues to Evolve

With many companies evaluating their WFH and remote work policies going forward, a survey was done earlier this year to gauge candidate and client WFH preferences.The idea that remote work has opened doors for individuals to work in cities across the country is shifting quickly towards more frequent requests to relocate. This evolution in policies is generally translating to a hybrid model where individuals are expected to come into the office on a partial or as-needed basis. With that said, there are still plenty of roles open to those that are seeking a fully remote position, but they are not the majority of roles available as often assumed or reported by the media.Based on discussions with a wide range of companies looking to hire, it is evident that they are looking for candidates that are open to going into the office at least twice a week, and there has been a greater push across industries to return to office in some capacity. This trend will continue to be monitored.

What’s to Come?

Looking into the future, as we continue to see more hiring and investment allocated to building data teams, the demand for data engineering is poised for significant growth and continued focus. To read more about these takeaways and more, be sure to download the full 2022 Data Engineering Salary Report.