Data engineer, 15+ years, doubting AI engineering will last?
The answer is sitting in your own resume: Hadoop and Spark.
On a call this week, a data engineer with 15+ years behind her told me her honest doubt about the move into AI engineering:
“I doubt how many jobs will be there after three, four years.”
It’s a fair challenge. If AI is automating other engineering work, why would AI engineering roles be safe?
Instead of arguing, I asked her a question.
In her 15 years, what technology did she personally watch arrive, and watch companies adopt?
Her answer: Hadoop. Then Spark.
Look at that timeline. Hadoop arrived around 2011. Spark took over data pipelines around 2016.
Ten years later, companies still run on Spark. And the engineers who built those pipelines are still needed — maintaining them, evolving them, extending them.
That’s the pattern she had lived through without naming it.
A new technology doesn’t sweep through the economy in a year. Big tech transitions in two or three. Insurers, banks, logistics companies — far longer. She said it herself on the call: every company still has to make this transition.
And after the build comes the longest phase of all: running the thing, fixing it, growing it.
AI is at the very start of that curve. The engineers who build AI systems into companies over the next few years are the same people who will be maintaining and evolving those systems for a decade after.
Will the field change along the way? Constantly. But the people building inside it change with it — the Spark engineers of 2016 didn’t stay 2016 engineers.
The risk sits with the people waiting outside until it settles. It won’t settle.
Kirill
P.S. The transition wave is only starting, and companies need AI engineers for it. Want to be one of them? Book a call.