Leadership · Data
From data analyst to data engineer, through one good question.
Our first data ingestion pipelines ran on an established open-source tool. Instead of choosing a replacement myself, I asked our data analyst to find out whether something better existed. He researched the options, made the case for a modern connector-based tool and led the switch. Our pipelines got faster, and he grew into a data engineer. This is the approach, and how AI assistants make it easier today.
Give people a real decision to research, a framework to judge it, a time-boxed proof of concept and the authority to act on the result. You get a better decision and a more capable engineer. AI assistants now shorten the path from analyst to engineer, as long as the learning, not just the output, stays with the person.
The situation
At Cedar Planters I was building the company's first data platform. Our first ingestion pipelines ran on an established, flow-based open-source tool. It worked, but I wanted to know whether more modern, faster options existed. I had my own opinions about alternatives, and it would have been quicker to just pick one.
The question
Instead, I asked our data analyst: is our current tool still the right one for us, and if not, what is? The framework started from what he knew best. He had deep, hands-on experience with our current process, so his first job was to use it: map how data moved today and find where it was slow. From there, we made space to exchange and compare architecture options as a team, with one clear goal: improve the performance of that process. He had the time to test the options properly.
The landscape he explored
Data tools do different jobs, and part of the learning was seeing where each kind fits:
| Kind of tool | What it is best at |
|---|---|
| Flow-based routing tools | Visual pipelines that route and transform data between systems |
| Code-first orchestrators | Scheduling and running pipelines written as code |
| Connector-based ingestion tools | Ready-made connectors that extract data from many sources and load it into a database or warehouse |
| Lightweight workflow engines | Fast orchestration of scripts and small workflows |
| In-warehouse transformation tools | Modelling and testing data after it lands |
What he found
He tested the options against our real sources and came back with a clear recommendation: a modern, connector-based ingestion tool, with the evidence to support it. I didn't hand him the answer. I helped him through the proof of concept and the rollout, and let him lead the transition. We later paired it with a lightweight orchestrator, and built central reporting on top.
What changed
- The stack. Data pulls became faster.
- The person. He moved from an analyst's mindset, using data, to a builder's mindset, designing how data moves. He now works as a data engineer.
- The team. Everyone saw that anyone, whatever their title, could question the stack and be taken seriously.
The guided discovery method
- Pick a real decision, one that matters to the business and stretches the person.
- Start from their expertise. Ask them to map the current process they know best, and agree on the goal, such as performance, before anyone looks at tools.
- Time-box a proof of concept on real data.
- Ask for a recommendation with evidence, and challenge it the way a senior engineer would.
- Hand over the rollout. Ownership is where most of the growth happens.
- Reflect together on what worked and what they would do differently.
Where AI assistants fit
The jump from analyst to engineer used to mean months of learning Python, APIs, testing and deployment before being productive. AI coding assistants have shortened that a lot. Used well, they help a growing engineer:
- draft connectors, SQL transformations and tests, then learn by reviewing and fixing them;
- get unfamiliar code or error messages explained in plain language;
- compare tools quickly at the research stage, as a starting point for their own testing.
Two rules keep it healthy. First, AI summaries of tools can be out of date or wrong, so every claim gets checked in the proof of concept. Second, the mentee should be able to explain every line they ship. If AI does the thinking, you get output without growth.
Why it works
Telling someone the answer teaches them the answer. Letting them find it, with a framework and support, teaches them how to make the next decision without you. For a small team, that is how leadership capacity grows.
More on the platform this was part of: A data platform from zero at Cedar Planters.
Get my next case studies and articles in Google Search.