“AI engineer” has become one of the least precise titles in hiring. It gets used for people building retrieval pipelines, people fine-tuning models, and people who added an API call to an existing product. Those are three different jobs with three different failure modes, and hiring the wrong one is expensive in a way that only becomes obvious in month three.
The real difference isn't the code
A strong software engineer can call an LLM API on their first afternoon. The API surface is small and well documented. What doesn’t transfer is everything around it: knowing that retrieval quality dominates output quality, that a prompt change needs a regression suite behind it, that cost per request compounds in a way page-load time never did.
The distinguishing skill is working with a component that is non-deterministic. Traditional engineering assumes the same input produces the same output. Once that assumption breaks, testing, debugging and even the definition of “working” all change shape.
A useful test. Ask a candidate how they’d know whether a prompt change made things better. If the answer isn’t some form of “evaluate it against a dataset of real inputs”, they’re a software engineer who has used an AI API — which may still be exactly who you need.
Which one does your project need?
Most products with an AI feature need a software engineer more than an AI engineer. The feature is one part of a larger application, and the hard parts are still the ordinary ones: data model, auth, billing, reliability.
- Software engineer — the AI is one feature in a normal product, using a hosted API in a well-trodden pattern.
- AI engineer — output quality on your own data is the product, and getting it right needs retrieval, evaluation and iteration.
- ML engineer — you’re training or fine-tuning models and need pipelines, serving infrastructure and drift monitoring.
- Data scientist — the question is analytical, and the deliverable is an answer rather than a running system.
What to look for in hiring
The strongest signal is someone describing a time an AI feature failed in production and what they changed. Nobody who has shipped one is short of stories: hallucinated citations, prompt injection through user content, a bill that tripled after a launch, a model update that silently changed output format.
Anyone can demo an AI feature. The engineer you want is the one who can tell you how theirs broke.
That’s why our vetting puts every engineer on a live task rather than a quiz. You see how they reason about the failure cases, not whether they can recite the vocabulary.
Posted by