AI developers: what they do, the four types, and how to hire one

Virtual Minds
A developer at a cluttered office desk stares at screens turned away, capturing the challenge of hiring the right AI developers.

AI developers build software that uses machine learning models to do work a rule-based program cannot: classify a document, answer a customer, generate an image, decide the next step in a workflow. In 2026 most of them do not train models from scratch. They select a model, wire it to company data, define the prompts and tool calls around it, build the evaluation harness that proves it works, and ship it behind an API with logging, cost controls and a fallback for when the model is wrong. That last half of the sentence is the job. The first half is a weekend.

That distinction is why so many AI hires disappoint. A candidate who can fine-tune a classifier in a notebook is not the same person as one who can keep a retrieval pipeline accurate after the source documents change, or hold inference cost per user under a number the business can live with. Below: the four types of AI developer worth hiring, what each one actually ships, the skills that separate them, what the market pays in the United States, and a technical test you can run in ninety minutes.

What AI developers actually do, step by step

Strip away the job-board language and the work falls into six repeating activities. A good AI developer spends most of their week in three and four, not in one.

  1. Scoping the task against a model's real ability. Deciding whether the problem is a retrieval problem, a classification problem, a generation problem or a plain software problem that does not need a model at all. The best developers kill more AI features than they build.
  2. Getting the data into shape. Extraction, chunking, cleaning, deduplication, labelling, permissions. If your knowledge lives in PDFs, Slack threads and a CRM with three spellings of every company name, this is where the schedule goes.
  3. Building the loop. Prompts, tool definitions, retrieval, context assembly, state between turns, an approval gate where a wrong answer costs money. This is where agent behaviour is decided.
  4. Evaluation. A test set of real inputs with known good outputs, scored automatically on every change. Without it, nobody on the team can tell whether a prompt edit helped or quietly broke four cases.
  5. Deployment and cost control. Streaming responses, caching, routing cheap requests to a smaller model, rate limits, retries, a circuit breaker when the provider has a bad hour.
  6. Monitoring after launch. Watching quality drift as inputs change, keeping a log of failures a human reads weekly, and shipping fixes that go into the eval set so the same failure cannot return.

We wrote the longer version of this pipeline in AI application development: how production AI apps get built, including the parts that only show up at scale.

The four types of AI developers

"AI developer" covers four jobs with different day-to-day work and different failure modes. Hiring the wrong one is the most common mistake we see.

TypeWhat they shipCore toolsHire them when
AI application engineerProduct features built on hosted models: assistants, generation flows, document Q&ATypeScript or Python, model APIs, vector stores, queuesYou have a product and want AI inside it
ML engineerTrained or fine-tuned models, feature pipelines, serving infrastructurePyTorch, scikit-learn, Spark, feature storesYour advantage is proprietary data and a custom model earns its keep
Data / MLOps engineerIngestion, labelling, versioning, deployment pipelines, monitoringAirflow or Dagster, dbt, Kubernetes, model registriesModels exist but nothing reaches production reliably
Research engineerNew architectures, training runs, published resultsDistributed training, CUDA, evaluation researchYou are a lab, not a company buying AI

Most businesses asking for "AI developers" need the first type, sometimes with the third alongside. Posting for a research engineer and interviewing on transformer internals will filter out the people who would have shipped your feature.

Who the major developers of AI are

Two layers answer this question and they get confused constantly.

Model developers build the foundation models: Anthropic, OpenAI, Google DeepMind, Meta, Mistral, and in the Gulf the Technology Innovation Institute, which produced Falcon and the Noor model family. They absorb the training cost and publish APIs or weights.

Application developers build what people actually use on top of those models — the studios, the agencies, the in-house product teams. Virtual Minds sits here. We have run production apps on hosted models since 2022, across seven consumer apps and 500K+ users, with Room AI and Headshot AI both reaching #1 in their US categories. Headshot AI grew 450% after a rebuild. What we learned doing it is written up in Building on Claude: lessons from 500K+ users across seven apps.

The practical consequence for a buyer: you almost never need to talk to a model developer. You need an application developer who knows which model to use for each job and can switch when a better one lands. Virtual Minds is an Anthropic partner and is working toward the next partnership tier, with a target of ten Claude-certified team members.

The skills that actually separate strong AI developers

Every listing asks for Python, machine learning and "problem-solving." Those filter nobody. These are the signals that correlate with shipped work.

Evaluation discipline

Ask how they know a change improved the system. A strong answer describes a held-out set of real cases, a scoring method per case type, and a habit of adding every production failure to it. A weak answer describes eyeballing a few outputs. This single question sorts most candidate pools.

Cost arithmetic

They should be able to estimate tokens per request, requests per active user per month, and the resulting cost per user without reaching for a calculator excuse. Consumer AI dies on gross margin far more often than on model quality — the arithmetic is in the real unit economics of AI apps.

Failure design

Models are wrong on a predictable fraction of inputs. Ask what happens on that fraction. Good developers name the mechanism: confidence thresholds, a human approval gate, a second model checking the first, a visible "I am not sure" path. Developers who say the model rarely gets it wrong have not run anything at volume.

Data plumbing

Retrieval quality is usually a chunking and metadata problem, not a model problem. Someone who has debugged retrieval will talk about chunk boundaries, hybrid search, reranking and stale indexes. Someone who has not will talk about embedding dimensions.

Conventional software engineering

Version control, tests, code review, observability, on-call. AI features sit inside normal systems and fail in normal ways. A developer who cannot debug a 500 error cannot debug an agent.

A developer sketches an AI pipeline on a whiteboard in a co-working space, showing the hands-on work AI developers actually do.
Mapping a retrieval pipeline before writing a single line of code.

What AI developers cost in the United States

Compensation varies by type, location and whether equity is on the table, so treat these as bands rather than quotes, and verify against current market data before you budget.

  • Full-time hires. AI application engineers in major US metros command senior software engineer pay with a premium; research engineers at frontier labs sit far above that. Remote-first hires outside the Bay Area and New York come in lower.
  • Contractors and agencies. Billed hourly or per sprint. The rate is rarely the deciding number — delivery speed and whether they hand back maintainable code matter more.
  • Fixed-scope builds. A defined feature with an agreed evaluation bar. Works well when the scope is genuinely knowable, which rules out most agent projects on the first attempt.
  • Operated systems. A monthly fee where someone else runs the thing, including model upgrades and prompt maintenance. Cheaper than a first hire for a single workflow, more expensive than one once you have five.

The in-house versus outsourced decision is rarely about the hourly rate. It is about whether AI is a feature of your product or the product itself. We laid out the trade-off in building an in-house AI team vs running an operated AI system.

How to test AI developers before you hire

Portfolio links and model-trivia questions both reward preparation over ability. Run this instead. It takes about ninety minutes and works for full-time candidates and agencies alike.

  1. Give them a messy sample. Twenty real inputs from your business — support emails, invoices, listings — with four deliberately awkward ones. Ask what they would build and, more importantly, what they would refuse to automate.
  2. Ask for the eval plan first. Before any architecture talk: how will we score this? If they cannot define success in measurable terms, nobody will ever know whether the project worked.
  3. Price it out loud. Cost per thousand requests at the volume you expect, and what they would change if the number came back double.
  4. Walk a failure. "The system confidently gives a wrong answer to a customer. Walk me through detection, containment and fix." Listen for logging, replay and a regression test — not an apology.
  5. Read their code. One real repository, not a demo. Look for tests, prompt versioning, and whether the model call is isolated behind an interface you could swap.

Agencies get two extra questions: who specifically writes the code, and what happens to the system in month seven. The gap between the pitch team and the delivery team is where most agency engagements go wrong. More on that in AI agency: what one actually does, and how to choose.

What the best AI model for developers is

There is no single answer, and any developer who gives you one without asking about your workload is guessing. The useful framing is per job.

  • Long-context reasoning and tool use. Frontier models from Anthropic, OpenAI and Google. This is where agent loops and document work live.
  • High-volume classification and extraction. Smaller, cheaper models. Running a frontier model on a task a small one does at the same accuracy is pure margin loss.
  • Data residency or offline requirements. Open-weight families you can host yourself, including Falcon from TII.
  • Images and video. Separate specialised models entirely, chosen on output quality for your subject matter rather than general benchmark position.

The architectural answer matters more than the model choice: route each request to the cheapest model that clears your evaluation bar, keep the routing layer swappable, and re-test when a new release lands. That is how our apps stay affordable as volume grows, and it is the same pattern described in From one app to seven: the Cortex platform story.

Where AI developers fit next to packaged systems

Not every AI problem needs developers. If the job is booking more meetings, qualifying inbound leads or chasing a pipeline, a running system beats a build. LeadsMind does that work on Cortex, the AI operating system our products run on, and we run our own outbound on it — the mechanics are in how we run outbound on our own system.

Hire AI developers when the thing you want is specific to your business: your data, your workflow, your product surface. Buy a system when the workflow is the same one a thousand companies run. The expensive mistake is hiring a team to rebuild something you could have switched on in a week — and the other expensive mistake is buying a generic tool for a problem that is genuinely yours.

We build production systems — not prototypes. The difference shows up in month four, when the eval set catches a regression nobody would have noticed by eye.

If you are hiring in the Gulf specifically, the role, skills and local pay picture are covered in AI developer: what the role is, what it pays, and how to hire one. If you want to talk through whether your problem needs developers or a system, write to us with the workflow and the volume — that is enough to give you a straight answer.

  • ai-developers
  • hiring
  • ai-engineering
  • production-ai
  • ai-teams