How to Design Technical Interviews That Predict Work

A technical interview can produce a confident answer and still tell you very little about how someone will work on Tuesday morning.

That is the design problem. Interviews place candidates in an unusual setting, give them incomplete context, and ask them to perform while being evaluated. Some people become unusually fluent under those conditions. Others become less articulate than they are with colleagues, code, data, documentation, and time to think. If a hiring team treats the performance itself as proof, it may select for interview skill rather than job skill.

Generative AI makes the weakness easier to see. Candidates can rehearse behavioral answers, improve portfolio language, and practice likely questions. Employers can generate question banks, summarize conversations, and score application material. More polish is available to both sides, but polish is not the same as evidence.

The answer is not to make interviews more adversarial. It is to make them more deliberately connected to work.

This note uses a decision framework for that purpose. It is narrower than What Tech Teams Should Hire for in the AI Era, which discusses the qualities a modern technical team should value. Here the question is operational: how should an interview be designed so those qualities can be observed and compared fairly?

Start with an evidence map

Before writing questions, write down what the role requires and what evidence could reasonably demonstrate each requirement. This prevents the interview panel from inventing criteria after meeting a candidate.

An evidence map can be simple:

The fourth column matters. Hiring teams frequently use a convenient signal in place of a meaningful one. Fast recall substitutes for technical judgment. Extroversion substitutes for communication. Enthusiasm substitutes for sustained motivation. Experience at a recognizable company substitutes for individual contribution.

None of those signals is worthless in every case. The problem is that they are often accepted without establishing a link to the job.

The U.S. Office of Personnel Management defines a structured interview as an assessment of job-related competencies using systematic questions about past behavior or hypothetical situations. Its guidance emphasizes asking candidates the same predetermined questions and evaluating answers against the same rating standards. That is a useful baseline well beyond government hiring: decide what you need to observe before the conversation begins.

Choose the smallest method that can reveal the skill

Once the evidence map exists, choose a method for each important requirement. A conversation is enough for some capabilities. Others need a small piece of work.

For a data analyst, a panel might present a dashboard where revenue rose while completed orders fell. The candidate does not need access to private company data or three hours of unpaid analysis. Fifteen minutes of discussion can reveal whether they inspect definitions, time windows, refunds, duplicates, joins, currency, and instrumentation before declaring a conclusion.

For an AI engineer, show a retrieval-backed assistant that gives a fluent but unsupported answer. Ask for a diagnosis plan. A useful response separates retrieval failure from generation failure, requests traces or test cases, considers access permissions and conflicting documents, and defines how a fix would be evaluated. The exercise reveals more than asking for a definition of RAG.

For a backend engineer, offer a short pull request with a retry loop, an external API call, and an idempotency problem. Ask what they would review before approving it. This resembles work without demanding production-ready code from a candidate.

For a technical manager, use a scenario where product wants an AI feature released while security and engineering disagree about the risk. The signal is not whether the candidate chooses the panel’s preferred answer. It is whether they clarify decision rights, distinguish reversible from irreversible choices, surface missing evidence, and establish an accountable next step.

Realism does not require a large assignment. In fact, large take-home projects introduce new noise: free time, caregiving responsibilities, access to tools, and willingness to provide unpaid labor. The right exercise is the smallest one that exposes the behavior the role needs.

Structure the interview without turning it into a script

Structure sometimes sounds cold, but an unstructured conversation is not necessarily more human. It may simply give interviewer preference more room to operate.

A practical structured interview has five parts:

1. A defined competency linked to the role.
2. A core question asked of every candidate.
3. Permitted follow-up prompts that clarify evidence.
4. Behavioral anchors describing weak, acceptable, and strong evidence.
5. Independent scoring before panel discussion.

Suppose the competency is handling ambiguity in a data role. The core question could describe a stakeholder who reports that a trusted metric suddenly changed. A weak answer may jump to rebuilding the dashboard without checking the claim. An acceptable answer may confirm the definition, reproduce the issue, and inspect recent pipeline or business changes. A strong answer may do all of that while ranking hypotheses, protecting downstream decisions, communicating uncertainty, and defining how the correction will be verified.

The anchors do not need to predict every possible good answer. They need to help interviewers distinguish evidence from impression.

Follow-up questions still matter. “What would you inspect first?” and “What evidence would change your mind?” can deepen the conversation. Structure means comparable opportunity, not robotic delivery.

Score before the debrief because group discussion changes memory. Once a senior interviewer says a candidate felt strong, other panelists may reinterpret their notes. Independent scores preserve disagreement long enough to examine it. The debrief can then ask which observed behavior supports each rating instead of whether everyone “liked” the person.

Ask for decisions, not self-descriptions

Candidates know that employers value curiosity, ownership, communication, motivation, and teamwork. Asking “Are you curious?” only tests whether they know the desired answer.

Ask for a decision trail instead:

• What did you initially believe about a project that later proved wrong?
• Which technical shortcut did you accept, and what made it reasonable at the time?
• When did you stop investigating and decide you had enough evidence to act?
• What did you document so the next person would not repeat the same work?
• Tell us about a solution you simplified after feedback.

These are not trick questions. They invite a candidate to connect a claim to behavior. Strong answers do not have to be dramatic. A careful explanation of fixing duplicate events in a pipeline may reveal more judgment than a heroic story with vague personal contribution.

Communication should be assessed the same way. Do not rate accent, speed, charisma, or comfort with small talk as if they were communication competence. Observe whether the person listens, clarifies, separates fact from assumption, adapts the explanation to the audience, and states uncertainty without becoming evasive.

That distinction is especially important in data and AI roles. A person may be persuasive while explaining an invalid metric or an unevaluated model. Another may speak slowly while precisely identifying the decision risk. The job usually needs the second kind of clarity more than the first kind of fluency.

Decide the AI-use policy before the candidate arrives

AI use during interviews is becoming another source of inconsistent judgment. One interviewer assumes any assistance is cheating. Another expects candidates to use the same coding tools they would use at work. A third does not disclose that an automated system is summarizing or evaluating the interview.

Set the rule in advance and connect it to the role.

If engineers normally use coding assistants, a work sample can permit AI while requiring the candidate to explain, test, and own the result. This can reveal whether the person notices faulty suggestions, protects sensitive information, and validates behavior. If the goal is to observe a foundational skill without assistance, say so clearly and explain the boundary. A mixed exercise can include one short unaided segment and one tool-enabled segment.

Whatever policy you choose, apply it consistently. Tell candidates:

• which tools are allowed
• which part of the exercise they may assist with
• what information must not be entered into an external service
• whether the interview itself is recorded, transcribed, or summarized by AI
• what the evaluator will score: output, reasoning, verification, or all three

AI should not become a hidden proxy for integrity. A candidate who uses an allowed tool transparently and checks its work may show stronger professional judgment than someone who produces an unaided answer but cannot defend it.

Employers also need to examine their own automated selection tools. The EEOC’s guidance on employment tests and selection procedures explains that a selection method can raise discrimination concerns when it disproportionately excludes a protected group and is not job-related and consistent with business necessity. Adding a model, ranking score, or video analysis does not remove the employer’s responsibility to understand what the procedure measures and how it affects candidates.

Separate capability, context, and preference

Interview feedback becomes unreliable when these three categories are mixed.

Capability is evidence that the person can perform or learn the work: diagnosing a failure, reasoning about a system, writing maintainable code, explaining a tradeoff, or navigating ambiguity.

Context describes what shaped the evidence: the candidate misunderstood an instruction, lacked access to the normal tool, was unfamiliar with the company’s internal vocabulary, or needed an accommodation. Context may change how the observation should be interpreted.

Preference is what an interviewer personally enjoys: a conversational style, a career path, an educational background, a degree of assertiveness, or a shared interest. Preference may matter for a social conversation, but it should not quietly become a hiring criterion.

Panel notes should make the category visible. “Did not mention monitoring until prompted” is an observation. “May not be proactive” is an inference. “Did not have enough energy” is an impression with no stable definition. Moving from the third statement to the first makes the debrief more useful and more contestable.

This is also why vague “culture fit” scoring is dangerous. Teams do need people who can collaborate within real constraints and values. Assess those behaviors directly. Ask how someone handled disagreement, shared information, responded to a production problem, or raised a risk. Do not ask interviewers to decide whether a candidate feels like one of us.

Audit whether the interview predicts anything

Interview design should be treated as a system, not a tradition. A question does not become valuable because the company has asked it for five years.

After a hiring cycle, review the process:

• Do interviewers apply the rating anchors consistently?
• Which questions produce evidence that influences the decision?
• Are some stages rejecting candidates without adding distinct information?
• Do accommodations preserve the competency being measured?
• Are pass rates materially different across groups, stages, or interviewers?
• After hiring, do strong interview ratings relate to the relevant work outcomes?
• Are candidates being asked to demonstrate skills the job rarely uses?

This review needs care. Early job performance is affected by onboarding, management, team conditions, and assignment quality, not only individual ability. A small sample should not be treated as scientific certainty. Still, a team can look for obvious mismatches. If a whiteboard algorithm score has no relationship to the work engineers actually do, its continued use needs a better justification than habit.

Current market change makes periodic review more important. The World Economic Forum’s Future of Jobs Report 2025 reports that analytical thinking remains a leading core skill while AI and big data, leadership, resilience, and technological literacy are rising in importance. That does not mean adding an AI trivia round. It means checking whether the interview reveals how a person reasons, learns, collaborates, and uses changing tools responsibly.

For a broader view of selecting people for changing roles, Hire for Future Skills, Not Just Today’s Job explains why exact tool matching is fragile. The interview framework here provides the complementary discipline: future potential still needs observable, job-relevant evidence.

Give candidates a fair chance to recover

One weak answer should not automatically define the whole interview. People mishear questions, choose an unhelpful example, or become temporarily stuck. If the competency matters, give the candidate another route to demonstrate it.

An interviewer can restate the problem, provide missing context, ask the candidate to compare two options, or move to a second scenario measuring the same capability. Record that support was needed; do not pretend it was not. But distinguish a recoverable interview moment from a stable lack of skill.

This is not lowering the bar. It is protecting the measurement. If the role requires someone to respond to feedback, then seeing what happens after a prompt may itself be valuable evidence. A candidate who recognizes a false start and improves the approach may reveal exactly the learning behavior the team needs.

Candidates also learn about the organization through this process. Clear instructions, relevant exercises, consistent interviewers, and respectful recovery communicate that the company cares about work rather than theater. That strengthens the hiring decision in both directions.

The interview should resemble the standard of work

Good technical teams ask people to define problems, use evidence, review assumptions, explain tradeoffs, test outputs, and revise after feedback. Their interviews should do the same.

That standard is harder to game than a collection of clever questions, but it is also more humane. It gives polished candidates less room to win on presentation alone and capable candidates more than one way to show what they can do. It helps interviewers compare observations instead of defending intuition. It makes AI use a transparent design choice rather than a hidden contest.

Candidates still need to prepare. They should understand their own projects, practice explaining decisions, and be ready to work through unfamiliar problems. How to build practical AI skills for today’s tech job market offers guidance on creating the proof behind those conversations. But the employer owns the measurement system.

The useful question after an interview is not “Did this person interview well?” It is “What did we observe that gives us reason to expect good work?”

If the panel cannot answer with specific, job-related evidence, another round of interviews will not solve the problem. The design needs to change.