How I interview engineers to assess ability to deliver impact
Up until recently, I have personally been responsible for running all of our Deep-dive Interviews in our Product Engineer interview loop at Metaview. Our deep-dive focuses on the candidate’s ability to deliver impact. So we let the candidate choose the topic. All we care about in this interview is how they think about impact, and how capable they are at achieving it (relative to tenure). The objective of the interview is to find out as much as we can about the impact they are most proud of in their professional life, and what role they played in making that impact occur.
This post will demonstrate how we meet this objective through our “proven ability to deliver impact” interview that all engineering candidates face as part of their interview process at Metaview.
It's an in-depth article, because understanding the "why" behind the approach will help you make it your own. But if you want the headlines, check out this 1-page guide.
Interview preparation [pre-interview]
Deep-dives are not (and shouldn’t be) heavily scripted interviews. The interviewer needs to be able to adapt to what they’re hearing in order to truly “go deep”. So preparing for the interview is all about getting in the right mental state: one of genuine curiosity about how this person behaves when working.
To do this, I remind myself of what I want to discover, not the specific questions I’m going to ask. The things to discover as part of this interview are:
- How impact-oriented is the candidate? Do they think critically about where to apportion their skills and energy based on maximising customer or business impact?
- How do they work towards impact? Do they prefer to go alone or bring others with them?
- How broad is their understanding of the context in which they operate? Do they think about the problem-space and solution end-to-end and step beyond the boundaries of their skill-set to make it happen?
- How impactful have they actually been in their career so far? Did they achieve something that is genuinely impressive relative to their tenure?
By focusing on what it’s your job to discover as an interviewer, you’ll find yourself asking a higher volume of more meaningful follow-up questions.
In fact, you can see this in my personal Interview Metrics:
Candidate priming [minutes 0-10]
Getting the candidate in the right state of mind to go deep is just as important. It’s highly valuable to do this explicitly and verbally at the start of the interview, rather than just relying on the written interview description they’ve received.
As part of introducing the interview to them, I make sure I say the words:
“it might seem like we’re getting into very low-level detail, but just go with it”
and
“I want to feel like I was there with you and know exactly how you operated.”
It’s easier to show than tell, though. Here’s a Metaview recording of how I primed the candidate for the impact deep-dive in a recent interview:
I find this works well to ensure expectations are aligned, and candidates don’t second-guess whether they’re going down a rabbit-hole. I make it clear that, as the interviewer, that is my job to manage.
Once that is done, we’re ready to start the interview.
The assessment [minutes 10-55]
Low-confidence interviewers are often tempted to ask questions that attempt to make them look smart. The reality is that, for a deep-dive interview, the key is not to ask esoteric questions. The key is to get the candidate talking about a project or achievement that is fresh in the memory, and then to actively and critically listen in order to ask the right follow-up questions that help you meet the objective described earlier. The questions themselves should almost always be simple.
The assessment stage of the interview is split into three sub-stages:
- The opening, where you ask the the high-level question that the interview will focus on. The aim is to create shared context and set up the rest of the interview.
- The exploration, where you start to dig into various aspects of their answer in more detail. The aim is to assess the breadth of understanding the candidate had, and identify an area to deep-dive into.
- The deep-dive, where you choose one element of their answer to focus a series of detailed questions on. The aim is to learn how they actually work and how they make decisions.
Opening [minutes 10-20]
Kick-off the interview with an open-ended question that invites a descriptive response. At Metaview, it’s always some form of:
Can you tell me about a recent project you’ve been involved in where you’re especially proud of your impact?
“…recent project…” — you want it to be recent so that the memory is fresh and they can go into details. Additionally, it’s an interesting data point if the candidate’s most impactful work was done a long time ago.
“…especially proud…” — framing the question around something they’re proud of means that you’ll get a glimpse into the type of thing that motivates the candidate. Some are proud of hitting ambitious growth goals, others by hitting aggressive timelines, others by giving their colleagues increased leverage via internal tools.
“…your impact…” — you want to make it clear that you will be assessing their specific inputs to the project, not just evaluating the impact of the project as a whole, so they know to choose a project where their personal contributions were especially vital.
What to watch for
- Do they get excited when talking about the work? We’re reviewing the most impactful thing of their career here. If they’re not passionate about this, then they’re probably not motivated by impact.
- Do they understand the broader context of their work? The most impactful engineers think AND act beyond their job title. They know the details of the project beyond the role they happened to play. They know why the project was prioritized, what was wrong with the world before they did the work, and what broader objectives it played into.
- Can they communicate impact in a way that makes it clear who’s life got better as a result of their work? They could be internal or external customers, but we are looking for engineers that think backwards from the customer in everything they do.
You shouldn’t expect the candidate to cover off everything in their initial answer. But one of the advantages of starting with such an open-ended question, is that you get to see what the candidate naturally focuses on.
If they focus only on the work they themselves did, then they might have experiences that have rendered them siloed in their thinking. We can ask follow-up questions about the broader context in order to disprove that.
If they focus on the complexity of the project (ie, the journey) rather than the outcome, then perhaps they don’t tend to focus on the customer so much. We can ask follow-up questions about the end-user impact in order to disprove that.
Exploration [minutes 20-40]
The initial set of follow-up questions should focus on putting flesh on the bones of the story the candidate shared in response to the opening question.
What to ask
Depending on what the candidate revealed in their response to the opener, you’ll want to explore the following:
- What was the objective?
- A strong candidate has this front of mind. A weak candidate tries to work it out on the spot because they’ve never thought about it before.
- Why was this a priority?
- A strong candidate understands the why this was prioritized over other things. A weak candidate never bothered to ask so doesn’t know.
- Who was the customer / Who benefited from this piece of work?
- A strong candidate understands customer context and can clearly articulate how this impacted them. A weak candidate just knows that “the customer wanted it”.
- What was your actual role in building this?
- A strong candidate is specific about the things they built and contributed to the project beyond their job description. A weak candidate is vague and played a one-dimensional role.
This is not an exhaustive question list. Your job is to come away from the interview with clarity on these topics. Often this means asking multiple follow-up and clarifying questions.
Deep-dive [minutes 40-55]
By this point in the interview, the candidate has probably referred to a particular milestone or deliverable that they were the protagonist of within the broader project. If not, ask them: “Was there anything that was especially challenging about this project?”.
From there, you will seek to dive-deep. As a reminder, you want to come away feeling like you know exactly what sort of environment this person thrives in, and how they operate when they’re at their best.
What to ask
For demonstration purposes, here’s a (heavily abridged) run of questions I asked in a recent interview:
- What was the thing you found most challenging?
- How did overcome it?
- What did you do next?
- How did you go about deciding to take that approach?
- How did you communicate this decision?
- How did you actually communicate it, though? Was that via a meeting? a presentation? a slack message?
- How did the [stakeholders] react?
- What was the outcome?
Again, context is key. This is not a script. The most important thing is to remember that your job is to come away with as much signal as possible on how the candidate works, and break through any personal anxiety around drilling into detail.
Closing out [minutes 55-60]
The focus of this interview is very much on understanding the candidate’s ability to create impact. Any by this point, you should have a thorough understanding of how they operate when they’re at their version of their best.
To close out the interview, I take the opportunity to better understand what it was about the project that animated the candidate:
“Thanks for bearing with me and going into all those details. And congrats on your impact here. Final question from me: What was it about this project that you think motivated you to do your most impactful work?”
Not only is this an elegant way to close-out and reflect on what has been discussed, it can also be very revealing about the candidate’s true priorities. A good answer focuses on the customer feedback and impact as the thing that was motivating. An average answer focuses on the team being the motivating factor. A bad answer focuses on the technologies they got to use.
In summary
My general recommendation is to:
- Internalize that your job is to go deep into specific actions and decisions during this interview.
- Have the shape of the interview clear in your mind: Prepare, Prime, Open, Explore, Deep-dive.
- Tell the candidate that you intend to go deep, so “it’s almost as if I was there with you”.
- Start with a very high-level, open-ended question.
- Actively listen for parts of their story that you’d like to learn more about.
- Don’t be embarrassed to ask simple questions that put the ball in the candidate’s court (what happened next? why did you decide that? etc).
If you think this structure would work well for you too, then download the handy 1-page guide here. It’s perfect for referencing to keep you on-track while you’re running the interview.
Or, if you’re looking to spread high-quality interviewing within your organization more broadly, then check out Metaview or reach out to hello@metaview.ai.
