Tester Interviews: Techniques and Tasks

Reading Time: 8 minutes

 

 

This post is inspired by a recent discussion on the The Club at Ministry of Testing.

 

Edit: It has also been translated into Russian on Software-Testing.RU.  Please note that this translation was not requested, proofed, or approved by me.  I bear no responsibility for the content or accuracy of the translated post.

 

I’m a software tester with a past life in the IT recruitment industry.  I’m also a serial interviewee.  Immediately before starting my current position at MaibornWolff, I attended interviews with five companies before choosing the right opportunity for me.  In the previous five years, I’ve interviewed with a further five to ten, as well as conducting my own interviews as a recruiter and advising companies on how to conduct theirs.

 

The above post on The Club has inspired me to write about the different techniques that have been used during the interviews I attended this year, and the kind of tasks I was set.  I’ll also include some methods I’ve heard about being used, and briefly give my opinion on the advantages and disadvantages of each.  Hopefully this helps you to think about how you interview testers in your organisation, and perhaps inspires you to try some of the techniques described.

 

Please note that I won’t describe any technical coding tasks, as I don’t have enough personal experience of these to comment on their effectiveness.

 

 

Contents

 

Theory

Straightforward Questioning

Behavioural / Competency-based Questioning

Knowledge Questions

Industry and Community Awareness

Practical

At Home vs. On Site

Critical Thinking Games

Testing Requirements

Three Amigos

10 Minute Test Plan

Testing Example Software

Testing “Real” Software

Pairing

My Recommendations and a Pinch of Salt

 

 

Theory

 

For the purposes of this post, “theory” refers to traditional interview techniques that do not require interviewees to test in any way.

 

Straightforward Questioning

What it is: Usually generic, unstructured questions that could be asked in an interview for any role.  Examples include:

• What attracted you to this role?
• What does a normal day look like in your current job?
• Where do you see yourself in five years?

Main advantages: Allows interviewers to be direct about exactly what it is they want to know.

Main disadvantages: It’s very easy to predict and rehearse answers for this type of question, and interviewers are unlikely to learn anything about the candidate’s suitability for the position.

 

Behavioural / Competency-based Questioning

What it is: These questions usually start with, “Tell me about a time when…” and candidates are required to draw from real experiences (in work or personal life), or describe how they think a relevant scenario might play out if there are no real examples.

Main advantages: Gives interviewers an idea of how the candidate would respond in a real situation and can be scored to offer a more systematic way of comparing candidates.

Main disadvantages: It’s also possible to predict, practise and even fabricate answers for these questions.

 

Knowledge Questions

What it is: Whether written or oral, these questions are very similar to those found in a test or exam and are intended to determine candidates’ knowledge around specific subjects.  Examples include:

• What is the Agile Manifesto?
• What is regression testing?
• Name the layers of the test automation pyramid

Main advantages: Similar to “straightforward questioning” described above, interviewers can be direct in finding out where candidates’ knowledge lies; written tests can be easily scored and used to rank candidates.

Main disadvantages: Many things in testing are very complex and cannot be reduced to a “right” or “wrong” answer; even where facts are available, the ability to memorise and recite these is not related to candidates’ testing skills.

 

Industry and Community Awareness

What it is: Questions designed to determine how invested candidates are in developing as testers.  Examples include:

• How do you keep up with the latest developments in the testing and software industry?
• What testers do you admire?
• What testing events have you attended recently?

Main advantages: Helps interviewers to identify candidates who engage in testing-related activities outside of their normal roles and responsibilities.

Main disadvantages: Lack of community involvement does not necessarily indicate lack of interest or competency; candidates also may be restricted to the time and money they can invest outside of work (be careful not to discriminate).

 

 

Practical

 

For the purposes of this post, “practical” refers to interview techniques that typically require interviewees to test in some way.

 

At Home vs. On Site

What it is: Tasks or exercises that candidates are asked to complete either at home or as part of a face-to-face interview.  As well as testing tasks, this can include personality or general intelligence assessments.

Main advantages: Candidates might feel more relaxed completing tasks at home and so perform better, but interviewers can gain more insight by asking candidates to “think out loud” while running through exercises face-to-face.

Main disadvantages: Candidates might not have time to complete tasks at home (be careful not to discriminate) and could “cheat”.  Performing tasks during interview increases the time needed and candidates may be unable to get adequate time away from their current job or commitments in order to attend.

 

Critical Thinking Games

What it is: Problem solving or strategy games that are not necessarily directly linked to testing, such as the dice game or questions and assumptions; designed to demonstrate candidates’ critical thinking and problem solving skills.

Main advantages: As games may not be directly linked with testing, candidates may be more comfortable in showing their creativity and ability for abstract thought; may be particularly effective for junior candidates with limited practical testing experience.

Main disadvantages: The unfamiliar format may throw off some candidates; difficult to assess.

 

Testing Requirements

What it is: Candidates are asked to study a list of requirements and encouraged to ask questions and identify risks and gaps.  Candidates might also be asked to describe how they would test that the software meets these requirements.

Main advantages: Interviewers can gain an understanding of how candidates might test during the conception phases (shift left*), without needing to see the system under test; good if the successful candidate will have to explain their testing as part of the role.

Main disadvantages: This kind of scenario might not be a true reflection of the environment in which these skills are needed; it’s more likely that testers are not prompted for their input and need to be able to raise their concerns effectively, even when not invited to.

 

* I read a really good article on this a while back, but I’m having trouble finding it now and don’t want to link to Wikipedia 🙂

 

Three Amigos

What it is: Role-playing exercise based on the Three Amigos technique involving a tester (candidate) and two others playing the product owner and developer.  All three participants discuss requirements, user stories, and share ideas and / or concerns.

Main advantages: This exercise is likely to be closer to a real scenario than the “testing requirements” task described above, with the potential for other “amigos” to add personality challenges and counter-arguments; allows candidates to show how they participate in a group discussion and make their voice heard.

Main disadvantages: More challenging than some other tasks and candidates may underperform due to nerves or unfamiliarity with the format.

 

10 Minute Test Plan

What it is: Candidates are asked to create a test plan in ten minutes.  It could be written or drawn; on paper or whiteboard.  The time constraint is designed to force candidates to include only the most important aspects and see how they work under pressure.

Main advantages: Interviewers get a glimpse into how candidates create test plans and what they deem important to include; knowledge of test heuristics and mnemonics might also come to light.

Main disadvantages: Testers in different organisations might be required to include different information in test plans, rather than feel certain elements are less important; some candidates might never have been required to create test plans before, and so lack the necessary knowledge / experience.

 

Testing Example Software

What it is: Candidates are asked to test a publicly available application, or purpose built “dummy” software, as opposed to “real” software that the hiring company tests.

Main advantages: Interviewers can witness how candidates actually test software; “dummy” software can even be prepared with bugs or issues for candidates to find and investigate.

Main disadvantages: Example software used might be too basic or fundamentally different from the software that will be tested on the job to meaningfully predict performance.

 

Testing “Real” Software

What it is: Candidates are asked to test the same software that they would be testing if successful.

Main advantages: Interviewers can see how candidates interact with the software they’d actually be testing on the job, and identify approaches or skills that the existing team is lacking; candidates might find real bugs that were previously unidentified.

Main disadvantages: “Real” software might be deemed too complex or unfamiliar to use in an interview situation; it might be necessary to restrict candidates to certain areas of the application or workflow.

 

Pairing

What it is: Candidates engage in pair testing with an interviewer or team member; instead of testing alone and being asked to “think out loud”, candidates are required to to constantly communicate and work with their testing partner.

Main advantages: Interviewers and testing partners can see how candidates interact and work with others; some candidates might feel more comfortable having a working conversation with someone about testing if “thinking out loud” makes them feel self-conscious.

Main disadvantages: Some participants might be distracted by how to pair “properly”, as opposed to focussing on testing; some candidates might shrink back to give their partner room, while others might rely too much on theirs.

 

 

My Recommendations and a Pinch of Salt

 

The comments on each of the techniques and tasks above are, of course, purely a reflection of my own experiences and perceptions.  Others may have a different opinion, based on their own experiences.  There are also a number of other techniques that I haven’t mentioned, so this list is by no means the “be all and end all”.

 

When it comes to deciding which are best for your own interviews, I would always encourage you to think about your own context in the business, including the team hiring and the position you’re recruiting for.  What kind of skills and experience do you need the successful candidate to have?  Which of these techniques will help identify these most effectively?

 

Not all interviews must (or should) follow the same structure, and it’s fine to experiment until you find something that works well for you, in your context.  Just make sure that candidates interviewing for the same position have a similar experience, in order to keep things fair.  You should try different methods during the next round of recruitment, as opposed to subjecting directly competing candidates to different processes.

 

Generally speaking, I’d recommend using a mixture of techniques and tasks from both the “theory” and “practical” groups, as it’s important to understand candidates’ motivations and team fit, as well as their abilities and potential.  Different candidates will also be comfortable with different methods, so including a variation will provide more opportunities for candidates to shine in different areas, which you might not see otherwise.

 

I would discourage from only using “theory” techniques, as I think it’s better to let candidates show what they can do, rather than relying on what they can tell you about it.  I’ve heard  some refer to this as “interviewing vs auditioning”.

 

Not everyone is comfortable boasting about their good qualities, so some candidates might not do themselves justice when questioned, while others might exaggerate their abilities.  There are even suggestions that traditional interview techniques lead to worse decisions.

 

There are also a growing number of claims that interviews don’t work, so please take everything with a pinch of salt during the recruitment process.  Although practical tasks help, it’s impossible to accurately predict a person’s job performance based on interviews alone, no matter how many stages or tasks you put them through.  Of course, we’d all like to make the perfect hire first time around, but that’s what probation periods are for – to see how candidates actually perform on the job.  You’ll never know until you let someone try.

 

 

What do you think about the techniques and tasks in this post?  Please share your stories about what has and hasn’t worked for you – either on the interviewer or interviewee side – in the comments.