Interviews for software testers vary with the company, job needs, and interviewer.  Here are some of the types of questions I’ve seen at Microsoft and other companies, and my advice to interview candidates for each.

Behavioral questions / “tell me about a time when …”
These questions try to find out how you’ve approached challenges in the past, since that can be a good predictor of your future.  For example, “Tell me about a time when you had a conflict with someone you worked with” or “tell me about a time when your work was late and a deadline was approaching”  These questions can probe time management, taking on new challenges, and thinking creatively.  Areas like the
Microsoft Company Values would often be covered with these sorts of questions.

Behavioral question advice
Try to relax, be yourself, and answer honestly.  Tell a good story that shows  you succeeding or learning from failure.  Think quickly about the beginning, middle, and end of the story and key points you want to make.   Don’t portray yourself too negatively.  An answer like “well I punched him and was arrested” wouldn’t be the best answer for a question on resolving a work conflict.

Testing questions / “how would you test…”
Experienced testers develop skills that help them test.  Unfortunately, testing skills are not taught in most computer science curriculum, so an interview for a new college hire often focuses more on assessing testing aptitude.  For someone without testing work experience, the interviewer is looking for passion and creativity, and the ability to think about what bugs could be in the code.  Example of these include “Test the font dialog in Word”, “How would you test a door”, “Test a shopping website”, etc. 

Testing question advice
Ask questions to resolve ambiguity and expand scope, if needed.  Expanding scope shows you can think about the bigger picture. Ask about the customer requirements. Ask how customers will use it. Ask about the developer / development process / previous bugs. Generate LOTS of test ideas!  Break the problem (or software application) into smaller units, and focus on each.  Equivalence classes / boundaries: group ranges of values that might be equivalent internally (e.g. month has classes 1-12, <1, >12, negative, non-integer, non-numeric, out-of-range). Within each range, pick exceptional / borderline values (e.g. month values 0, 1, 12, 13, -1, 1.1, “AA”, MAX_INT+1).  Non-functional test areas: think about tests for accessibility, performance, stress, concurrency, globalization / localization, security, usability, compatibility, and more.

Coding questions  / “write a program to …”
These questions attempt to assess the learned skill of programming.  Although I’ve worked with great testers who’d rather not code, it is considered an important skill at Microsoft, since testers need to express their test ideas in automation and tools.

Coding question advice
Ask questions to resolve ambiguity and ask to reduce scope, if possible, by asking if preconditions can be assumed.  Reducing scope makes it easier to solve the problem in a limited amount of time. If possible, pick the programming language with which you’re most comfortable. Write the function prototype first (identifying inputs + outputs). Ask about preconditions for inputs. Think out loud. Think about the algorithm!  Before writing the code, describe some tests and consider special cases.  Use standard API’s, STL, class libraries if you know them.

Code reviewing questions / “spot the defect …”
These questions attempt to assess the ability to spot defects in code.

Coding reviewing advice
Think out loud.  Ask about any language features that you’re not sure about but might matter.  Skim the code quickly then look in more detail.

Puzzle questions / “there are twelve coins and you have a scale …”
Microsoft generally doesn’t ask these sorts of questions anymore, since many at Microsoft think they aren’t good predictors of work behaviors.  These were common years ago at Microsoft.

Puzzle question advice
If you do get asked such a question, ask more about the problem and think out loud.  Don’t stress if you don’t get the answer.

Domain questions / “From your pet grooming experience, how would you trim ..”
Do you have experience with statistics, accounting, car manufacturing, security, performance, grocery store design, or some other domain relevant to the position?  Expect questions to test the depth of your knowledge.

Domain question advice
Don’t claim to be an expert if you’re not.  If the interviewer knows more about a narrow area than you, it reflects badly on your claims of expertise.  Be honest about what your limitations are and highlight real experience in the area.

More good interview details are on Steve Yegg’s blog about the google interview and the Microsoft jobs blog about the SDET interview.

Chef Mario Batali was in town this week for questions and a book signing.  Although the topic was food, I took back some tips that help me approach software testing.

  • Burn some food tonight  Mario spoke about the intense caramelization that happens to food right before it burns.  His advice to aspiring chefs was to go home and burn some food, to witness the point right before burning.  Similarly when testing, right before software breaks it can exhibit weird behavior that a keen software tester will pick up on.
  • Simplicity vs. complexity  His preference when dining out was simple food.  Software has a challenging balance between the complexity needed to support features and simplicity for ease of use.  As long as important features are present, simplicity likely wins.
  • Certification doesn’t guarantee success  There are Michelin 3 star restaurants across the company, and Mario indicated that all they guaranteed was that rich people could get the same food everywhere.  He much preferred local ingredients; seafood in Seattle or red beans and rice in New Orleans.  Is software testing certification needed for success, or will it result in the boring consistency as the certified restaurants?  Not sure on this, since I’m not certified, and not aware of certified testers.
  • Employees matter  Mario spoke about the need for career development and happiness in his employees.  Software companies that care about employee development can have more engaged employees resulting in a better experience for customers.
  • Confidence vs. arrogance  I found Mario to be confident, but I know people who consider him arrogant.  A wise lesson for any work environment is the fine line between the two.
  • Celebrity chefs  Mario was asked if the celebrity chef phenomenon is good or bad.  He said it was working well for him.  Similarly in testing, should all the ideas of famous testers like Cem Kaner or James Bach be taken more seriously because they are famous?  Is this at the expense of talented unknown testers?

I wasn’t enough of a fan to wait for an autograph, but I was impressed by his attitude and approach to business, and his obvious passion for his work.

Test tip #18: keep learning  

When I was a developer, my brightest teammates read books like “Effective C++”, “Design Patterns”, and “Programming Pearls”.  They also kept up to date with new programming language features and software development techniques.  These were computer science graduates who already had years of training, but were excited about learning more.  

Sadly, many testers I meet aren’t as enthusiastic about learning about their craft.  Many testers, even those who have been testing for 5 or more years, aren’t aware of books, blogs, conferences, or test talks.  Nor did they learn much about testing in school.  

For the rare few who have read a book on testing, most can’t recall the author or title.  To continue the discussion, I pull out my testing eyechart, which I have printed on an 11″ x 17″ page.  

Software Testing Books

Software Testing Books

 

I haven’t read all of these books yet, but I do have my favorites, like “A Practitioner’s Guide to Software Test Design” for new testers wanting to learn test design, or “Lessons Learned in Software Testing” for experienced, cynical testers.  I’m not a fan of any of Boris Beizer’s work.  My attention span can’t appreciate “Testing Object-Oriented Systems: Models, Patterns, and Tools”, but it seems popular at Microsoft.  I still like “Testing Computer Software”, even though it is getting dated.  There are now many more software testing books than these, and lots of more general books on thinking and software engineering, but I like the chart as a conversation starter.  

Did I miss your favorite book on software testing?  If so, let me know.

I started to make a list of 10 tips I’d give to junior software testers.  But then I saw a talk by Ben Huh, of lolcats/icanhascheezburger fame.  Ben made the point that, with the internet, content is free, but the organization, editing, and presentation all require skill.

Inspired by Ben and the cheezburger franchise,  I asked 60 successful testers to each provide 3 tips they’d give to a junior tester.   I received tips back from more than 40 of the testers, and ended up with a list of more than 100 tips.

To respect their privacy, I won’t provide the verbatim tips here, but I did find it interesting that there was a lot of commonality and the tips were collectively much better than what I came up with.

I grouped the collected testing tips into these 19 themes.

1. Focus on the customer  Keep the customer in mind when testing.  Develop empathy for their needs.  Talk to customers and observe them using your software.

2. Read bugs  If you work with a group of testers, read all the bugs they log each day, especially any logged in the area you’re testing.  You can learn a lot from how other testers approach bug finding.

3. Read code  Find the code that implements your feature.  Even if writing code isn’t your thing, reading code critically helps find potential boundaries and flaws.

4. Take pride in your bugs  Bug advocacy starts with a well written bug title and description. I re-read the bugs that I log to assess whether they are fair and properly detailed.  For important bugs, if they aren’t getting fixed, be the driver to make sure the right decisions and tradeoffs are being made.

5. Get involved in your feature’s design Before code is written, find out about the planning and get involved while big changes are still possible.  This also helps to understand the tradeoffs and compromises that are being considered.

6. Design your tests Whether it is finding boundary values, applying combinatorial techniques, drawing pictures, or creating models, it helps to put thought into your test design.  During exploratory testing, consciously alternate your test planning and product learning.

7. Know your feature  Whatever features you’re testing, you should know the design, limitations, bugs logged by others, code churn, and interactions with other features.

8. Work with others to test your feature  Work with testers with different expertise to test your feature.  Brainstorm test ideas and ask for their feedback.

9. Learn your product  Even if you’re only responsible for a small feature area, becoming an expert on your product and the other new features can help you be a better tester.

10. Foster good relationships with your developers  Testing can be adversarial, and it is easy to flip the “bozo” bit on some of the people you work with.  To find out the latest and get your bugs fixed, it helps to develop strong relationships with the developers that fix your bugs.

11. Expand your scope and network  Successful people have strong networks they can rely on for expertise and advice.  Work both within your company and externally to meet new people and develop professional connections.

12. Find mentors/role models I’ve worked with some amazing testers, and learned a lot from them.  To improve your skills, look for mentors to meet with, or role models to emulate.

13. Be self-critical  Testers are great at finding flaws in software.  It can help to turn some of that attention inward, and identify what you need to improve.

14. Manage your time It is easy to get focused on big tasks or distracted by meetings, and not make time for learning, bug hunting, or a healthy life.  To avoid burnout, learn how to manage your time.

15. Automate wisely Automated tests may lack the same “peripheral vision” as a skilled tester.   When done wrong, automation can lead to spectacularly unmaintainable code that doesn’t provide value in assessing product quality.  But careful automation can provide value in noticing defects faster.

16. Improve your coding skills  I’ve met some talented testers who’d rather not write code.  Arguably, just as movie critics become jaded and don’t appreciate the move-going public, when I put on my coding hat I think differently from customers.  But coding is a valued skill, and it can help you become better at reading code and understanding the internals of your product, as well as become better at small tools to make mundane tasks easier.

17. Attend triage  In the final days before shipping, triage teams review bugs to decide what to fix and what to postpone.  If you don’t normally get invited, ask to attend.  It is fascinating to see the roles of tester reputation, customer impact, and perceived risk balanced to make decisions.

18. Keep learning  Whether it be a soft skill,  like public speaking, programming language, or new test technique, successful testers take time from their busy days to keep learning.

19. Enjoy what you do, and do it well  If you can’t afford to leave your job, you need to learn how to love it.  Testers can get cynical, especially during challenging release cycles.  Testers who enjoy their work, and exceed expectations will do well.

Special thanks to all those who provided tips.  For privacy, I’m not listing the names here, but I’ve emailed you all to thank you.  I learned a lot from all of you.

Am I an expert?

Posted: May 1, 2010 in Software Testing

Ottawa Senator Andy Sutton’s evasive interview shows the perils of claiming to be an expert.

For every software testing area I’ve studied, practiced, and had some success, I’ve been able to find role models who know more about some aspect of the area than I do.

When I’m asked if I’m an expert, I describe my related experience in an area and some of my “known unknowns”, areas that I know I don’t understand yet.  Then I start asking questions to understand their problem and context, and see if it relates to what I’ve seen work or fail before.

During a job interview, claiming to be an expert can be especially dangerous.  If an interview candiate claims to be an expert in some area, the interviewer will usually ask some hard questions  that they expect an expert can answer, to probe their depth and breadth of knowledge.

I admire those testers who can defend why they are experts, and especially those who encourage others to become experts.

But me?  I’m still learning.

I’ve found cool bugs while stepping through code and thinking about what could go wrong.  I’ve been asked by other testers for my tricks.  Attaching a debugger and stepping through code can help you better explore your application and discover hidden boundary conditions.  These are steps that I use.

1. Attach or start under a debugger.  I usually use Visual Studio, but whatever debugger you choose, you need to know how to attach to the process of interest or start the process under the debugger.

2. Find the source (and symbols). For C++, it really helps to have access to source code and symbols!  Even for languages with good decompilers, like Java or C#, it helps to have the original source code with comments.  Many companies restrict access to source code, but if you’re a tester at that company, you should be able to request access.  I have stepped through the dissassembled binary for C++ and been able to find bugs, but it can be a painful and tedious process.

3. Find code of interest.  I’ve debugged large applications, and it amazing how the abstraction layers can make the code of interest difficult to find.  

  • If you are on a dialog, the break execution, and look down the stack to find the stack frame with the application logic.
  • If you know the application read or writes the registry, filesystem, or network, then set a breakpoint on common API’s for doing so. 
  • If you can guess function or variable names, use code searching to find it
  • If you can find fixed bugs in the same area you are interested in, use the code fix diff to spot the issue.

4. Spot problematic code.  After a while, bad code will jump out at you.  Some things to look for:

  • improper (or missing) error handling
  • uninitialized variables depending on code path
  • “todo” comments that indicate the code is unfinished
  • confusing code logic, especially and’s and or’s  in if conditions

To improve your skills, play “spot the defect” games, look at fixes for bugs, and learn what API’s return on error.

5. Force execution and data values.  Cheating can be ok.  If you force execution to a line of code, or set a data value in the debugger can be a fast way to prove hunches. You then need to come up with a test case without debugger tricks to get the same result to make it a real bug.

6. Deal with confusion due to optimized code.  Especially with C++, stepping through an optimized build can be confusing when common code gets collapsed, variables are in registers and discarded when no longer needed, and unneeded code is optimized out.  I usually find that, when confused, stepping through the dissassembled machine code helps.

7. Log  bugs with code details.  If you find a defect while stepping through code, and you can reproduce it without a debugger, then you can log a very convincing bug that demonstrates the issue + pinpoints the problem.  This will help build credibility with developers, since it reduces their debugging time.

8. Search code for similar defects.  Developers often make the same mistakes repeatedly, or copy/paste bad code. Once you’ve found a defect by stepping through code, search the codebase for similar code, and review it to see if the same problem is found.

9. Practice with unfixed black box detected bugs.  Having trouble finding bugs this way?  To practice, take some black box bugs that you or someone else found, then make it your mission to hunt down the defective code faster than the developer can.

At first, you may find that this is a slow way to find bugs.  With practice you may spot cool bugs that would otherwise have escaped.

It has been a few years since I attended a general software testing conference, although I have been fortunate to go to security conferences including CanSecWest and BlackHat. 

I was thrilled to be accepted as a speaker at StarEast 2010

A few things I learned:

  • Talking with people is as important as the sessions.  I already knew this, but I tend to be shy and introverted.  It is an effort to engage strangers and discover their stories.  A lot of the value is what I learned informally from others.  Even on the way to the airport had an interesting discussion about the changes than come when a company is acquired.
  • Conferences can be both exhausting and energizing.  I’m not used to spending hours listening to concepts.  Add in the time talking to other speakers and delegates, and it can be very tiring.  But I also feel energized with new ideas.  It is easy for the energy to fade once back at work.  With other pressing issues, the bright ideas from the conference can fade and become memories, and not change how new work is approached.
  • Tools are valued more than ideas.  My talk was about ideas, not tools.  It should have been more tools focused, to make the lessons more transferrable.
  • Open source is everywhere, especially selenium.  I joined Microsoft 5 years ago.  I feel I’ve been isolated from the explosion of open source tools for testing.  Am I the only tester who hasn’t used selenium for web testing?
  • Twitter adds value to the conference shared experience.  I’ve noticed this with cansecwest and blackhat, although the attendees at those conferences can be more mean spirited.  I didn’t attend CanSecWest this year, but felt I got the major points of the talk from the twitter feed.  The StarEast twitter feed wasn’t as comprehensive, but it did add value, and increased the shared experience.
  • I signed up for twitter too late.  Apparently, it can take 4 days or more to get indexed, so I wasn’t able to join into the chorus, since none of my tweets showed up in the stareast feed.
  • Executable test design is realistic.  Great last minute presentation with fitnesse and selenesse.  I’ve been playing with executable test design for years.  I tried fitnesse years ago, and tried custom solutions with action words in Excel and test tables in Word.  I never achieved the readability I wanted.  The Slim scenario tables seem to achieve this, and the demonstration at StarEast was convincing.
  • and lots more… great presentations on agile, tester reputation, test design, automation, and thinking like a tester.

Overall, it was a great experience attending StarEast. I’d recommend SQE conferences. It was well run, and an honor to meet so many bright testers.