Myth #1: Software can be tested completely
Many books on software testing start with a discussion of the impossibility of testing all the data or paths in non-trivial software, but I’m still surprised by people who ask how long it will take to finish testing, without considering that being “done” is a subjective tradeoff between the risk of bugs and the cost of delaying a software release.
Myth #2: Testers can find all the bugs
Related to the impossibility of testing everything, a tester won’t find all the bugs, and likely won’t even find all the important bugs.
Myth #3: It is the tester’s fault when an important bug is missed or found late
Related to #2, I so often see blame focused on the tester when an important bug escapes or is found late. I’ve certainly felt it, and I’ve seen root cause analysis done that squarely identifies that the tester should have found the bug, with no consideration that the developer could have avoided the issue.
Myth #4: All bugs should be fixed
Changing code to fix a bug is a risk. There is the risk that some other side effect may pop up, or that customers rely and expect the current behavior. There are also can be design choices where 2 approaches are both reasonable, but they both have flaws, or two bugs are logged and choosing to fix one means the other can’t reasonably be fixed. Software teams usually triage all the bugs, but make hard choices to determine which should be fixed.
Myth #5: An automated test is equivalent to the same manual test
When I started testing, I felt that any test worth doing was worth automating. It took a while to realize that both the cost of that was prohibitive, and that there is value in testing closer to how the customer will use the software. Good manual testers notice anomalies that automated tests usually miss. I’ve heard it described as “peripheral vision”. Testing while thinking about how the software will be used can find bugs that would otherwise be missed by an automated test.
Myth #6: A quick passing automated test is free
Even if machine execution time is close to being free a poorly written test can be harmful, even if it runs fast and passes consistently. A passing test may require maintenance if the product changes. If the test covers code without validating the correctness, it will lead to a false sense of security if code coverage is used to assess the effectiveness of the test, bad decisions if code coverage is used to prune tests with redundant code coverage, and skews metrics on the percentage of passing tests.
Myth #7: There exists a code coverage percentage goal that makes sense
I’ve heard 90%, 80%, 60%, and 50% as reasonable code coverage numbers for black box testing, but none of those goals take into account the style of the code. Redundant error handling, when the caller and callee both check for an invalid condition can make for more reliable code as maintenance is performed. But redundant error checks lead to lower coverage. And if you use a library of useful routines but the compiler can’t optimize out unreachable code, then that also leads to unreachable code. When high code coverage is achieved, it is often through unit tests that tests code in isolation or fault injection to trigger conditions that won’t otherwise happen, and neither of these are representative of the real world use of the system.
Myth #8: Uncovered code is untested code
Since testing all data values isn’t feasible, we make assumptions about equivalence of classes of data values, and try to test at the boundaries. In code, a macro or inline that gets expanded in several spots should be equivalent if we trust the compiler and the usage is similar. But, if measuring block coverage, inlined code or macros may be tested in some blocks and not others, but it is the same code and we’d normally assume it to be equivalent.
Myth #9: Covered code has been fully tested
Code coverage measures the lines, blocks, or arcs that are tested. Even if you cover all the code, the coverage doesn’t measure whether your program was functioning correctly. It may have failed in ways that you didn’t perceive. There might also be untested data values that would have caused incorrect behavior.
Myth #10: The best way to test is the [exploratory/model based/…] approach
Whether it is model based, exploratory, requirements-focused, or some other, there are solid techniques that help us think of software differently. But focusing on a single technique means missing bugs that the other techniques find faster.
Myth #11: Software testing is a mundane job
New computer science graduates, who may have only been exposed to testing for a few hours in a software engineering course, often think of testing as mindless and boring. In the movie Elf, Will Farrell has the mundane job of testing Jack in the Box toys. He cranks each, and then braces for the puppet to pop up, and then moves on to the next one. That’s not what professional software testing is like. Testing requires creativity, alertness, and a passion for quality. There are sometimes mundane tasks of repeating tests or time-consuming environment setup, but a thinking tester usually finds creative ways to solve repetitive tasks.