DFT benefits for developers — and entire organizations

DFT benefits for developers — and entire organizations

Changing market forces are making design-for-testability tools a more critical part of the savvy design engineer’s toolset

BY DAVID GEHRINGER
Fanfare
Mountain View, CA
http://www.fanfaresoftware.com

Design for testability (DFT) is not a new concept. But the reasons why developers need to test — and can benefit by testing earlier in the product lifecycle — are changing. Thanks to advanced development tools that enable faster feature design and creation, testing requirements and feature complexity are increasing exponentially. As a result, half the time required to get a product to market is now spent on testing — the same amount of time that is spent on development. More importantly, the longer companies wait to test in the product lifecycle, the longer it will take to get their products to market.

Keeping pace with development has become nearly impossible for most testing organizations. No matter how hard testers work, the cycles continue to stretch. Developers, however, may hold the key to easing this burden. By designing for testability, developers can help get products through QA faster and help their companies deliver high-quality products to market ahead of competitors.

DFT defined

Designing for testability describes a thought process that developers go through as they develop a product. It can be as simple as asking themselves: How is this product going to be tested? What could I do to make testing easier? How could I make tests easier to automate?

In response to these questions, they might make sure that certain features and commands can be easily accessed, such as a direct CLI command, or that specific commands or outputs are revealed for testing purposes but kept hidden from users. They might even create a small harness or an API to make testing easier.

Few developers, however, are quick to embrace the concept of designing for testability. Understandably, many view it as an extra task, one more thing that can hamper their productivity. Others are concerned it will restrict their design freedom.

Yet with the right tools, designing for testability can actually accelerate their design and its acceptance, as bugs can be caught earlier and resolved more easily. In addition, each time a build of a product is tested, the investment made by developers can pay dividends not only for engineering but for QA, with faster times to market and better quality products.

Debugging early

Many developers are already using or starting to adopt extreme programming (XP) and agile programming approaches to development. By nature, this iterative approach to development requires more frequent testing. Perhaps more importantly, this approach is predicated on testing at the development stage (see Fig. 1 ).

Fig. 1. Reiterative testing in the development stage can ultimately speed design as well as production.

By building testability into the product, agile developers can participate in this early-stage debugging without slowing down on the development front. They can leverage testing tools that make the act of testing more automated, reducing the effort needed to test and ensuring high quality and testing coverage.

Developers typically need to script, code, or build a harness when they unit test. With an automated testing tool, they can quickly test and run the same test case every time they do a build. The only time investment they need to make is learning how to use a new tool.

Reusing test assets

Developers can further improve efficiency across their organizations by creating test assets that both testers and developers can leverage. They simply need to build testability into the product and then use standard testing tools to do the testing and reporting.

The resulting test assets can help QA teams quickly understand which feature aspects need to be tested and how they are supposed to work. QA can then get to work right away, trying to break them. This not only saves time by eliminating guesswork, but also prevents confusion, which can lead to missed bugs.

For most developers, reproducing a QA-reported problem or bug is half of the challenge. If they have a test case that can do that—replicate the same actions to see the bug and eliminate other variables—then half of their work is done.

Developers who have a test case to reproduce the bug also can solve the bug two to five times faster—and validate the fix. For this reason, mature organizations typically add the test case to the developer and QA regression testing for subsequent releases.

Expanding DFT’s boundaries

The pursuit of quality can be a frustrating journey for network equipment manufacturers and their customers. It often follows a cycle that begins with customers finding bugs and then both sides growing frustrated as the manufacturers try to reproduce the problems. But until they can reproduce the bugs, manufacturers cannot fix them.

Customers often exacerbate the matter by introducing many variables to the process, such as diverse testbeds, different use models, complex system testing, and proprietary equipment. This adds to the frustration by making dialogue difficult and time consuming.

This process could be greatly simplified if developers and testers on both sides had an easy way to communicate the details of the test — such as configuration, devices, and responses — in a clear, consistent manner, regardless of linguistic or geographic differences (see Fig. 2 ). Often, though, test cases are written in languages, whether spoken or scripted, that are difficult to understand by all groups working on a project.

Fig. 2. Having a way to clearly communicate the details of the test greatly simplifies cooperation among diverse groups.

By building testability into the product and leveraging automated testing tools, developers can improve efficiency for both manufacturers and their customers. First, they can create test cases written in the language of the device, so they can be universally understood. Second, they can create reports for each test execution that definitively capture and display each device used in the test, as well as its configuration, and that capture and represent the entire response or display to every command.

Manufacturers can then take steps to include the corresponding test cases in their testing suites, preventing a recurrence of those bugs. In this way, both manufacturers and their customers can better meet their quality objectives.

A real-life example

Fanfare, an agile software shop, has created a company exercise that underscores the value and benefits of designing for testability. Every month, the company pulls developers off their projects for a day or two. The QA manager writes use models and gives developers a pass to break someone else’s code and hunt down bugs. Through this exercise, Fanfare can extend its QA team for a few days, and developers gain a fresh perspective, approaching the product as new users would and working with parts they did not write.

Besides giving developers bragging rights if they find bugs in the CTO’s code, this exercise helps them learn more about the product and better gauge the quality of their own code. At a minimum, it creates a semblance of checks and balances among developers and testers. The largest benefit of this experience, however, is that developers come away motivated to design for testability. Because they know that at some point they will need to get “dirty” testing, developers feel an incentive to build testability into the product to make this process easier. Some developers even provide ideas to the QA manager on what should be tested.

Making DFT a reality

By incorporating testing earlier in the design process, organizations can make testing easier and more appealing to developers. In addition, designing for testability can strengthen the working relationship between developers and QA, while streamlining the overall testing process to improve quality and speed time-to-market. It also can give companies a competitive edge as they build an automated testing process.

Despite the benefits, few developers enjoy testing. For this reason, learning more about automated testing tools, particularly those that can integrate with and complement existing homegrown automation systems, is an important next step. With the right testing tools, developers can streamline unit testing by easily reproducing and then solving bugs. They can then focus on design and leave any heavy lifting to the testing team. ■