When developers and engineers measure code coverage in embedded systems they can improve their device’s safety and performance. Embedded systems play a foundational role in countless everyday devices. An unchecked code error can be bad for business and potentially dangerous.
Developers and engineers should be aware of why code coverage matters and what they can do to increase coverage in their testing processes.
Here’s why code coverage is important
High code coverage indicates that a large part of a system’s code is being tested. Developers may find that their code appears to be bug-free and working well. However, if only a small portion of the code is running, there could be hidden errors or bugs that do not show up in test results. So, code and test effectiveness both rely on getting code coverage as high as possible.
While it may not always be feasible to reach 100% code coverage, there are several reasons why it is important to prioritize a higher percentage.
Higher quality products and systems
The clearest benefit of increasing code coverage is higher quality code and products. Higher code coverage indicates that a larger majority of the code is tested. As a result, products will ship with fewer bugs, leading to a better end user experience and better overall product performance.
Surveys show that 89% of developers believe that “undetected errors” result in a negative impact on business. When code coverage is low, there are likely conditions that are not getting a chance to run in test cases. This includes critical cases that could potentially cause the program to fail or misbehave.
Improved safety and compliance
(Source: Shutterstock)
In some industries, it is a serious matter of user safety to measure code coverage on embedded systems. This is especially true in fields like medical, automotive and aerospace. Software is becoming increasingly crucial in industrial spaces and in higher risk products, such as cars or planes. Even seemingly harmless devices like IoT devices are becoming high-risk today.
A bug or glitch in these types of devices could be life-threatening to the user, so achieving high code coverage is critical for ensuring the device meets safety standards. This is particularly important from a cybersecurity perspective. For instance, Mercedes-Benz made headlines in 2020 when hackers at a security convention revealed numerous security bugs that allowed them to remotely control Mercedes-Benz cars.
Luckily, the hackers in this instance were security researchers, not malicious hackers. However, this is a perfect example of its importance in high-risk devices like vehicles. With things like apps or video games, developers can wait until users report bugs to fix them. That approach is not an option with high-risk devices, though. In these cases, waiting for users to experience bugs could seriously endanger someone’s well-being.
In fact, many industries have safety regulations that require developers to meet certain testing standards, such as ISO 26262 for the automotive industry. High code coverage is part of meeting these safety standards, indicating that a system is thoroughly tested for errors.
More effective system testing
Higher code coverage can lead to a more efficient and effective code testing process. While correcting bugs may be tedious for developers, going back repeatedly to correct missed errors is even more tedious. High code coverage minimizes the likelihood that a bug or error slipped through the gaps in the code testing phase.
There are different types of coverage metrics that developers can use to pinpoint what parts of their code is being run when tested. For example, a statement coverage metric will tell developers what percentage of statements in their code are running at least once in tests. By using a variety of coverage metrics, developers can be more deliberate about choosing tests for their code, allowing them to save time.
For example, a program might have high function and condition coverage but low branch coverage. This tells the developer they need to run more tests that run the if/else statements in their code.
How to increase code coverage in embedded systems
Increasing code coverage can seem easier said than done. There are a few key strategies that developers and engineers can use to improve code coverage on their embedded systems.
Leave sufficient memory space for tools
Conserving memory space is an easy way to improve code coverage. Running code coverage tests require some minimum software and hardware resources, particularly memory. Many of these measurement tools have to add a bit of code to run, such as a counter, requiring the use of some memory space. If there isn’t enough available memory, it won’t be possible to effectively measure code coverage.
Try lower-resource measurement tools
If memory or other computing resources are a challenge, developers can try using less resource-intensive measurement methods. Sometimes it is simply not feasible to use a full 32-bit code coverage tool, whether due to memory or other technical limitations.
In these cases, alternative measurement tools may be a better fit. For example, partial instrumentation code coverage analyzers can be as small as 16 or 8 bits. Partial instrumentation counters may have higher processor needs, but they require less memory.
Developers may benefit from the boundary scan or on-chip debugging method. This approach tests code coverage without needing to add any new code. This means it does not require excess memory space like other analysis methods.
Check for different types of code coverage
Code coverage can refer to general coverage for a whole program or it can home in on specific types of code within a program. Using metrics for specific types of code can be a great way to measure code coverage on embedded systems since it allows developers to see what types of code their tests are missing.
The main levels of code coverage are statement, function, branch, loop and condition. Some standards use a simplified version of these levels. For instance, ISO 26262 mainly uses the statement, branch and condition levels. Achieving high code coverage in more complex levels of code will naturally also cover preceding levels. For instance, in the levels for ISO 26262, high code coverage at the condition level would confirm high branch and statement coverage as well. This is helpful to keep in mind since it can streamline the testing process.
If there is a certain layer of code that developers need to have high code coverage for, they can prioritize testing through that layer and everything above it. Tracking code coverage for different types of code will also reveal what areas of the code are not getting tested, allowing developers to adjust their test methods accordingly.
Improve the quality of test cases
Finally, it is important to consider the quality of test cases themselves. Code coverage is meaningless if the tests running on a program are not effective. Developers need to pair high-quality tests with a high code coverage rating.
In fact, Google’s best practices emphasize the importance of analyzing test case quality. Specifically, guidelines from Google experts point out that it is possible to have high code coverage alongside poor test coverage. This means a developer could be misled by high code coverage when in reality their test is not covering all possible cases.
Code coverage itself cannot tell a developer if their program is running well, only how much of it is run in tests. So, increasing code coverage that actually reflects the thoroughness of the testing process, requires ensuring that high-quality tests are being used.
Taking the time to measure code coverage on embedded systems and get it as near to 100% as possible will result in safer, higher quality systems. Many embedded devices play a vital role in end products like cars, planes and medical equipment, where a missed error could be life-threatening. Striving for high code coverage is critical for ensuring that embedded systems meet the quality, performance and safety standards that OEMs and end users need.
About the author
Emily Newton is a technical writer and the editor-in-chief of Revolutionized. She enjoys researching and writing about how technology is changing the industrial sector.
Advertisement
