How to reproduce unreproducible defects in embedded systems
Unreproducible defects are one of the most frustrating phenomena in the area of Software and Firmware validation.
This problem is painful in Embedded systems where products have very limited capabilities for debug and tracing. It is even more painful in embedded Security systems, where debug is yet more limited due to security concerns.
So what is the magic?
Instruction-Based profiling (=IBP) is a well-defined hardware feature that some CPUs support. It allows logging detailed information about software execution with minimal impact to the system execution.
IBP is a very powerful tool that can give insights on how the software really behaves on the actual target platform.
An advanced IBP mechanism can support also full Data Trace, that can log not just the executed instructions, but also the data attached to them.
The combination of IBP with Data Trace can be used for full “Record & Replay” flow on instruction-accurate simulator, thus allow to fully reproducing the “unreproducible” defects.
In addition, this feature has significant advantages for Security system, as full track of all function in/out parameters and return values. It can also help catching not allowed memory accesses.
For safety systems, IBP allows significant improvements for code coverage. It can gives not just simple code coverage, but also flow coverage, showing exactly how many times each function was called and from where. All this with minimal impact on the flow execution.
Udy Hershkovitz has been working at Intel for 17 years, as a senior Firmware developer, an engineering manager and now as an architect in the field of embedded security systems.
In the last few years, Udy invested a significant effort in investigating mechanisms for automated profiling systems, to create an environment for continuous monitoring embedded security products for performance, size, and resource consumption optimizations.
As part of this effort, Udy designed infrastructures that allow DevOps and validation teams to build automated and semi-automated tools to log and detect security and performance defects.
This effort led into several issued patents and also to this presentation.