An error occurs when software cannot complete a requested action

as a result of some problem with its input, configuration, or environment.

A high-quality error report allows a user to understand and

correct the problem. Unfortunately, the quality of error reports has

been decreasing as software becomes more complex and layered.

End-users take the cryptic error messages given to them by programs

and struggle to fix their problems using search engines and

support websites. Developers cannot improve their error messages

when they receive an ambiguous or otherwise insufficient error indicator

from a black-box software component.

We introduce Clarify, a system that improves error reporting by

classifying application behavior. Clarify uses minimally invasive

monitoring to generate a behavior profile, which is a summary

of the program’s execution history. A machine learning classifier

uses the behavior profile to classify the application’s behavior,

thereby enabling a more precise error report than the output of the

application itself.

We evaluate a prototype Clarify system on ambiguous error

messages generated by large, modern applications like gcc, La-

TeX, and the Linux kernel. For a performance cost of less than 1%

on user applications and 4.7% on the Linux kernel, the prototype

correctly disambiguates at least 85% of application behaviors that

result in ambiguous error reports. This accuracy does not degrade

significantly with more behaviors: a Clarify classifier for 81 La-

TeX error messages is at most 2.5% less accurate than a classifier

for 27 LaTeX error messages. Finally, we show that without any human

effort to build a classifier, Clarify can provide nearest-neighbor

software support, where users who experience a problem are told

about 5 other users who might have had the same problem. On average

2.3 of the 5 users that Clarify identifies have experienced the

same problem.

In  PLDI

Details