Heuristic Evaluation of Static Analysis Tools in IDEs
Thanks for your help! You've been asked to participate in an evaluation on how developers work with two versions of Quick Fix. Quick Fix is a tool that runs within the Eclipse Integrated Development Environment (IDE), a text editor with features that are designed to help professional developers write code. We call the two versions of the tool
Quick Fix Version 1 and Quick Fix Version 2.
You can access the
evaluation via this Qualtrics form.
The link will open in a new
window, allowing you to view the videos while
performing the evaluation.
Brief Tool Descriptions
Both tools are designed to provide suggested fixes for potential problems in source code. Developers typically want to understand what fixes are available, evaluate the different types of fixes, and identify the changes that the fix will make to their code. In addition, developers sometimes perform secondary edits to the code immediately before, during, or after the fix, such as changing the name of, or renaming, a variable.
We ask you to evaluate tools through Donald, a hypothetical developer who exists in two parallel universes. In the first universe, Donald only has access to Quick Fix Version 1. In the second universe, Donald only has access to Quick Fix Version 2. In each universe, Donald performs two tasks. Thus, you will watch four videos in total.
Task 1: Catch Exception Defect
In this task, Donald wants to resolve a catch exception defect. To do so, he reads the description of the defect, then selects each of the different suggested fixes, comparing each against the original code. However, Donald is not satisfied with any of the default fixes provided by the tool and needs perform some follow-up actions to move the suggested exceptions to better locations in the code.
Watch the two videos below of how Donald performs this task in Quick Fix Version 1 and Quick Fix Version 2.
Quick Fix Version 1 (Video 1 of 4)
Quick Fix Version 2 (Video 2 of 4)
Task 2: Open Stream Exception Defect
In this task, Donald wants to resolve an open stream exception defect. To do so, he reads the description of the defect, then selects each of the different suggested fixes, comparing each against the original code. While evaluating the different suggestions, Donald realizes that there are two additional issues with the code: 1) the variable oos isn't very descriptive and he wants to rename it to toDisk, and 2) the code needs additional logic that avoids writing tmp contents to disk.
Watch the two videos below of how Donald performs this task in Quick Fix Version 1 and Quick Fix Version 2.
Quick Fix Version 1 (Video 3 of 4)
Quick Fix Version 2 (Video 4 of 4)
Direct Video Download
If you have difficulty watching the videos within the browser, you may download them directly here (right-click, save):