============================================================= Heuristics ============================================================= ------------------------------------------------------------- 1 Visibility ------------------------------------------------------------- 1.1 Thorough explanation 1.1.1 E1: QF1 does have the potential advantage of providing fairly verbose explanations and examples of each suggested fix in-line as the developer chooses the fix. 1.2 Ability to focus 1.2.1 E6: Highlighting is helpful in providing focus. 1.2.2 E6: V1 = Hard to follow which info is being edited in one case where multiple rows were highlighted at the same time. 1.3 Toggle options 1.3.1 E3: The visibility of the tool, as seen in the radio button box in the upper right, was really good. You could see easily which option was chosen, and then the changes were IN your code, instead of squished into a box with different formatting than the code itself (V2). 1.3.2 E1: Still, the ability to toggle between original and last change should still be useful. 1.4 Fix preview/diff 1.4.1 E5: For V1, I can see a summary of what the fix will be as well as a preview window containing the changes. This is good enough for me to get an idea and see whether this is indeed the fix I want to apply or not. 1.4.2 E3: That took away the main advantage I saw for that tool - allowing a true dif comparison between the code as it is now, and what it will look like when one of the options from the tool is chosen (V1). 1.4.3 E3: The con was that the code is formatted so differently in the preview box than it would be in the normal code window (V1). 1.4.4 E2: I notice that in Version 2 it takes an extra action to see what the effect of something like "Use try-with-resources" will be, while in Version 1 the code changes are shown along with some summary name (I think). 1.4.5 E1: It's then left to the developer to toggle back-and-forth to see these (note "the Donald's" undo-redo to compare code), and to properly identify the critical changes when they need to be modified from the initial suggestion (again, "the Donald's" explicit cut-paste to move exceptions as thrown or multiply caught). 1.4.6 E4: Version 2 seems to elaborate more on a suggested approach to solve the pE2lem. 1.4.7 E1: QF1 inserts the selected edit in-place with no additional visuals (e.g. highlighting) to indicate critical changes from the original. 1.5 Feedback 1.5.1 E6: Drag and drop significantly more helpful in persisting context of changes and providing sufficient feedback during actions. 1.5.2 E3: (V2) As far as status goes, I have no complaints because it was obvious when the tool was in use and what state it was in. 1.5.3 E2: It's not a clear win for Version 1, though, because I think Donald needs to choose the right location in the code for the suggestion to be made. 1.5.4 E6: (V1) Changes occur a little too fast, making it a bit difficult to find the context of what was removed. Appears as if cursor jumps to the top row after change, which causes a loss of context. 1.5.5 E2: With respect to visibility, I think the two versions are about the same, assuming an experienced programmer with knowledge of the source code. That is, they both depend on Donald understanding the effects of his actions; in Version 1 mainly the standard responses of the programming environment, in Version 2 more of the changes suggested by the tool. 1.5.6 E6: V2 = Not completely sure what happened after selecting '[Quickfix ver 2] Fix the Catch...' Presuming the (green and blue) highlighted areas are the same three catch exceptions. 1.6 Intelligence 1.6.1 E1: The drag-and-drop also appears to have some smarts behind it (e.g. when the exception is dragged up, a properly syntax'd multi-exception is generated). 1.6.2 E1: An obvious issue here is that QF2 actually identifies and highlights what's really important in the change. This particular example pE2ably represents base-case, I can imagine other situations where highlighted text isn't what a developer wants to change, so you'd be back to the cut-and-paste approach. 1.7 Fix Justification 1.7.1 E1: QF2 seems to put this down in the navigation tab, (I assume) expecting the user to "understand" the change and its rationale when it's applied, or forcing them to scroll through the bug explanation to see the justification for each suggested change. 1.7.2 E5: For V2, the tool just says "Fix the bug". I have no idea what this means, it provides no preview. I would never click something like that. 1.8 Code Obfuscation 1.8.1 E3: The visibility of the tool (V1) was fine, but it occluded the code it was suggesting to replace. 1.9 Cognitive Load 1.9.1 E4: Version 1 did provide information on system status. But the information was very long. Given the small window, it was difficult to follow and place significant loads on memory. 1.9.2 E3: (V2) But I would say that this version of the tool requires more working memory from the user because you can view the original and view the suggested changes serially, but you can never view them side by side. 1.10 E1: QF2 simplifies this by: (1) highlighting what it thinks are the important changes to the code; (2) offering an explicit toggle to go back and forth between original, suggestion, and last change by user; and (3) offering drag-and-drop on the highlighted changes. ------------------------------------------------------------- 2 Match between system and the real world ------------------------------------------------------------- 2.1 Language 2.1.1 E4: The first video of version 1 was difficult to follow as I the system was speaking very programming-knowledge-dependent language. 2.1.2 E4: As a person with minimal programming training (although a little knowledge with C++, I can kind of make general sense of the symbols/variables and what the programmer in the video was doing), version 2 is much easier to understand. 2.1.3 E1: I didn't see a lot of different here. Same text is use (mostly), just in different situations or contexts. I'm looking specifically at what's in the Navigation tab and what the system pops up when fix choices are presented. All of it appears like something a competent developer would understand. 2.1.4 E5: V1 uses code-oriented terminology, such as multi-catch, etc. 2.1.5 E6: V1 = presuming the language used is familiar to the intended audience V2 = same as above 2.1.6 E3: This was a little harder to evaluate because I don't necessarily speak the language of the tool. But I'll do my best to note what I saw. 2.2 Cognitive load 2.2.1 E4: The load on working memory was also very high in version 1. 2.3 Change description 2.3.1 E3: But facing the same pE2lem in video two, it looked like the user only got radio buttons for original and "default suggestion." That didn't seem very descriptive of what the change would be. 2.3.2 E5: "Fix the bug" gives absolutely no information. ------------------------------------------------------------- 3 User control and freedom ------------------------------------------------------------- 3.1 Option Toggle 3.1.1 E6: V2 = The radio button options in the upper-right corner dialog of more useful than in v1. 3.1.2 E2: Version 2 seems to be a win here, giving Donald improved control, in particular the ability to easily switch between the effects of different suggestions. 3.1.3 E1: I give QF2 a slight edge here, because it has an explicit dialog that allows for original, default change, last change. 3.1.4 E4: Version 1 system did provide such functions through the menu on the topic left corner. 3.2 Undo/Redo 3.2.1 E1: QF1 resorts to the IDE's undo-redo, which works fine, but requires the developer to remember where they're at their edit chain. 3.2.2 E2: Maybe this was a personal habit for Donald, or done for the purpose of the video, but the repeated "Undo typing" in one of the videos of Version 1 suggested that Donald had to do a little more work than might be ideal to back out of a decision, aside from reverting a file. 3.2.3 E3: Both tools supported undo and redo, but Version 2 supported it more explicitly. 3.2.4 E4: Version 2 makes it a lot easier for making changes, redo, undo, and comparing both. Given it is in general easier to pay attention to two locations that are horizontally rather than vertically arranged, the spatial layout of main content and control is better in version 2. 3.2.5 E3: In version 1, it seemed up to the user to use the normal ctrl-z kind of undo, which is inherent in the system not the tool. But, it works great and it's something we're used to. 3.2.6 E5: For me, it is actually very easy with V1 as well, since I would just Ctrl+Z to undo my changes if I did not like them. No need to go through the context menu. Maybe biased? 3.2.7 E3: But I'm not sure how doing that affects undo/redo. It seems like the options offered by the tool will always be the same, so if I chose an option but then modify it by dragging and dropping... I assume I"ll lose those changes if I click "original" and then re-click the suggestion I modified. That might be tough, but I see why you'd want that in case the dragging and dropping was a mistake. But it makes it harder to undo/redo the exact state you were in. 3.3 Too Fast Feedback 3.3.1 E4: However, every time there was a change, the entire content on the main screen feel almost completely changed, due to unhighlighted multiple changes. Such Rapid shifts of information (with drop down menu covering main content area and requiring visual search) in the visual field can impair the memory processing of previous entered information. Given the high memory load in version 1, the task was rather difficult. 3.3.2 E6: However, not optimal given the specific usage pattern in the tool. The undo typing incrementally removes terms in a quick manner. The function moves a bit too incrementally and too quickly. Confusing that it performs undo by term in a string and then goes back to a full blue screen. 3.4 Accepting the Change 3.4.1 E3: In version 2, the radio button state shows exactly how to change the code back to the original or one of the changes. What I couldn't really see in the videos was how the user accepted the change suggested by version 2. I could see the code change with it, but not what the user did to say "yes, I want to accept that change and end my interaction with the tool." 3.5 Labels 3.5.1 E6: Reconsider placement of function or function label to make more intuitive and prominent. (V2) 3.6 Interaction Affordances 3.6.1 E6: V1 = Follows commonly known convention and presumably utilizes short-cut key functionality. 3.6.2 E3: I did like the drag and drop options created by version 2, so the user could keep his or her hand on the mouse to fix the bug (in some cases). ------------------------------------------------------------- 4 Consistency and standards ------------------------------------------------------------- 4.1 Unclear/Inconsistent text 4.1.1 E5: If you ask me, again V2 is "N/A". It provides me the same text and options no matter what the pE2lem/bug is. Yes, it is consistency, but I see its hurting instead of helping. 4.1.2 E3: Again, the only item I would call out here is that version 2 seemed to offer different, poorly named suggestions for fixes in the first pE2lem compared to version 1. 4.2 Unclear functionality 4.2.1 E6: Not clear on what makes the 'Resume' function appear 4.3 Labels 4.3.1 E6: Consider more salient labels. 4.4 Feedback 4.4.1 E6: V2 = Is 'Original' revert back to original text (i.e., undo and restore)? Drag and drop significantly more helpful in persisting context of changes and providing sufficient feedback during actions. 4.4.2 E1: These seems well integrated into the IDE, and both offer their own ways to indicate their actions and results. ------------------------------------------------------------- 5 Error prevention ------------------------------------------------------------- 5.1 E2: Refactoring by changing a variable name that propagates to other occurrences, autocompletion, and other syntax-sensitive editing features are good for error prevention. 5.2 Error Diagnosis 5.2.1 E3: Version 2 puts the preview right into the code. Although I kind of love this, because it really lets you see what's going to happen, I'm still not sure what it's like to deal with customizing the code before committing. However, the highlighting it does shows the user where the changes are, so while not preventing an error, it would help with diagnosis. 5.3 Fix Confirmation 5.3.1 E3: I didn't see any conformation options about applying suggestions from either version of the tool. 5.3.2 E4: Version 2 seems to allow the programmer to view intended change and then confirm that no undo or further change is needed. 5.3.3 E6: I don't recall a commit option for either version. 5.4 Obviousness of Error Prevention 5.4.1 E1: I didn't see any examples of eliminating error-prone conditions directly, that is, without the user's participation (unless you mean applying suggestions, and I wouldn't consider that eliminating error-prone conditions directly). 5.4.2 E3: It looks like in Version 1, perhaps they click on the tool code preview window and then that code is put into their "real" code window. Then, if they don't like it, they ctrl-z and bring up the tool window again. It does look like once they apply it to their code it's just "there." Not marked in any special way. ------------------------------------------------------------- 6 Recognition rather than recall ------------------------------------------------------------- 6.1 E3: I think this version (V1) could be designed to better support recognition since the elements are all there for it, just a few design choices get in the way. 6.2 Ease of comparison 6.2.1 E1: QF2 wins out here, since it highlights, plus it has a dialog to allow a user to easily toggle between states. 6.2.2 E3: I enjoyed looking at this version more, but of course I wasn't trying to use it myself. I really like the highlighting because as you go back and forth between suggestions, it calls attention to what is different. (V2) 6.2.3 E2: Version 2 makes objects out of some tokens so that they can be moved around rather than being cut and pasted; it shows changes to the in place and allows Donald to switch back and forth for comprehension and comparison rapidly and easily. 6.2.4 E3: What I would imagine someone would do with Version 2, to support their recognition over recall, is to switch quickly back and forth between original and a suggestion. This is hard to do with tiny clickable radio buttons. Perhaps it could be a list that has focus, like Version 1, and you could use either a click OR the arrow keys to switch between suggestions (and have them highlighted in the tool box accordingly). 6.2.5 E4: Version 1 requires more steps (e.g., copy and paste). Version 2 provides suggested changes, immediate contrast among various code change options, and highlight the most critical information 6.2.6 E3: Officially, I think version 1 does better than version 2 with recognition over recall. This is because version 1 offers a preview WITH the original code still in place, so the user can see both at the same time. Version 2 shows one suggestion or the original at a time, so the user can look at the suggestion, but needs to recall the original since it's no longer visible. 6.3 Formatting 6.3.1 E3: In Version 1, the code suggestions are formatted very differently from the real code. This would be hard to read and compare. It's all squished in the little box. What if there were changes suggested in multiple places? Would the little window have to be scrolled? Yikes. 6.3.2 E3: In Version 2, it's just officially recall required. But it wouldn't have to be. Is there some way to show the original at the same time as the nicely formatted and readable changes? 6.4 View Obfuscation 6.4.1 E3: Also, the box seemed to overlay on the code, so you couldn't really look at both at the same time and compare. That means there is recall after all. 6.5 Menus & Navigation 6.5.1 E2: Donald navigates extensively through the source code (at least relative to the size of the tasks he's completing) in order to identify places that changes might be made, to see what changes the systems suggests, and to compare these (in his mind, as far as I can see). 6.5.2 E6: V1 = alleviate the need to use File menu. 6.5.3 E1: QF1 simply applies the requested change, then leaves it to the use to work through the IDE's UI to maintain mental model state. 6.5.4 E6: V2 = helpful action menu in upper right corner 6.5.5 E6: [action menu] could be placed in a more intuitive and prominent location 6.6 Cognitive load 6.6.1 E6: use of color which requires user to learn what the colors mean 6.6.2 E5: For both versions, once you apply the fix, I don't see a way to remember what the fix was. If this information needs to be carrier over, then it should be in both cases (I would be okay if it was not carried over). 6.6.3 E4: The function that allow drag and drop would help to reduce memory load. ------------------------------------------------------------- 7 Flexibility and efficiency of use. ------------------------------------------------------------- 7.1 Cognitive load 7.1.1 E6: V1 = inefficient. higher cognitive load 7.1.2 E6: V2 = use of color, auto-highlight, actions buttons is useful to reducing cognitive load and performing task more efficiently 7.2 Required Expertise 7.2.1 E4: Although I pE2ably missed such function in the videos, it seems to be that version 2 would be a higher capability of doing so based on the expertise level of programmers. 7.2.2 E5: Don't see much difference. Maybe the fact that version 1 provides fixes separately means that an expert can select the ground-truth fix faster? I am not sure. 7.2.3 E1: I guess QF2's drag and drop of highlighted code is an acceleration, although I'd argue it's not unseen by a novice, I'd assume (but would need to check in an experiment) that even a novice user might start messing around with the highlighted code to see what they can do with it. QF1, clearly, provides none of this. 7.3 Inapplicable 7.3.1 E3: Cannot rate - no accelerators were seen (and I don't want to give it "very poor" because I think that implies there could or should be accelerators for this simple tool.) 7.3.2 E2: I'd say that this isn't really relevant. The standard shortcuts are available in the interface, but go unused, I assume to avoid the videos going incomprehensibly fast. This isn't really set up for novice programmers. 7.4 Fix Comparison 7.4.1 E2: Adding accelerators for the suggestions might give a small boost to Version 2, but this would be a very small time savings, I'd guess, in comparison with the programmer's evaluation of the effects of the changes. Maybe for comparison purposes, switching between suggestions. ------------------------------------------------------------- 8 Help users recognize, diagnose, and recover from errors. ------------------------------------------------------------- 8.1 Interaction/Change Detection 8.1.1 E4: Version 2 demonstrates a lot of improvement and the suggested edits are much easier for evaluation, comparison and execute. The user can click on one of the two or three buttons on the right, which would lead to a immediate change of the programming content. This could largely improve programmer's performance to spot changes and errors. In psychological terms, this is called 'change detection', which is largely affected by the interval between two images. Having a menu that will drop down between the two image appearances will reduce performance. 8.1.2 E1: This is pretty much based on how I'd use this tool. Since I consider myself an expert developer, I wouldn't be interested in QF1's slightly more verbose set of suggestions, and I like the direct interaction QF2 provides. 8.2 Error Recovery 8.2.1 E3: It's easier to see how to recover with version 2 - just click original. 8.2.2 E6: V1 = nothing stands out as being highly effective in highlighting and providing suggestions on recovering from errors 8.3 Flagging critical areas 8.3.1 E1: The real win here, though, is actually highlighting what the system thinks are areas of concern. Assuming it doesn't go Microsoft Clippy on me and keeps the suggestions to a minimum and makes them relevant, then I'd pE2ably use this. 8.4 User Friendliness 8.4.1 E4: Version 1 has functions to assist recognition and diagnosis. But information and guidance was not quite presented in a user friendly way. 8.4.2 E3: It looks like it's on the user to know you can ctrl-z with the first version of the tool. So, really not much help is offered. (V1) 8.5 Fix Assessment 8.5.1 E6: V2 = provides the ability to toggle between options and to see effects ------------------------------------------------------------- 9 Help and documentation. ------------------------------------------------------------- 9.1 E3: It was interesting to me that both versions offered one tool tip and the tips were so different. Version 1: Press tab from proposal table or click for focus Version 2: You can select an option or drag and drop exceptions 9.2 Cognitive load 9.2.1 E4: In the case of verison 1, the programmer has to compare the standard to the current codes, leading to lots of demand on mental workload. 9.3 Brevity 9.3.1 E1: QF1 seems more chatty, which makes it slower (maybe) but more clear what it's suggesting (maybe). 9.4 Fix Assessment 9.4.1 E3: Version 2 seems like a more helpful tip, especially the drag and drop which is an unusual feature. 9.5 Defect description 9.5.1 E2: Both versions provide explanations of the defects to be fixed, in a text explanation; much of what both do seems self-explanatory. 9.6 Step-by-step help 9.6.1 E1: QF2 implies steps by simply applying the default and highlighting to indicate what changed and why. 9.6.2 E4: Version 2 provides suggested steps to take (thus is somewhat intelligent and situation based), rather than just abstract information on how a sentence should be wrote.