In this assignment you will do a focused Knowledge Elicitation to uncover details of student thinking in a given task domain. Specifically, the goal is to identify and describe the knowledge, strategies, and skills that students need to solve problems, the misconceptions or shallow knowledge that may interfere with learning, any “hidden skills” that may make problems easier than expected, and the kinds of errors that novices typically make. You are asked to start with a theoretical task analysis in the given task domain, and then to collect protocols of novice problem solving with participants from the target population, analyze the protocols, and compare the results of this analysis to your theoretical model of correct performance.

Ideally (beyond this assignment), you would not rely exclusively on theoretical task analysis to study correct performance, but also do empirical task analysis with experts. However, to keep the assignment manageable, that is not a required part of the assignment. If you have easy access to experts, it may be very instructive to compare them to novices. If you are an expert in the domain yourself, you might run yourself as a subject, asking someone else to take the role of the experimenter.

Based on your analyses, you will create knowledge models in the form of either a flow chart or a concept map. You will also identify error categories and strategies, and discuss implications for the design of an educational game. The task domain could be: (1) geometry problem solving or (2) a task domain of your own choice.

If you choose to focus on the geometry problem solving domain, you should use the set of sample problems and answer key, which cover geometric proofs involving circles and angle measures. I've used this task for this assignment for a few years now and it tends to work well with adult participants who have likely forgotten the topic. The problem set provides a few potential variations in the task that you might explore, such as providing different levels of scaffolding and reference material, however I do recommend you use the entire packet with participants for the most interesting results.

If you prefer to select your own task domain, you may choose a task domain that you are interested in and knowledgeable about. Because of the limited timeline and scope of this assignment it is strongly recommended that you pick a traditionally "academic" task domain in the sense that it deals with a school subject; e.g., pure puzzle-solving tasks such as Sudoku or collaboration skills that do not involve a school-adjacent task would not be a great fit. Select a set of tasks that (as a set) takes 10-20 minutes to solve for a novice in the domain. Please make sure you select a task domain for which you have access to novices. This could be a task domain that you are interested in pursuing for your final course project (but does not have to be). You should receive confirmation on your custom task with me (either by email, slack, discussing in class, or setting up a meeting.) prior to submitting Assignment 1A. This is to make sure the task is of appropriate scope and meets our requirements for being an "academic" task.

Steps

Collect tasks. If you use choose to do the geometry problem solving task, this step is largely done for you however, there are several variations of these tasks you might consider. For example, you might omit the table of sub-goals at the bottom of each problem to get a sense of how participants explore the space. Alternatively, you might withhold the set of theorems. If you are using your own task domain you should collect a concrete set of specific tasks for participants to do.
Do a theoretical task analysis of correct performance on the tasks you selected. Do a theoretical analysis to identify a correct method for carrying out the tasks, such as you might expect to see from a competent novice or expert. Present this method in the form of a flow chart and/or concept map, depending on the nature of the task. You should complete this step before collecting or analyzing any think-aloud protocols.
Collect protocols of at least two novice problem solvers. Recruit participants (at least two, but to be on the safe side, you might recruit and run a couple more), run sessions with them following the Gomoll guidelines, and record the protocols. It is important that you recruit novices, that is, people who will not ace the tasks that you present to them. It is also important that you assign the same set of problems to everyone. For the purposes of this assignment we will use a flexible definition of novice. For example in the geometry task, it may be hard to find true novices easily but you can use people who may not have done geometry in a long time and are rusty. If, unexpectedly, one of your participants executes the task flawlessly, then that protocol will not count as a novice protocol and you will need to run an additional participant.
Analyze the protocols. Carefully study the protocols you collected. You should try to follow in an almost line-by-line manner what the thought process was behind each protocol. For each participant create a table showing for each problem attempted whether they completed it correctly, a brief summary of the strategy they employed, and any other important notes about their problem solving. Then create a flow chart and/or concept map that summarizes each participant's problem solving approach across the problems they attempted.
Synthesize novice approaches. Compare the novice approaches to each other and take note of any similarities or differences. Summarize the main difficulties (2 or more) that the novices seemed to be having. For each, point to at least one protocol and the specific place in the protocol in which it occurs. Based on your analysis of the novice protocols, identify a typical novice approach. Create a flow chart or concept map of this approach. Briefly discuss how the novice strategies you observed differ from your theoretical model created in step.
Discuss implications for the design of an educational game for your task domain. What kind of exercises should the game present to students to make sure that the students get enough practice with the hardest parts of the task? What sort of support might be necessary, and how might you provide it.

Deliverable

To scaffold timing the deliverables for this assignment will be broken into two stages. The first deliverable (Assignment 1A) will involve your initial breakdown description of the task and you theoretical task analysis of it. The second deliverable (Assignment 1B) will include the empirical portion of your task analysis with 2 novices, along with comparison to your original theoretical results.

The general expectation of format for each stage of the assignment will be a written report submitted as a pdf, however several of the elements of the report will be tables or diagrams rather than paragraphs of text.

Assignment 1A

Task Description - Give us a description of the final task you ultimately had participants do. If you used the geometry task, include any variations on the provided packet that you gave them. DO NOT simply say you used the provided packet describe the task you had people do with the packet. (~1 paragraph)

Theoretical CTA Diagram - Create a flow chart or concept map of the theoretical task analysis that thoroughly covers how an expert might solve the task. There should be one chart describing a general strategy rather than one for each problem in the problem set. (1 chart)

Theoretical CTA Synthesis - Provide a written synthesis of the theoretical task analysis that describes the overall problem solving strategy and digests the contents of the flow chart. In the write up, make predictions of difficulties novices will have with the task. (~2-3 paragraphs)

Assignment 1B

Empirical CTA Participant Recruitment - Provide a description of the participant recruitment strategy along with the backgrounds of the individual participants. (~1 paragraph)

Empirical CTA Participant Strategy (x2) - For each participant create a table from your notes on their problem solving. This does not need to be an annotated transcript. You can use the following format: (2 tables, 1 per participant)

Problem	What was their answer?	What was the strategy they used?	What difficulties did they have?	Additional Notes
Problem 1
Problem 2
...

Empirical CTA Individual Diagrams (x2) - For each participant, provide a flow chart or concept map of the strategy used by each participant that thoroughly covers the problem solving task as the participants did it. There should be one chart describing a general strategy for each participant rather than one for each problem in the problem set. (2 charts).

Empirical CTA Unified Diagram - Provide a unified diagram that integrates the strategies from each of the individual CTAs into a general novice strategy. There should be one chart describing a general strategy across the novices rather than one for each problem in the problem set. (1 chart).

Empirical CTA Synthesis - Provide a written synthesis of the unified empirical CTA that describes the overall problem solving strategy novices use, digests the contents of the flow chart, and provides a summary of the common difficulties they faced. (~2-3 paragraphs).

Reflection and Discussion - Provide a discussion of the differences between the predictions made in theoretical task analysis and the results of the empirical task analysis is provided. Additionally, discuss the implications for designing a game for this domain. (~3-5 paragraphs).

Questions

If you have any questions about the requirements of expectations of this assignment you can ask them here.