Workshop: Cognitive Labs on Tablets
Detailed Procedural Overview
Initial Demonstration
Duration: 1 hour
Overview
The existing materials demonstrated to workshop attendees are drawn from a cognitive lab session conducted with middle school students concerning the interpretation and modification of graphs and diagrams as part of a life sciences curriculum. Attendees can examine all materials, both paper and digital, and a run-through of the actual protocol is conducted with volunteer attendees.
Procedure
- Introductions (10 min)
- Who we are and why we’re here…
- Present Materials (15 min)
- Some context for the cognitive lab sessions and a review of the materials…
- Example cognitive lab materials (AECT2013-CogLabExample)
- Example materials from completed session (AECT2013-CogLabExample-Completed)
- Cognitive Lab Protocol Review (20 min)
- A quick walkthrough of the cognitive lab session protocol/procedure…
- Example cognitive lab protocol (CogLabProtocol-CBTEco6)
- Questions and Answers (15 min)
Design and Develop
Duration: 2 hours
Overview
A brief overview of the design and development workflow is provided for attendees using the previously demonstrated materials. Attendees will then develop their own materials (using the “Illustrator > Preview” portion of the workflow) to prepare a 15-minute cognitive lab session containing 2-3 tasks in a protocol designed to address their own research goals.
Procedure
- Workflow Overview (20 min)
- Slide presentation: Cognitive lab materials development workflow (Google Slides) (Workflow Overview)
- Workflow job aid (Workflow Job Aid)
- Design and Development (90 min)
- Workshop participants work individually and collaboratively to generate cognitive lab materials
**Please note: The workshop facilitator can and will answer questions concerning content and process of cognitive lab materials generation, but does not provide instruction for software or hardware issues.**
Implement
Duration: 1 hour
Overview
Each attendee transfers finished cognitive lab materials from laptop to iPad (via Dropbox) for paired cognitive lab sessions with another attendee. Each attendee has the opportunity to experience digital tablet cognitive lab protocols as a researcher and a participant. Following these paired sessions, attendees prepare and transfer completed session files from the iPad back to Dropbox. This portion of the workshop concludes with a debriefing session regarding options for data management as well as data analysis, summarization, and reporting of results (with examples provided by the facilitator).
Procedure
- Laptop to iPad (5 min)
- At least two clean copies.
- Cognitive Labs, Round 1 (20 min)
- Workshop participants pair up to experience each other’s cognitive lab sessions.
- Cognitive lab participant views iPad. Cognitive lab scientist reads protocol from laptop screen.
- Cognitive Labs, Round 2 (20 min)
- Workshop participants stay in same pairs, switching roles.
- Debrief (15 min)
- Taking notes from cognitive lab data (Notes from Cognitive Lab Sessions)
- Writing a summary report of cognitive lab sessions (Summary of Cognitive Lab Sessions)
Small Group Discussions
Duration: 1 hour
Overview
Attendees break into groups of four (pairs of the previous pairs) to critique the materials of each of the members of the opposite pair using a provided rubric. Attendees are encouraged to modify this rubric as part of the critique process, especially when conducting a follow-up critique amongst all four participants in the group (during the second half of the hour). The facilitator will roam amongst the small groups to foster discussion and answer specific questions.
Procedure
- Grouping and Rubric Introduction (10 min)
- Qualitative evaluation rubric (Cognitive Lab Quality Rubric)
- Quick questions
- Initial Critique (30 min)
- Pair 1 critiques Pair 2
- two critiques (eg., 1a>2a, 1b>2b)
- 1a>2b and 1b>2a also possible
- Pair 2 critiques Pair 1
- same pairs, opposite critique (eg., 2a>1a, 2b>1b)
- Pair 1 critiques Pair 2
- Follow-up Critique (20 min)
- Compare Notes
- Discuss Rubric
Full Group Discussion
Duration: 1 hour
Overview
Attendees reconvene to reflect on issues raised during the critiques conducted as small groups. The discussion focuses upon affordances, limitations, and resulting design nuances associated with implementation of cognitive labs on digital tablets using this particular set of tools. Identified affordances might include, for example, the efficiency of page ordering and cloud-based delivery, whereas limitations might include options for recording student performance. Identified design nuances might then include considerations for matching verbal cues in the protocol with students’ verbal descriptions of constructed visual responses on the session materials (especially concerning ease of analysis). Ample time is provided for specific questions from attendees.
Procedure
- Full group discussion (45 mins)
- Wrap-up and farewell (15 mins)