A novel whiteboard system
The goal of this project was to develop and evaluate Crowdboard, which is a novel whiteboard system that enables many potential stakeholders to provide real-time input during early stage design activities such as concept mapping. Local design teams develop ideas on a standard whiteboard and are assisted by online participants who can add annotations and comments.
Salvatore Andolina, Daniel Lee, Peeyush Goyal
I was the user experience research lead for this project. Responsibilities included designing experimental studies; recruiting participants and conducting lab studies; analyzing research findings and communicate these findings to the developer through wireframes and mockups.
How Crowdboard works
A WhiteboardCapture module written in Matlab uses the webcam to capture and upload whiteboard screenshots to the web server. Ustream is used to send audio to the web server. The web server is written in Node.js and is responsible for storing the status of the system and keeping the clients synchronized. Socket.io is used to send information back and forth between the server and clients – sending whiteboard screenshots and audio from the server to the online participants and sending annotations and comments from online participants to the server.
Annotations and comments added by the online participants are then projected on the whiteboard in the physical studio and assist the local design team in improving the quality, quantity, variety and novelty of ideas generated.
Crowdboard in action
When I joined the project, the core technology had been developed and my role was two-fold:
1. To design and run studies to suggest usability improvements
2. To design and run studies to assess the usefulness of Crowdboard
Recruited local design team using CBDR (Center for Behavioral and Decision Research)
Before I joined the project, the local design team was recruited using university community sourcing. However, since the posters and flyers were limited to the university, there was not sufficient participation from professional designers.
When I joined, the recruitment strategy was modified and I used CBDR to encourage participation from professional designers as well as design students.
Recruited online participants using Amazon Mechanical Turk
Before I joined the study, synchronous crowd recruitment was followed to recruit online participants from Amazon Mechanical Turk wherein participants were pre-notified a few days in advance and requested to show up at the time of the study. The strategy was not very effective.
When I joined, the strategy was modified to real-time crowd recruitment wherein online participants were recruited 20 minutes before the study and the strategy proved highly successful, with a significant increase in the number of online participants that showed up.
Designing and running studies
Objective - To improve the usability and usefulness of crowdboard.
We recruited a total of 6 local design team participants and 83 online participants. The Wizard of Oz method was used to operate the system during the studies.
Designed within subject studies to assess the usefulness of Crowdboard
The local design team was exposed to two conditions – a traditional brainstorm session and a brainstorm session using crowdboard.
Each study was followed up with interviews and questionnaires where the local design team participants commented on the quality, quantity, novelty and variety of ideas generated under the two conditions.
Used different methods to propose usability improvements for Crowdboard
|Fly on the wall observation||To understand usability issues||Participants spent time in filtering comments, found it difficult to understand the meaning of bullets, incorporate crowd comments late in the session|
|Questionnaires||To gain insights into the participants’ experience with the system||The questionnaire served as a prompt for interviews|
|Interviews||Used notes from observations and questionnaire responses to gain insights into the participants’ understanding of the system||Learned how the system can be made more easy to use. Data was organized on post its and analyzed.|
|Video observations||Recorded videos of participants as they interacted with the system||Revisited videos and recorded key statements on post its for analysis.|
|Affinity diagramming||Organized the post its to identify patterns of problem.||Provide a clear mapping between on-screen bullets and thread names, color code bullets to display their status and to alert the team of new comments, visually represent comments using smaller bullets, display top comments to eliminate filtering.|
Based on the affinity diagram, I developed wireframes to convey the design intent to the developer
We are still in the development phase and running pilot studies to improve the system.
Once the system is fully developed and the study has been clearly designed, we will gather more data to perform a quantitative analysis and assess the value crowdboard adds in improving the quality, quality, novelty and variety of ideas generated.