Multicraft + Tangicraft

Multicraft + Tangicraft works on multimodal interfaces to empower individuals with physical impairments to play and create in the popular video game Minecraft. Multicraft utilizes Gaze and Speech to replace the traditional mouse and keyboard interface. By looking at a location on the screen and uttering a command, users can build in the Minecraft world. Tangicraft utilizes tangible blocks with webcam detectable codes. Users can construct a physical structure in front of a webcam and call it in the Minecraft world by pressing a button on an Arduino microcontroller. Our team ran user tests to drive iteration on our designs.

A person playing Minecraft using gaze and speech controls instead of mouse and keyboard

Example set up. Wooden workbench with wood blocks sitting to the right of a laptop running minecraft software

Videos

Eye Tracking with Multicraft

Text Commands with MultiCraft

Speech Commands with Multicraft

Obstacle Course Challenge

Publications from This Project

Worsley M., Mendoza Tudares K., Mwiti T., Zhen M., Jiang M. (2021) Multicraft: A Multimodal Interface for Supporting and Studying Learning in Minecraft. In: Fang X. (eds) HCI in Games: Serious and Immersive Games. HCII 2021. Lecture Notes in Computer Science, vol 12790. Springer, Cham.

PDF DOI