Faceted Prototyping

Animistic Collaborators in Mixed Reality
Project Lead: Philip van Allen
2017 / 6-week project

CATEGORIES: UX, interaction, speculative
KEYWORDS: augmented reality (AR), mixed reality (MR), creative collaboration, virtual assistants

Using the Microsoft Hololens and Unity, I designed a Mixed Reality demo of what prototyping alongside AI-based collaborators might look like:

How might collaborating with AI-based, animistic entities in mixed reality amplify the creative process?

↳ Screenshot of Animistic Collaborators in Mixed Reality project page.

PROJECT OVERVIEW

Faceted Prototyping is one of six demos that comprise Animistic Collaborators in Mixed Reality, a research project led by Philip van Allen. These demos were probes into how virtual, AI things could work with people in augmented reality, especially in creative contexts. The creative context for Faceted Prototyping is product design.

PROJECT OUTCOMES

Faceted Prototyping proposes three AI-based collaborators that work with the product designer to amplify the “exploratory sketching” process. In product design, exploratory sketching is a form-finding and idea-molding exercise. It’s a way for the designer to move between and across ideas quickly, resulting in more divergent and unexpected outcomes than if the designer had started their process with CAD (computer-aided design).

I prototyped how these three AI-based collaborators—each with their own unique personalities—would interact with a product designer in their studio. In a fully working prototype, the collaborators would be trained on existing formal, structural, and cultural, paradigms in product design (in this scenario, the subset of furniture design), as well as on the designer's individual sketches.

By inferring 3D models from the designer’s 2D sketches, hallucinating new divergent models, and making sense of the multitude of possible forms, the collaborators convert the practice of sketching on paper into a responsive and multi-faceted prototyping activity.

SCENARIO CONTEXT

the ACTIVITY FIELD

The ACTIVITY FIELD is a studio space for Mixed Reality. Although it is only visible when the designer is wearing their headset, the Activity Field demands an open physical studio space where the designer can move around in concert with the three virtual collaborators, who inhabit this space.

THE USER

the PRODUCT DESIGNER

In this scenario, the DESIGNER is a product designer who specializes in furniture design. The designer wants to channel the freedom of exploratory hand-sketching, directly into the ability to see their ideas change in the room, at scale. The designer is open to new ideas and looks to the three collaborators as a way to explore forms that they couldn’t have on their own.

Collaborator 01

the REFRACTOR

Based on the designer’s initial sketches, the REFRACTOR generates a 3D model of the new form. The refractor continuously responds to the designer’s sketches and updates the 3D form accordingly—virtually bringing the sketches from paper to life.

Collaborator 02

the DIVERGER

Based on the refractor’s interpretation of the designer’s sketches, the DIVERGER “hallucinates” many new forms for the designer to consider.

Collaborator 03

the CONVERGER

Taking into account the output of the diverger and the designer’s responses, the CONVERGER finds common threads and refocuses many forms and sketches back into one.

USER JOURNEY

the DESIGNER, the REFRACTOR, the DIVERGER, and the CONVERGER work together in Mixed Reality.

Part ideation and part exploratory sketching, Faceted Prototyping allows the product designer to work through ideas quickly by hand-sketching, but also projects those ideas into a high-fidelity 3D form. The collaborators reveal the “facets” of the designer’s creative process—showing them alternatives that they might not arrive at on their own. The designer can see their ideas come to life, at scale, in a real world context.

↳ Excerpts from Phil van Allen’s concept of “faceting” information.


↳ Synthesis of my visual research into “faceting” as a concept. From optical lenses that scatter or focus light, to the separation of a mixture chromotography, to the morphology of “faceted polyhedra”.

Phase 01: Conceptual + Visual Research

Animistic Collaborators in Mixed Reality builds on Phil van Allen’s prior work in animistic design—where multiple, non-anthropomorphic actors have distinct personalities and expertise. The research team began by familiarizing themselves with Phil‘s existing projects and then brainstorming ways in which we could imagine virtual “collaborators” interacting with the physical world in a creative way.

I decided to dig deeper into Phil’s existing work and looked at his design experiments on a concept he calls “facets”. Facets create a way for the n-dimensions of a body of information to expand and be accessible in a physical and interactive way. They make information more spatial.

Inspired by Phil’s concept of “faceting” information I wanted to explore what the physical properties of faceting could be and how those might be augmented in Mixed Reality. If a body of information has facets, then how might we reveal and utilize the facets of a creative process (i.e. a body of ideas)?

↳ Initial storyboard for my demo concept, sketches of the “personalities” of my virtual collaborators, and note on the “activity field”.

Phase 02: Storyboarding + Interviewing

I picked the creative process of product/industrial design because it can be so physical in nature, and I wanted my Mixed Reality scenario to push on the intersections of the physical and the virtual.

After interviewing Jenny Rodenhouse (who has a background in Industrial Design and teaches Product Design at ArtCenter), I used this resource as a way to structure how Faceted Prototyping would fit into the typical Product Design workflow.

↳ Captures from Holo Toolkit and 3D Draw Hololens apps.

Phase 03: Technical Research + Final Storyboarding

Knowing that physical prototyping in Mixed Reality is something that the Microsoft Hololens has already made possible, I tested the existing apps and tools to see how the Faceted Prototyping user experience could be both different and useful.

↳ Final narrative storyboards + annotations made in preparation for deploying and editing the concept demo.

↳ Excerpt from Phil van Allen’s presentation at the 2017 Sketching in Hardware conference.

NEXT STEPS

Animistic Collaborators in Mixed Reality was presented by Phil van Allen at the Sketching in Hardware conference in 2017. You can see the presentation deck, here. The project and research has informed Phil’s ongoing work to create a toolkit for sketching in AI.