Using machine learning to enabling people to discover great works of art by doodling.
Every great work of art started as a sketch, but what if sketches could help you discover great works of art? That’s the idea behind Draw to Art, a Google Arts & Culture installation in the form of an easel. It uses machine learning to match doodles to paintings, sculptures, and drawings from museums worldwide.
We used the Sketchy database to create Draw to Art to train a neural network, teaching it to match doodles to the artwork. By training a neural network in both sketches and images, the project enables users to use the sketch as an input to large archive objects, paintings, and textiles.
Playful Search Experience
The objective of Draw to Art was to have an open-ended input and a low barrier to entry. We wanted the experience to be rewarding regardless of skill by the doodler so there was never a “no-match” as a result of a doodle.
Part of the play experience was to visualize the machine thinking as it cycles between possible matches. Lastly we emphasized the idea of ‘breadcrumb’ exploration, letting one artwork lead to the discovery of another.
Sketch to image retrieval has been explored in arts and academic projects. We were inspired by a paper published by Caltch entitled “Learning how to retrieve badly drawn bunnies” that tackles the problem by developing a model that embeds both drawings and sketches in the same feature space. Allowing images to be retrieved by sketches and the reverse.
The visual search worked by having both sketches and images embedded in the same feature space through a learning algorithm trained on drawings and pictures. The algorithm we developed searched images based on their visual and semantic relationship.
The installation consisted of a series of interactive easels that enabled users to draw on a touch screen and locate the closest artwork in Google Arts and Culture that matched their sketch.
I developed the project’s prototype using the Google Quickdraw API and GoogleArts & Culture Collection and worked on interaction design, technical development of the machine learning platform, user interface, and installation design. Draw to Art was developed a collaboration with Google engineers and external vendors.