The State of AI in the creative industry: from Watson to WhatTheFont.
Right now, artificial intelligence is telling you what to listen to on Spotify, deciding which artwork you’ll prefer on Netflix, and suggesting things to buy on Amazon. Companies have been using AI to quietly optimize their business for years, and its presence seems only set to grow—with its vast potential touted as the future of the automotive industry, a replacement for manual labor, and even the answer to loneliness.
Within the field of AI, machine learning—which involves feeding a program vast amounts of data to learn from, rather than relying on explicit instructions issued via code—has become something of a buzzword. Its ability to draw upon vast reserves of information, far more than a human counterpart ever could, means it’s capable of reducing or removing human error and revolutionizing the world of work.
AI and machine learning serves a valuable practical purpose for many industries, but it has a more uncertain relationship with the creative world. Creativity and ingenuity are often seen as uniquely human qualities that can’t be replicated by a machine, and as such, AI’s place as a creative partner has been viewed with suspicion by some.
In many cases, AI’s most useful role has been one of a behind-the-scenes helper. A 2015 ad campaign by M&C Saatchi surprised even its own creative team, when its artificially intelligent poster examined how long people looked at its different creative iterations and evolved accordingly, reaching peak effective poster within just 72 hours.
AI is also hard at work optimizing ad buys, making sure the right creative is seen by the right people and, via Olapic’s Photorank algorithm, helping brands decide which photos are most likely to drive conversions in an e-commerce environment. In 2016, Shutterstock abandoned keywords, and instead used machine learning to analyze its database of 70 million images and 4 million video clips—everything from colors and shapes to tiny details—to more accurately make recommendations to users.
In addition to automating some of this grunt work and improving outcomes, some uses of AI suggest that it may also force us to reconsider our understanding of creativity.
In 2016, ad agency J. Walter Thompson used machine learning to produce a ‘new’ Rembrandt based on the painter’s previous works, and AI has also been used to transform photos into paintings in the style of Van Gogh. Even music doesn’t seem safe, with researchers at Sony unveiling an AI-composed pop song in 2016, based on analysis of thousands of different tracks.
The world of design is also experimenting with the technology, with AI system ANGELINA busy designing experimental video games since 2011, and San Francisco startup, The Grid, using AI to design complete websites.
At Monotype, machine learning and AI isn’t quite designing typefaces yet, but it is being investigated as a creative partner, used both to identify typefaces and to understand the qualities that humans perceive in them. It’s given rise to the new WhatTheFont app, which identifies typefaces instantly using photos taken by the user.
“There’s no field in computer science advancing as quickly as this,” says Monotype’s Research and Development Director Sampo Kaasila, whose team oversaw the creation of the deep learning algorithms powering the new WhatTheFont app. Kaasila has been fascinated with the possibilities of AI for years and has seen it develop from the early days, when IBM launched its chess-playing supercomputer Deep Blue, to Google’s AlphaGo and AlphaZero, which has won against some of the best ranked Go players in the world.
“I’m fascinated by what the machine can do, what its limits are, and its impact on society and what it means for us now and in the future,” he adds.
WhatTheFont is capable of recognizing over 133,000 fonts, and has learned to do so only after being fed vast amounts of data. Identifying typefaces, it turns out, is far more of a complex challenge than teaching a program to recognize an image of a cat or dog.
“This is actually what I describe as an extreme classification problem,” says Kaasila. His team had to take into account that WhatTheFont wouldn’t always be fed crisp, clean images of fonts, but blurry photographs taken from different angles and rotations, and often with the interference of overlays, background images, or minor modifications made by the designer. All this, while taking into account the fact that many fonts are very similar and the photographs contain only a limited amount of letters.
In order to counter this the research team created a 50 layer deep neural network and built an advanced ‘data augmentation/distortion pipeline’ that simulated real world interference such as shaky hands or strange perspective.
“The goal was for the machine to expect anything,” says Kaasila, “so we spent a lot of time creating the data augmentation/distortion pipeline to create an immense variety of training data, which increased our accuracy by an amazing amount. Deep learning is the secret behind this.”
Kaasila’s research team has also created machine learning-based technologies that suggest font pairs and recommend similar typefaces, potentially saving graphic designers hours of digging through font libraries. Other AI projects include teaching the computer to understand the more emotional qualities humans perceive in typefaces, whether it’s confident, happy, casual or dozens of other characteristics.
While humans still hold the reins when it comes to creating typefaces, all these examples show how AI could become an indispensable assistant in the creative process. It’s easy to see how it can encourage designers to think outside their (possibly biased) creative choices and consider new solutions.