Sparking Global Connections to Art through Open Data and Artificial Intelligence

Within the next ten years, artificial intelligence (AI) will transform how audiences around the world connect with art. To help prepare for, and shape, that future, we are today marking the second anniversary of The Met's Open Access program by releasing a new subject-keyword dataset that enhances how both humans and machines can filter and analyze the Museum's collection. We are also revealing a set of AI prototypes developed through a unique collaboration with Microsoft and the Massachusetts Institute of Technology (MIT), which seeks to leverage the new dataset and The Met Collection API to spark new global connections to art through Microsoft AI.

A Paradigm Shift

Our ambition is to make The Met collection one of the most accessible, discoverable, and useful on the internet. We want to educate and disseminate knowledge beyond the physical perimeter of the institution and foster understanding of the multiple histories of the artworks. Releasing the hi-res images of The Met collection and its data into the public domain under Creative Commons Zero (CC0) has proven to be a paradigm shift in how we achieve that ambition. So far the program has:

• affected how we think about increasing digital access to the collection, which has evolved from metmuseum.org visits to greater access through third-party platforms;

• increased the reach of The Met collection on Wikimedia by a factor of four—from four million pageviews a month to seventeen million in two years;

• provided a platform for communities to create new ways of understanding the collection;

• enabled sustainable data integrations with third-party websites, which has resulted in a 300% increase in engagement with the collection;

• inspired new art by artists including Simone Seagle and Mushon Zer-Aviv; and

• contributed to a 15% year-over-year increase in pageviews of the collection on The Met's website, while image downloads have also increased proportionally.

It has been humbling also to see our Open Access program referenced by peers at the Barnes Foundation, the National Palace Museum in Taiwan, and, most recently, the Cleveland Museum of Art, when they launched similar programs.

Our Open Access program is a work in progress, and we regroup regularly to determine the best next step. There is no blueprint to follow, which means there are always lessons to learn, new opportunities to pursue, and challenges to overcome. Most recently we have focused on how to enrich the dataset so that it better reveals the depth and complexities of the collection. The solution we reached was to add quality-controlled subject keywords to the more than 300,000 digitized artworks in the collection.

How Much Dancing Is There in the Collection?

The dataset of subject keywords provides the foundation for new types of analyses and parsing of the collection. The taxonomy was developed specifically for our collection, and was designed to be accessible and user-focused. The keywords simply answer the question, "What do you see?"

The 1,063 subject keywords range from "trees" to "castles," from "floods" and "portraits," and from "ritual objects" to "cats." By adding subject keywords to the artworks' data, it has enabled us to determine the most commonly depicted subjects in The Met collection: men (59,249 artworks), followed by women (35,433), portraits (33,178), and flowers (21,691). There are also 1,325 artworks that depict dancing!

Six representations of dancing in The Met collection created across five millennia by a group of diverse cultures from across the globe. See below for full image credits.

These keywords enable users to better understand how a common subject has been depicted across time periods and cultures. For example, you can now easily explore depictions of dancing on an incredible Sumerian plaque from the third millennium B.C., a small Greek terracotta statue from the fourth century B.C., a beautiful sixteenth-century Japanese hanging scroll, and an eighteenth-century oil painting by Giovanni Domenico Tiepolo. Or if your interest is in the portrayal of turkeys, you can now easily discover examples in an ancient vessel from Mexico, a German tureen, an American pastel on wove paper, and an oil painting by Jean-François Millet.

The subject keywords are now available under CC0 in both a downloadable CSV file and via The Met Collection API. There is more information available about the keywords on our website.

One of the most exciting outcomes of adding subject keywords to every artwork is that it has opened up the collection for new types of analysis and exploration through AI. Not only do the keywords provide a unique, quality-controlled data set for visual recognition AI to use as training data, they also open the door for building new scalable and personalized ways for users to interact with the collection.

The Met x Microsoft x MIT: #ArtMeetsAI

As a proof-of-concept for the ways in which art, open data, and AI can scale the global reach and relevance of The Met collection, late last year members of our Digital Department, several curatorial departments, and the Department of Scientific Research joined a group of engineers, researchers, students, designers, artists, and technologists from Microsoft and MIT to participate in a two-day hackathon. The goal of this event was to imagine and develop scalable new ways for global audiences to discover, learn, and create with one of the world's foremost art collections through Microsoft's AI platform. The vision and development of a series of basic AI prototypes was initiated at the hackathon, and, over the past two months, we have been developing and refining those further.

A group of participants at The Met x Microsoft x MIT hackathon event in December. Pictured at left is Chief Digital Officer Loic Tallon (front) and Met curator Kurt Behrendt (back). Photo by Natasha Moustache / Listen

This evening in The Met Fifth Avenue's Great Hall, together with Microsoft and MIT, we will reveal five of the AI prototypes that were developed through this cross-disciplinary collaboration:

• Artwork of the Day: Discover the artwork in The Met collection that will resonate with you today based on your location, preferences, world events, local news, and historical data.

• Tag, That's It!: Partner with AI to increase the accessibility of The Met collection for the millions of users across the Wikimedia platforms.

• Storyteller: Use AI to find artworks in The Met collection to illustrate any story you might want to tell, or conversation you are having.

• Gen Studio: Employ AI to creatively navigate the shared visual features and dimensions underlying The Met collection.

• My Life, My Met: Utilize AI to turn your Instagram feed into a work of art.

For more information about the prototypes and to test them yourself, please visit the project website.

We are excited by the potential each of these prototypes demonstrates by mixing AI, open data, and human creativity. Each one provides a unique new perspective about how AI could transform the ways audiences connect with art, pointing to promising opportunities where a personalized artwork of the day could be displayed on your bathroom mirror every morning, or where a child's storytime could be automatically illustrated with artworks from The Met collection—no matter the story or the language with which it's told.

We hope these AI prototypes inspire, invite, and empower communities to follow in these footsteps, and to use The Met Collection API and AI technologies to create new forms of engagement with art that can be scaled around the world. It is this aspect of the Open Access program that is most transformative for The Met: opening the door for all types of makers, creators, artists, and technologists to participate in the collection and develop new narratives from the five thousand years of human history represented by the artworks.

Next Steps in Open Access at The Met

• Integration with metmuseum.org: Integrate the subject keywords into the Museum's website to improve the browsing experience of the collection, and enable the easy filtering of the collection based on the keywords.

• From prototype to product: Establish product criteria and specifications to introduce the use of AI technologies in our existing digital products in a manner that is sustainable and scalable.

• Wikidata integration: Utilize the subject keyword as a Wikimedia Structured Data on Commons pilot to fill in the "depicts" field so that Met images are accessible through the Wikimedia platforms worldwide, and enable the ability for The Met to source additional subject keywords from the collective wisdom at work on the Wikimedia platforms.

Composite image of dancing in The Met collection, clockwise from top left: Terracotta statuette of a dancing girl, 4th century B.C. Greek, Corinthian, Late Classical. Terracotta, H. 4 15/16 in. (12.5 cm). The Metropolitan Museum of Art, New York, Rogers Fund, 1906 (06.1111). Sesson Shūkei (ca. 1504–ca. 1589). Seven Sages of the Bamboo Grove (detail), 1550s. Japanese, Muromachi period (1392–1573). Hanging scroll; ink and color on paper, image: 40 5/16 x 20 3/8 in. (102.4 x 51.7 cm). The Metropolitan Museum of Art, New York, Mary Griggs Burke Collection, Gift of the Mary and Jackson Burke Foundation, 2015 (2015.300.50). Giovanni Domenico Tiepolo (Italian, 1727–1804). A Dance in the Country (detail), ca. 1755. Oil on canvas, 29 3/4 x 47 1/4 in. (75.6 x 120 cm). The Metropolitan Museum of Art, New York, Gift of Mr. and Mrs. Charles Wrightsman, 1980 (1980.67). Inlay: dancing man, ca. 2600–2350 B.C. Mesopotamia, Nippur, Early Dynastic III. Sumerian. Shell, 1.81 x 0.94 x 0.24 in. (4.6 x 2.39 x 0.61 cm). The Metropolitan Museum of Art, New York, Rogers Fund, 1959 (59.41.53). Anonymous, Italian. Children Dancing (detail), 16th century. Red chalk, 8 1/8 x 10 3/4 in. (20.6 x 27.3 cm). The Metropolitan Museum of Art, New York, Gift of Cornelius Vanderbilt, 1880 (80.3.232). William Merritt Chase (American, 1849–1916). Carmencita (detail), 1890. Oil on canvas, 69 7/8 x 40 7/8 in. ( 177.5 x 103.8 cm). The Metropolitan Museum of Art, New York, Gift of Sir William Van Horne, 1906 (06.969)

Learn More

The Met, Microsoft, and MIT held a two-day hackathon session to explore how artificial intelligence could connect people to art. Learn about this collaboration and the prototypes it generated.