The five slender fresco fragments, reassembled in a building near the ancient Roman city of Pompeii, form half of a brown-haired woman’s face. Gazing at them, I wonder: Did she become one of the frozen-in-time bodies of Pompeii, an undying symbol of the fragility of life?
Mount Vesuvius towers over Pompeii, reminding visitors of the raging pyroclastic flow that erupted from the 4,000-foot volcano in A.D. 79. The speed and intensity of that blast — which covered the city in lava and ash — is the reason this once-thriving Roman city southeast of Naples presents amazingly well-preserved buildings and artifacts of daily life, and why archaeologists and scientists have been studying Pompeii for centuries.
The lava also preserved Pompeii’s art, such as the fresco of the brown-haired woman, still vivid in color as if it were painted yesterday. But up until now, recreating that art in its original form has been elusive. The sheer amount and irregular shapes of the fragments amount to an enormous jigsaw puzzle. It’s only now, with the help of robots and artificial intelligence, that scientists can literally piece together some of Pompeii’s treasures. This reconstruction of shattered frescoes might give historians fresh insights about Pompeii’s artistic styles. Even more important, we might get access to whole new perspectives about daily life in the Roman Empire.
For a year now, researchers from around the world have been working in Pompeii on a project called RePAIR — for Reconstructing the Past: Artificial Intelligence and Robotics meet Cultural Heritage. RePAIR, funded by the European Union, is bringing together computer scientists, robotics engineers, and archaeologists to create a novel robotic system that will perform one of the most manually intensive and complicated tasks out there: reconstructing shattered ancient artifacts.
Inside the Casina Rustica, a building north of Pompeii serving as a laboratory and a warehouse, about 15,000 pieces of ancient art are lying inside boxes and on shelves, in all shades of red ochre, off-beige, gray-green, eggshell white, and pastel salmon. The shards come in an incredible variety of angles and sizes. A close look at the fragments reveals floral ornaments, faces, palmette motifs, and hints of familiar figures. (Is that a leaping deer?)
“All these pieces you see probably belong to several frescoes,” says Ohad Ben-Shahar, a computer science professor at Ben-Gurion University of the Negev. “These must have been ceiling paintings of different rooms in one of the buildings that collapsed under the eruption.”
A team from the University of Lausanne estimates that four to eight shattered frescoes are waiting to be reassembled in the warehouse. By all accounts, Ben-Shahar says, they were the ceilings of two buildings: the Schola Armaturarum, the headquarters of a military association, and the Casa dei Pittori al Lavoro, or the House of the Painters at Work. Both were part of an urban complex known as the House of the Chaste Lovers, a bread factory whose frescoes portray chaste images (unlike typical Pompeii frescoes, which were more overtly sexual).
In RePAIR’s current phase, students with 3D scanners and cameras are capturing digital images of the artifacts and organizing them in a database. Next, an artificial intelligence will study the fragments’ features and calculate their compatibility according to their 3D morphology. The AI will determine which room and which fresco each piece belongs to.
Then a state-of-the-art new robot — “a big, black, rectangular frame with two mechanical arms,” says Marcelo Pelillo, a computer science professor at Ca’ Foscari University of Venice — will physically restore the frescoes. Scientists are still determining the robot’s final design, but it will delicately move around and scan the fragile fresco pieces with soft humanoid arms and hands, embedded with high-definition scanners, cameras, 3D digital recognition software, and sophisticated sensors. The frame, mounted onto a sliding mechanism, will also work its way around the Casina Rustica.
RePAIR kicked off in September 2021, but its origins go back to 2014, when Ben-Shahar — who primarily works with robots in agriculture and fisheries — asked Pelillo, a longtime friend, to collaborate on a puzzle-solving project. The Israeli scientist had already used computers to solve 10,000-piece digital puzzles that “no human would want to cope with — essentially one big mess,” he says. His lab at Ben-Gurion University had developed a prototype of a robotic puzzle-solver, with a mechanical arm, a camera for an eye, and the ability to reconstruct a visual artifact. Now, Ben-Shahar wanted to test it on a wider stage.
“Maybe we can implement the same technology in real life, on puzzles from archaeology,” he says.
Pompeii, one of the world’s most famous archaeological sites, was the obvious place to try. The city was founded in the 7th or 6th century B.C. by the Osci, or Oscans, from central and south Italy. It later came under the influence of Greeks, Etruscans, and Samnites (an Italian tribe) before coming under the rule of Rome in the 4th century B.C. By the time of its destruction, the city had exploded in Greco-Roman sophistication: it had an amphitheater, a gymnasium, a grid of about 100 one-way streets, effervescent fountains, and stylish villas with exquisite artworks.
The eruption of Vesuvius — which may have happened on August 24, A.D. 79, or around October 24 of that year — is estimated to have killed 16,000 people from Pompeii, the nearby city of Herculaneum, and surrounding towns and villages, burying everyone and everything in 13 to 20 feet of ash and pumice. It took 1,520 years for Pompeii to be discovered, in 1599, and it wasn’t until 1748 that excavators started to investigate the ruins.
In the mid-1800s, diggers in Pompeii noticed that the remains of skeletons across the site were surrounded by voids in the ash. They poured plaster into the spaces, and what emerged were the molds of bodies, many left in the exact poses in which they had met their demise. The bodies of Pompeii are essentially hollow plaster casts, but they have expressions that can raze your soul to the ground. One kneels and hides their face in their hands as if to nestle away from a nightmare; another seems to be crawling in defeat; a couple of others are huddled together for the last time.
One can only imagine the historical knowledge we might still be missing because the Pompeii frescoes are broken up.
On a small guided tour, I try to take in as much Pompeii as possible, from the grandiose Forum, Pompeii’s civic center, to an early type of gym called the Palaestra. The sublime Villa dei Misteri, or House of the Mysteries, is famous for its series of continuous, fiercely colored frescoes that cover three walls. They show the initiation of a young woman into the Greco-Roman mystery cult of Dionysus, the Greek god of wine, fertility, and orgies.
Pompeii was a society stratified by class: slaves had to wash their owners’ clothes with water and urine (for tough stains) at the Fullonica of Stephanus, the public laundry with a large tub in its center, according to Jessica, our guide. Pompeiians loved their street food. They even had their own Thermopolium, a type of ancient fast-food joint.
Jessica points to a tall, conical carving in the ground. “Why is it pointing at me?” says Ben-Shahar, who is right in front of it, and everyone in the tour group bursts into roaring laughter.
“There is a theory that the erect phalluses sticking out of the buildings meant the houses were brothels,” Jessica says. (Classicist Mary Beard has disputed this theory, though: in Ancient Roman times, she notes, they were often symbols of good luck, wealth, or fertility.) When we enter a brothel, we see one more alleged use of frescoes: a “menu” that might have depicted the specialties of the women who worked in the room.
Frescoes, works of wall or ceiling art created by mixing powdered pigment with water and applying it to wet plaster, were important to Roman society because they represented everyday life, the beliefs of the citizens, and the condition of Roman art and civilization in general. Our knowledge of Roman painting comes almost entirely from the preserved frescoes, which depict portraits, mythological scenes, gladiator contests, flora and fauna, or even entire landscapes and townscapes. Their vivid colors and designs adorned every Roman building, from military edifices and tombs to private villas.
One can only imagine the historical knowledge we might still be missing because the Pompeii frescoes are broken up. The frescoes of the House of the Mysteries, for example, seem to portray several characters, but it’s unclear from the fragments what the characters are doing. The murals in the villa are an example of megalography, a type of painting meant to present grand things, such as heroes and gods. This means that these murals are supposed to be read together, and that their characters are not random, but intriguingly related.
Piecing the broken frescoes together is a feat that before now would have required human dexterity, says Benedikt Kreis, a Ph.D. student at the Humanoid Robots Lab at the University of Bonn and a member of RePAIR. For this project, Kreis is responsible for the motion planning of both robotic arms and the precise placement of fresco fragments. He will use machine learning and intricate algorithms to teach the arms how to grasp, place, and push the fragments as gently as possible.
Grasping and placing comes easy to us. “But for a robot, it’s quite hard,” Kreis says. So far, most humanoid robots have performed actions and movements with remarkable sluggishness, he continues. Kreis wants to focus his research on making robotic grasping and placing more lifelike.
This is the direction that Manuel Giuseppe Catalano also wants to take. A researcher of soft robotics for human cooperation and rehabilitation at the Italian Institute of Technology and another member of the RePAIR project, Catalano is the co-creator of SoftHandPro, a novel robotic artificial limb that aspires to master the mechanics of the human hand. He believes the RePAIR project has the potential to advance, not just the knowledge of art, but the field of robotics.
“Robotics was made for industrial applications,” says Catalano. “These industrial applications were made to be very precise and repetitive. But in the last 30 years, there is this new paradigm where you want the robot to interact with a world that is not structured. For this, you need to have a robot with elastic body parts.”
The human hand is a splendidly complex and dexterous network of 27 bones and many muscles, ligaments, and tendons. Our tiniest grab of an object is the result of the synergistic work of all these hand components. Transferring the ability to grab from humans to robots will be further developed through RePAIR, says Catalano.
“RePAIR is a great union of AI and robotics,” says Waleed Meleis, an engineering professor at Northeastern University. “The robot has to have this tactile sensing ability to find the right pieces to pick, the computational ability to figure out what part matches with what other part, and the visual ability to actually recognize the pieces with what they contain and match them against databases.” He thinks the cutting-edge robotic technologies the group is developing for this new application tick all of the boxes.
This is not the first time an AI has reconstructed ancient artifacts, Meleis adds. A deep neural network named Ithaca, developed by an Google’s DeepMind lab and researchers from Ca’ Foscari University of Venice and Harvard University, has already restored damaged ancient Greek inscriptions, located their places of origin (for instance, the region of Delphi), and dated them with impressive accuracy (for example, to 273 B.C. or 413 B.C.). What makes the Pompeii project stand out is its interdisciplinary approach: “It brings together theorists and domain experts from different areas,” Meleis says. These are exactly the kinds of collaborations that produce the highest-impact results, he points out.
Some RePAIR researchers work in multiple fields. I met Arianna Traviglia, coordinator of the Center for Cultural Heritage Technology at the Italian Institute of Technology, as she experimented with hyperspectral imaging inside Casina Rustica. Hyperspectral imaging collects and processes data across a large number of wavelengths of the electromagnetic spectrum, revealing information about the color and images in the frescoes that the naked eye cannot see, such as the “forbidden colors” of red-green and yellow-blue, made up of pairs of hues whose light frequencies drown each other out in the human eye.
Traviglia studied history in Venice and archaeology at the University of Trieste before embarking on a Ph.D. in geomatics and geographic information systems. In her current capacity in RePAIR, she sees herself as a “translator between two worlds,” she says. Archaeology helps us understand where we come from, and technology can facilitate and enrich the chase.
For the task of recreating ancient artwork, knowledge of the humanities is especially useful. “If you find yellow dots in a fragment, that means the piece belongs to the middle of the fresco,” says Kreis. The RePAIR squad learned this from Michel E. Fuchs, a professor of Roman archaeology at the University of Lausanne, who had been studying the Pompeii frescoes before the RePAIR project began. “This is a key concept from when archaeologists reconstructed frescoes with their hands. None of us would have known,” admits Kreis.
Can we hope for a technology that will reconstruct not only frescoes, but other heirlooms from antiquity, such as papyri and vases? “It will be challenging, but we are hopeful,” says Pelillo. The coordinator of RePAIR looks forward to unlocking even more fascinating snapshots of Roman life. I keep his promise in mind as I leave Pompeii, taking a last look at Vesuvius, the shattered-peak mountain that both destroyed and perpetuated a beautiful Roman enigma.
Published on
