Hear how mixed reality and the Internet of Things are helping to preserve our past. Hosts Becca DeGregorio and Todd Whitney speak to Brad Smith, Will Lewis, and Yves Ubelmann.
Civilization and culture are built upon learnings from the past. But our rich tapestry of culture and language isn’t accessible to all, and in some cases, it’s in danger of disappearing entirely. With help from technology, innovators are redefining preservation. In this episode, we’ll hear from thought-leader and Microsoft president Brad Smith, as well as digital architect Yves Ubelmann and translation expert Will Lewis on how technology is playing a key role in protecting vital aspects of cultural heritage and making them accessible to all.
For more information about Microsoft's In Culture podcast, please visit: microsoft.com/inculture/podcast
Todd Whitney: And we're good to go, yeah? Cool. All right, let's go, yeah. Okay. All right. Hey, Brad. I am Todd, and would you mind introducing yourself?
Brad Smith: Hi, I'm Brad Smith. I'm the president of Microsoft.
Todd Whitney: Since we've been making this podcast, a big question for us has been where Microsoft sees themselves in culture. So when I found out I'd actually have the chance to fly out to Redmond and meet with Brad, I figured this is my chance. He's probably the biggest advocate at the company for putting their most sophisticated tech to use for good.
Brad Smith: We started with a portfolio of AI for good programs. I sometimes like to say, you know, we're a company that's known for its suite of products. Think about Microsoft Office and Outlook, Word, Excel, and PowerPoint. Well, we started with a program two years ago called AI for Earth, it focuses on how to use artificial intelligence to address issues like climate, water, biodiversity and the like. The farther we went, the more we realized that artificial intelligence can really address a wide variety of societal needs. So we've been working on building out the portfolio. We launched AI for Accessibility, then AI for Humanitarian Action, and then we realized this is an opportunity to focus on cultural heritage.
Becca DeGregorio: Cultural heritage is a pretty lofty concept. We hear about it a lot, but it doesn't always feel real or definable.
Todd Whitney: Okay, I'm definitely with you, but talking with Brad, I got the sense that cultural heritage really means something to him and for Microsoft.
Brad Smith: We focus on things like languages, places, people, historical artifacts. All of these are part of the rich tapestry of our world of communities.
Todd Whitney: And why would a big tech firm focus on stuff like that? It's because emerging tech has an important role to play in preserving it.
Becca DeGregorio: And on this episode we'll see that in action. We spoke to an architect turned digital preservation expert, and a researcher whose work in linguistics and machine learning is helping endangered language communities. Through their stories we see how tech is redefining and reinvigorating preservation, and making our cultural heritage more accessible to people around the world and for future generations.
Todd Whitney: From Microsoft, this is In Culture. I'm Todd Whitney.
Becca DeGregorio: And I'm Becca DeGregorio.
Todd Whitney: Let's go back to my conversation with Brad Smith. He said something during our conversation that surprised me, and that was that a big part of preservation in his mind is making the past more tangible and available for people.
Todd Whitney: Can we talk more about the accessibility aspect of that? What are we opening up? What are we making more accessible?
Brad Smith: Sure. It's a really great question and it's an important piece of this. Think about the great places in the world, a place you might go to. It might be a museum, it might be a monument, it might be a historical ruin. The only way you can experience those places today is actually to go there. What we recognized is with something like augmented reality, our HoloLens device, you can replicate the experience, and you can make that available to people everywhere in the world. So that's one way we make these places more accessible, but there's another way as well. We all know that if you go, for example, to one of the ruins in Greece, it might be the Parthenon in Athens, it might be Delphi. You can see what it looks like today. With augmented reality we can reconstruct the original place, and we can put you inside it. So it's both about making more accessible places that exist today, and also making more accessible places that in some cases haven't existed, at least the way they first did, for hundreds or even thousands of years. This kind of work isn't about technology or technologists alone. More so than many things it really is a multidisciplinary initiative.
Todd Whitney: When Brad told me this, my mind immediately went to a Skype call I'd had with Yves Ubelmann. Yves is the president and founder of Iconem, a French startup dedicated to digitally preserving buildings, monuments, and sometimes even whole cities. Here's the story. A few years ago, Brad was in Paris and visited a museum called the Museum des Plans-Reliefs, and it houses a collection of intricate relief maps. One of the most famous pieces in there is a scale model of Mont-Saint-Michel, this breathtaking abbey off the coast of Normandy, France.
Becca DeGregorio: We're definitely going to recommend you run an image search of this one. It's kind of beyond description, this fantasy castle rising up from the water.
Todd Whitney: Long story short, Brad wondered what technologies like augmented reality and artificial intelligence could bring to an experience like this. Like, how could it feel more relevant and tangible?
Becca DeGregorio: A year later, the museum opened a new Mont-Saint-Michel exhibit where visitors could see this incredible 18th Century scale model, then put on a hollow lens to explore the abbey in a way that's totally contemporary. Iconem handled 3D capture for the project.
Todd Whitney: Mont-Saint-Michel was personal for Yves. His grandfather had worked on the restoration of the Mont-Saint-Michel site many years ago, and highly visible cultural projects like this one are really important to Iconem's work. But what I found out is that what Iconem does, it goes way beyond what you can just put into a museum.
Yves Ubelmann: I'm Yves Ubelmann. So I'm an architect, I'm the co-founder of Iconem, who develop new technology in order to keep alive the memory of threatened cultural sites in many countries all over world in order to recall the very accurate imagery, 3D imagery of sites that disappear very fast today.
Todd Whitney: Now, why are a lot of these sites disappearing?
Yves Ubelmann: There is a lot of different issues. Conflict, war, destruction, bombing. So I saw a lot in Iraq and Syria. Sadly, some places that were completely destroyed by the fighting, even by groups like Islamic State who target this heritage. So we try to take a lot of 3D documentation to keep the memory of these places. It's shocking how when you come to a place one year, and you come back one year later and you see part of the site that was here is not here anymore. I was shocked by this. And so that's why I wanted to find a solution to at least be able to transmit these sites to the future generation.
Todd Whitney: And so when you're first getting started, people are first starting to use drones and artificial intelligence. Am I understanding that correctly?
Yves Ubelmann: At this time, when I was working at the beginning of when I was a young architect, we use some drawings by hand and low-tech to record this site. But it wasn't enough. With this kind of representation it's not possible to see the detail from one year to another. That's why I wanted to go further, and this imagery give us a really new way of seeing these problems. And now with artificial intelligence it's easier and faster to understand this past through this kind of representation, through this kind of mapping.
Todd Whitney: And a lot of the mapping that you're looking to do, and some points, some of the projects that you've done, you've made one-to-one replications of different sites, right?
Yves Ubelmann: Yes, we use cameras. We use drones in order to make hundreds of thousands of pictures from the field and then process the picture in order to get a photo realistic 3D model of the site. It's like a copy of the site as it is visible, but in the virtual world. It's interesting because you have point by point, centimeter by centimeter. Every detail you can see on the site.
Todd Whitney: The detail you get from Iconem's models would be valuable across the board. It gives researchers, but also lay folks, access to historical sites they'd never get otherwise. But that access, there's this other dimension to what he's providing.
Todd Whitney: And so it seems like the technology that you use and that you've come up with, a lot of it might not even be useful if you weren't able to do the basic human thing of making connections with people.
Yves Ubelmann: Yeah. I think the most important thing in this work is not technology. It's a connection of the people and the trust you can have in all these communities on the field, and this creates the job we are doing. And then after, just the technology is doing the rest. I mean, just processing and calculation.
Todd Whitney: Let me ask you about a lot of the technology that you're using.
Yves Ubelmann: There is a lot of different steps, but the first one is to go to the field. In the context of conflict area, you have to be fast, and so that's why you have to use specific technology to shoot the picture on the place. So we use drones, because drones can help us to take hundreds of picture in a few minutes. And it's useful because when you walk on places that was bombed or there is still landmines, for example, you can take a lot of pictures without put yourself in danger. So we use our camera from the ground with specific lens, and in few days we're able to take more than 100,000 pictures. So it's a lot of data. It's terabytes of data. And so then after, we come back in our office and we start processing the pictures, and processing this pictures we can get a very accurate 3D models.
Todd Whitney: Right now, Iconem is working on a project back home in France. It's happening outside of a conflict zone. But once again, it's personal.
Yves Ubelmann: The day of the fire of Notre-Dame, I was just in front of the monument. So I was completely shocked to see in real-time what was happening. So I was asking to myself, what can be done to help to restore this monument? So the idea was to build the biggest database of pictures from Notre-Dame in order to follow the evolution of the monument through time, because this monument was restorated a lot of times. And then after the idea was also to had a picture of the fire itself and to see in real-time, again, through the 3D model the destruction of the monument, and all this information, all this data, will help the restorer to see how was as a monument before the fire. And we found some very interesting pictures from different photographers, and using this processing method we were able to match every picture together in the big database. And now we are trying to make this database accessible to the people, so to the architect, and we hope soon to the public as well.
Becca DeGregorio: I feel like Iconem's technology is super complex, but it's probably not the hardest part of his job.
Todd Whitney: No, not at all. I asked him about this and he said AI drones, managing terabytes of photographs, all that stuff is truly the easy part. It's the human relationships he's forming and the politics that he's navigating. Those are the very real issues that come with traveling to some of the most dangerous spots on earth. And that's the stuff he's really having to flex on.
Yves Ubelmann: I have no background in technology. I'm not a computer scientist, I'm not a developer. I just came to the technology because I had a very specific issue in my daily work. Technology, for me, is just here to solve problems that human have.
Becca DeGregorio: Here's Todd and Brad Smith again.
Todd Whitney: When it comes to culture, I think for the most part everybody's like, yeah, yay culture. It's just something to be celebrated. But with the AI for Cultural Heritage Initiative, it puts a different emphasis on culture. It's not just something that's kind of there and cool, but it's imperative. Can you speak to that?
Brad Smith: I think there's two really important dimensions to what you're describing. First, I think communities want to retain their own identity, and they cannot retain their own identity unless they can preserve and protect and enhance access to their own culture. But there's another dimension that I think we're very focused on at Microsoft. We actually want to bring to life cultures so that other people can appreciate them as well. It's easy in the world today for people to try to preserve what is special about themselves by turning inwards. I think part of what we're trying to do is give people the tools to preserve what is special about themselves without feeling that they need to separate themselves from the rest of the world to do so. But the other thing we want to do with this program is give us all the opportunity to appreciate what's special in other people, what's special in other communities and other countries.
Becca DeGregorio: This takes me to another strain of cultural heritage preservation, something that's hiding in plain sight for most of us. It's language.
Todd Whitney: Obviously, we're speaking English on this podcast. It's our first language, and it's among the most widely spoken languages in the world. And we hear it all over the planet, across the internet, in movies, TV shows.
Becca DeGregorio: But that's a bubble. We forget that for speakers of many other languages, there's not nearly the same reach, the same resources, and it puts languages in danger. Through the grapevine at Microsoft, I heard about a researcher who's dialed in on this issue. I called him on Skype recently to hear more.
Will Lewis: My name is William Lewis. I am a principal PM architect on the Microsoft Translator team. We build machine translation engines for a variety of the world's languages, and this is the technology that allows us to translate between languages.
Becca DeGregorio: Will started as a software developer, but he landed in academia studying linguistics, specifically the field of endangered language preservation. About 13 years ago he heard about a brand new team at Microsoft doing work with machine translation and he's been there ever since.
Becca DeGregorio: Take me back a little bit to your interest in linguistics. You said that you were drawn to languages. Can you just pick apart why?
Will Lewis: Languages, I think, are the most unique human characteristic. We're the only animal on the planet that communicates in as sophisticated a manner as we do. Plenty of animals on this planet are able to communicate with one another, but a human language has characteristics that are very unique in that we can talk about abstractions, things that are not in the here and now, things that never would be or never will be, and be able to communicate these kinds of thoughts to other individuals. It's remarkable that we're able to do it, and it's also remarkable that languages exist almost as if they've always been here and we don't even really perceive our having learned them as children. It seems like it's so trivial and so easy, but as it turns out, languages are insanely complicated. It's remarkable.
Becca DeGregorio: That's a great linguistics nerd response. Thank you.
Will Lewis: With plenty of words.
Becca DeGregorio: Would you consider yourself a linguistics nerd?
Will Lewis: No doubt. I think my boss would consider me not a linguistics nerd, but a linguistics snob.
Becca DeGregorio: Can you talk a little bit about when machine translation kind of came into the mix? Because a lot of us associate the word translation with human-to-human translation.
Will Lewis: Machine translation has actually existed as a science for a very long time. The first work in machine translation occurred in the 1950s. At that time, they assumed all that would need be needed is that you would put in a dictionary of Russian content, put in a dictionary of English content, spend a little bit of time training and voila, you would have a machine translation engine able to translate between the two languages. And Russian was the focus, of course, because it was the middle of the Cold War. And the scientists at the time, it's interesting listening to some of the interviews at that time. It's like, oh, well, we'll have the machine translation problem solved in the next five years.
And we look back on that and we look at where the work that they had done and we go, how quaint. Now you have translation quality that is dramatically better than that period of time. It's possible now to actually read content in another language that's been machine translated. The Microsoft Translator team is focused on expanding language coverage. We want to expand to more and more languages of the world. That's something that's always been a goal of ours since the very beginning.
The actual work on endangered languages, or I would characterize this more not just endanger language, but under resource language, is really started with Haitian Creole. So the work on Haitian Creole was prompted by the earthquake that had happened in Haiti in 2010 and there were no tools available for Haitian Creole at the time. It was a language that was spoken by about 7 million people, but isolated primarily to that one island. And within a several week span we were able to build up a machine translation engine for Haitian Creole, more or less from scratch with community involvement with a variety of partners and that prompted some work on our side.
It's like, can we do this again? If we had the ability to build machine translation engines for languages that are under resourced, could we replicate what we did there again? And that's where we started to work on projects, like with the Hmong and with other communities where, let's look at other partners that we might have. Let's look at other languages that are spoken by communities in the United States and elsewhere in the world where we might be able to bring our technology to bear and be able to build machine translation engines for those languages.
Becca DeGregorio: What does the process soup to nuts look like?
Will Lewis: The way it's trained is it involves forms of machine learning where you provide content in the two languages that you're interested in. And so over a very large amount of data, it's able to not just figure out words in small phrases like that, but words within context, how words map between the two different languages. The learner actually starts to build a very large model of probabilities between the words and phrases in one language and what the potential output is in the other language. So when we're training lower resource languages, it becomes problematic because we're dealing with a lot less data.
Becca DeGregorio: Right. So when you're talking about endangered languages, it's not just about all being in the same place. It's about having a low scale of speakers.
Will Lewis: Yeah. Having a small number of speakers, and then also being under threat from majority languages that are spoken in the same region. How do we go about finding the data in the first place to be able to train the machine translation engines that we want for those languages? And that is not an easy task. So there are two parts to that. One is finding the resources that might exist. So as it turns out, Hmong is in fact being translated into in various communities, there's a fairly large community of Hmong in the Minneapolis-St. Paul area, and the municipal government there does actively translate content into Hmong. So working with people in that community allowed us access to some of the content that was in English and Hmong that that municipality had been translating. So that's one thing.
So there are people in these communities who speak the language, who would know, for instance, where there might be resources, would know, for example, that there is data that the Fresno city counselor and Fresno school district has translated. We may not have been able to find it, but they know where it is because they use that content. Likewise, they may be aware of the other materials that have been translated that are available for the community or content that exists in Hmong that we would never have known about that they're able to find.
Becca DeGregorio: So you feed all this data into the machine. What's the goal?
Will Lewis: Imagine if there were a technology that you could robustly build a machine translation for any language such that you would not have to communicate in another language. You could communicate in your own language and people would understand you, and then vice versa. I don't think we're quite at that robust stage yet, but we're moving in that direction. So the world could be a very different place in the not too distant future where people could in fact communicate in an indigenous or endangered language with someone who speaks majority language and not have to leave the confines of their language.
I had a meeting of the Hmong community members when we initially shipped, so there's a celebration about shipping the Hmong language. A Microsoft translator and one of the students got up and spoke about the fact that he could use messenger to communicate with his friends, and he thought it was so cool that he could use his language. So I think the ultimate goal is that, is to be able to facilitate communication between people that speak different languages, that represent or are representatives of their different communities, and not ask them to leave anything behind.
Becca DeGregorio: I asked Will what languages mean to him. Really, why he thinks they're so important. And he told me that it's all about how fundamental they are to us humans. He told me how Creoles are formed by the children of parents who speak combinations of languages, and that all you need to create language is two babies. Well, two children younger than 11 or so. In the absence of established language, we make a new one. It's inherent, and yet like all culture, it's fragile.
Todd Whitney: Through technology we can give cultural heritage a lifeline. We can make it more accessible, more relevant, and more current. I'll give Brad Smith the final thought.
Brad Smith: To me, we benefit from reminding ourselves that we actually live on a pretty small planet, and we should learn from and celebrate the different things that different communities have created, and not just what our own community has created itself.
Todd Whitney: To learn more about all the people and stories featured in this episode, visit microsoft.com/inculture. There you can read all about Microsoft's AI for Cultural Heritage Initiative and many of the other projects to come out of AI for good. And to see more of the sights from this episode in the series overall, check us out on Instagram @MicrosoftInCulture. In Culture is hosted by Becca DeGregorio, and me, Todd Whitney. It's produced by Jordan Rothlein and edited and mixed by Nat Weiner. Original music by Angular Wave Research. In Culture is production of Microsoft in collaboration with Listen, a sensory experience company in New York City.