The National Theatre has unveiled new smart glasses for hearing-impaired audiences. Developed over three years, the glasses flash the dialogue as the actors say it. Nick Clark tried out the ‘transformational’ eyewear
At the National Theatre’s autumn press conference last week, the most exciting news was not that Sam Mendes was returning to direct a play about Lehman Brothers or the premiere of a new work by David Hare.
Instead, what caught the eye was a pair of glasses held aloft by NT director Rufus Norris that soon may become become a regular fixture in the National’s three auditoriums.
The innovation, dubbed Open Access Smart Capture, was designed particularly with deaf, deafened and hard of hearing audience members in mind. It offers personal captioning, flashing the play’s dialogue in front of the wearer’s eyes as the actors say it.
It is a project that has been three years in the making and when the testing is complete in October 2018, it could prove “transformational”, according to Norris. It will be available for every performance the National puts on.
In launching its new smart glasses, the National pointed to predictions from Action on Hearing Loss that in less than 20 years, one in five will be affected by hearing loss. The theatre says this equates to 11 million potential customers.
“If you think about it even for a minute you can understand that if we can get this right and develop this type of technology, the possibilities in terms of broadening our audience and really serving the people of this country are pretty fantastic,” Norris adds.
It means hearing-impaired visitors will no longer have to rely on the four captioned shows – with large LED caption screens on either side of the stage – that the National programmes for each production run.
‘Open captioned’ refers to a system where everyone is reading from the same screen. The glasses are ‘closed captioned’ as they are used by individuals. Captions are distinguished from subtitles because they provide additional information including speakers’ names, sound effects and offstage noises.
The glasses, which are designed and manufactured by technology company Epson, look suitably futuristic. Epson has been involved in closed caption technology for glasses for more than five years following a collaboration in Italy, with a group called MovieReading, which used it in cinemas. The lenses of the glasses work like the screen of a computer with the images projected on them.
A wire connects the glasses to a hand-held device that allows the audience members to customise the text – from its size and colour, to the background, the position on the screen and even whether the sentences scroll or ‘snap’ in the manner of traditional subtitles.
Melanie Sharpe, chief executive of StageText, which provides captioning and live subtitling services to theatres, welcomed the development. “It is great they’re exploring new technology,” she says. However, she adds that the new glasses “must be part of a range of options available”.
The roots of Open Access Smart Capture can be traced back to 2014 with a project piloted by the National and StageText. It was funded by Nesta, the UK innovation charity that supports social ventures.
That project failed to take off, but when the NT’s technical director Jonathan Suffolk saw Sony using similar technology at its cinemas in the US, he decided to resurrect it. So, the National brought in Andrew Lambourne, one of the world’s experts in live-subtitling techniques who’d worked on the original project, to redevelop the software last year.
Six months later, Accenture came on board as the National’s innovation partner and to help develop the system. The consultancy found the eyewear, developed the technology that gets the text to the glasses and which underpins the onscreen menu. And, according to Suffolk, “injected nitrous oxide into the project”.
Lambourne believed that during a long career of subtitling services particularly within the television sector, the “one loose thread in my career was whether we could do it for theatre”.
He had started a decade ago with the “easiest problem to crack”, which was that of a teleprompter scrolling automatically as the text was read by a news anchor. “That was easy because it is a quiet environment with a single person speaking clearly,” he says.
“A stage environment is a lot more difficult technically. There are potentially multiple people, moving around in a noisy space. Synchronising the script with the dialogue is a matter of timing.”
The Open Access Smart Capture system picks up the dialogue from actors wearing radio microphones or from microphones pointed at the stage, with audio compression used to improve the quality of the sound.
The speech recognition is built on a Linux operating system. The information is fed to the platform, which matches what it hears to the text.
Speech and language technology can now reasonably transcribe speech, but in this audio context the results can be mixed, Lambourne says.
Seeing is believing: how the glasses work first hand
Journalists were given a sneak preview of how the glasses worked after the National’s autumn press conference.
In a studio at Rambert dance company headquarters opposite the National, we nervously put on the Star Trek’s Geordi La Forge-style specs and took our seats.
Initially the glasses pinched on the nose and were a little awkward to balance. I had just about become used to them as Olivia Colman and Olivia Williams (above, started a scene from Mosquitoes, their sell-out play that recently finished its run at the Dorfman.
Sure enough, as Williams’ character Alice tells Colman’s Jenny of the trials of childbirth, the words flashed up on the lenses, and kept fairly good time. It took a while to get used to the sensation of seeing the text snapping over the action, and an effort to stop the eyes crossing, but after a few minutes it worked well.
I could place the text between the two actors and read it without having to take my eyes off the action.
The mobile phone-sized control was a little fiddly but was fine once I got the hang of it, and was able to put a border around the text, turn it blue, pop it to the top of the glasses and then had it scrolling. A year of fine-tuning, and this could be as transformative as Rufus Norris believes.
So he set to work on a system that worked off “multiple clues” to ensure the text in the glasses matches the action on stage. “It feeds off the voices, the sound cues, lighting cues and how they relate to the dialogue.” The National aims for 97% accuracy in the timing of its captions by next year.
“So, you have a feed of what the system thinks the actor is saying, to which you match the script. You then meld that with the stage cue information and the timeline information. The engine decides which of those cues it follows.”
What happens if an actor wildly improvises or blanks?
“All bets are off if that happens. They follow what the actor says but they can’t recognise other stuff,” Lambourne says. There is a fall back. “The stage manager has a nudging box. So if they see the text is drifting, they can press a button and bring it back in line.”
Ultimately, the idea is that the system runs without needing a technician to oversee it, Lambourne says. “We’ve got to reach a point where it is good enough to be left on its own.”
This does not, however, spell the end for those who currently prepare captions for productions, the specialist adds. “It enhances their role. They will be bringing together all this information and, as the production evolves, and maybe the script changes, it will be edited and updated.”
The next year will serve as a pilot, with the results evaluated by the University of Roehampton. It will involve 46 people testing the system in different shows starting, appropriately, with Beginning at the Dorfman, followed by Pinocchio at the Lyttelton and then Macbeth in the Olivier.
The National is also developing an audio description service to roll out in April 2019, which will also test during Macbeth, alongside VocalEyes, the group that works on theatre access technology for the blind and partially sighted.
The National has put £70,000 into the project and taken a patent on the system. The plan is to make it more widely available to “theatres large and small. Part of the process is us creating a system that is deployable and affordable”, according to Suffolk.
In a year’s time, anyone will be able to use the glasses in any of the three auditoriums. Suffolk adds: “And who knows, in the future people may like it so much they start buying the eyewear themselves and bringing their own to the shows.”