Microsoft is developing an advanced AI technology that could enable viewers to hear different words from what a speaker actually said during live events. This innovation is based on a recently filed patent, which outlines a system capable of modifying both audio and video in near real time. Rather than merely adding subtitles or filtering sound, the proposed technology aims to edit the actual footage, creating the illusion that the speaker originally expressed the altered words.

The system would employ AI agents to identify specific words or phrases and determine whether they should be removed or replaced. Upon altering the audio, the technology would adjust the speaker’s mouth movements and facial expressions accordingly. This would ensure that the surrounding audio and video are also edited seamlessly, preventing viewers from noticing abrupt cuts or interruptions.

Additionally, the technology could allow for personalised viewing experiences, enabling the same broadcast to appear differently to various audiences based on predefined instructions or individual user profiles.

Applications Beyond Simple Translation

One notable application highlighted in the patent is live language translation. This would allow viewers to experience a speech in their preferred language, with the speaker’s mouth movements synchronised to the translated audio. Furthermore, the system could enhance fluency by eliminating unnecessary pauses, filler words such as ‘um’ and ‘ah’, and stutters. It could also correct mispronunciations without requiring the speaker to repeat themselves.

Moreover, the patent suggests that the technology could remove expressions that some viewers may find undesirable, including colloquial phrases or local idioms. For instance, terms like ‘y’all’ or ‘bite the bullet’ could be replaced with more universally accepted alternatives. The system might also identify and eliminate language deemed offensive or inappropriate, potentially utilising internet searches to find suitable replacements.

This level of control over live conversations would surpass traditional methods such as subtitles and audio filters, offering a new dimension to how audiences engage with live content.

Ethical Considerations of Real-Time Editing

The introduction of this technology raises significant ethical questions regarding the extent to which a live broadcast can be altered without misrepresenting the original content. While editing for clarity, such as removing stutters or translating speeches, may enhance audience understanding, changing a speaker’s expressions or words to fit viewer preferences presents a different challenge. A speaker’s choice of words, accent, and local expressions are integral to their communication style.

There is also the concern that different viewers might receive varied versions of the same speech without realising it. The alterations made to each viewer’s feed could result in a statement sounding entirely different, depending on the changes applied. To address these concerns, the patent suggests maintaining the original footage as ‘ground truth’, which would serve as a reference for the edited version.

The proposed system could be applicable across various types of live content, including lectures, business presentations, award ceremonies, television shows, cooking programmes, and live sports events. However, Microsoft has yet to announce any plans to develop this patent into a marketable product, leaving its future uncertain.

Long or Short, get news the way you like. No ads. No redirections. Download Newspin and Stay Alert, The CSR Journal Mobile app, for fast, crisp, clean updates!

App Store – https://apps.apple.com/in/app/newspin/id6746449540

Google Play Store – https://play.google.com/store/apps/details?id=com.inventifweb.newspin&pcampaignid=web_share