Deep-Learning based Joint Audio, Video Processing for Augmented Listening

Augmented listening implies the extraction of desired audio signal(s) from a distorted capture. Like human perception of speech, where visual and acoustic cues jointly contribute to the understanding, we aim to improve this extraction by augmenting the audio with visual information. A side-application is the detection of inconsistent streams, hinting at deepfakes or otherwise compromised streams.