publications

2022

1. arXiv

   LyricJam Sonic: A Generative System for Real-Time Composition and Musical Improvisation

   Vechtomova, Olga, and Sahu, Gaurav

   2022

   Abs Paper

   Electronic music artists and sound designers have unique workflow practices that necessitate specialized approaches for developing music information retrieval and creativity support tools. Furthermore, electronic music instruments, such as modular synthesizers, have near-infinite possibilities for sound creation and can be combined to create unique and complex audio paths. The process of discovering interesting sounds is often serendipitous and impossible to replicate. For this reason, many musicians in electronic genres record audio output at all times while they work in the studio. Subsequently, it is difficult for artists to rediscover audio segments that might be suitable for use in their compositions from thousands of hours of recordings. In this paper, we describe LyricJam Sonic – a novel creative tool for musicians to rediscover their previous recordings, re-contextualize them with other recordings, and create original live music compositions in real-time. A bi-modal AI-driven approach uses generated lyric lines to find matching audio clips from the artist’s past studio recordings, and uses them to generate new lyric lines, which in turn are used to find other clips, thus creating a continuous and evolving stream of music and lyrics. The intent is to keep the artists in a state of creative flow conducive to music creation rather than taking them into an analytical/critical state of deliberately searching for past audio segments. The system can run in either a fully autonomous mode without user input, or in a live performance mode, where the artist plays live music, while the system ’listens’ and creates a continuous stream of music and lyrics in response.

2. NLP4ConvAI

   Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

   Sahu, Gaurav, Rodriguez, Pau, Laradji, Issam, Atighehchian, Parmida, Vazquez, David, and Bahdanau, Dzmitry

   In Proceedings of the 4th Workshop on NLP for Conversational AI 2022

   Abs Paper BibTeX

   Data augmentation is a widely employed technique to alleviate the problem of data scarcity. In this work, we propose a prompting-based approach to generate labelled training data for intent classification with off-the-shelf language models (LMs) such as GPT-3. An advantage of this method is that no task-specific LM-fine-tuning for data generation is required; hence the method requires no hyper parameter tuning and is applicable even when the available training data is very scarce. We evaluate the proposed method in a few-shot setting on four diverse intent classification tasks. We find that GPT-generated data significantly boosts the performance of intent classifiers when intents in consideration are sufficiently distinct from each other. In tasks with semantically close intents, we observe that the generated data is less helpful. Our analysis shows that this is because GPT often generates utterances that belong to a closely-related intent instead of the desired one. We present preliminary evidence that a prompting-based GPT classifier could be helpful in filtering the generated data to enhance its quality.

3. JDSA

   Personalized multi-faceted trust modeling to determine trust links in social media and its potential for misinformation management

   Parmentier, Alexandre, Cohen, Robin, Ma, Xueguang, Sahu, Gaurav, and Chen, Queenie

   In International Journal of Data Science and Analytics 2022

   Abs Paper BibTeX

   In this paper, we present an approach for predicting trust links between peers in social media, one that is grounded in the artificial intelligence area of multiagent trust modeling. In particular, we propose a data-driven multi-faceted trust modeling which incorporates many distinct features for a comprehensive analysis. We focus on demonstrating how clustering of similar users enables a critical new functionality: supporting more personalized, and thus more accurate predictions for users. Illustrated in a trust-aware item recommendation task, we evaluate the proposed framework in the context of a large Yelp dataset. We then discuss how improving the detection of trusted relationships in social media can assist in supporting online users in their battle against the spread of misinformation and rumours, within a social networking environment which has recently exploded in popularity. We conclude with a reflection on a particularly vulnerable user base, older adults, in order to illustrate the value of reasoning about groups of users, looking to some future directions for integrating known preferences with insights gained through data analysis.

2021

1. ICCC

   LyricJam: A system for generating lyrics for live instrumental music

   Vechtomova, Olga, Sahu, Gaurav, and Kumar, Dhruv

   In Proceedings of ICCC 2021

   Abs Paper BibTeX

   We describe a real-time system that receives a live audio stream from a jam session and generates lyric lines that are congruent with the live music being played. Two novel approaches are proposed to align the learned latent spaces of audio and text representations that allow the system to generate novel lyric lines matching live instrumental music. One approach is based on adversarial alignment of latent representations of audio and lyrics, while the other approach learns to transfer the topology from the music latent space to the lyric latent space. A user study with music artists using the system showed that the system was useful not only in lyric composition, but also encouraged the artists to improvise and find new musical expressions. Another user study demonstrated that users preferred the lines generated using the proposed methods to the lines generated by a baseline model.

2. AAMAS

   Towards A Multi-agent System for Online Hate Speech Detection

   Sahu, Gaurav, Cohen, Robin, and Vechtomova, Olga

   In Proceedings of the 2nd International Workshop on Autonomous Agents for Social Good (AASG), AAMAS 2021

   Abs Paper Poster BibTeX

   This paper envisions a multi-agent system for detecting the presence of hate speech in online social media platforms such as Twitter and Facebook. We introduce a novel framework employing deep learning techniques to coordinate the channels of textual and image processing. Our experimental results aim to demonstrate the effectiveness of our methods for classifying online content, training the proposed neural network model to effectively detect hateful instances in the input. We conclude with a discussion of how our system may be of use to provide recommendations to users who are managing online social networks, showcasing the immense potential of intelligent multi-agent systems towards delivering social good.

3. EACL

   Adaptive Fusion Techniques for Multimodal Data

   Sahu, Gaurav, and Vechtomova, Olga

   In Proceedings of EACL 2021

   Abs Paper Poster Talk BibTeX

   Effective fusion of data from multiple modalities, such as video, speech, and text, is challenging due to the heterogeneous nature of multimodal data. In this paper, we propose adaptive fusion techniques that aim to model context from different modalities effectively. Instead of defining a deterministic fusion operation, such as concatenation, for the network, we let the network decide “how” to combine a given set of multimodal features more effectively. We propose two networks: 1) Auto-Fusion, which learns to compress information from different modalities while preserving the context, and 2) GAN-Fusion, which regularizes the learned latent space given context from complementing modalities. A quantitative evaluation on the tasks of multimodal machine translation and emotion recognition suggests that our lightweight, adaptive networks can better model context from other modalities than existing methods, many of which employ massive transformer-based networks.

2020

1. COLING

   Adversarial Learning on the Latent Space for Diverse Dialog Generation

   Khan, Kashif*, Sahu, Gaurav*, Balasubramanian, Vikash*, Mou, Lili, and Vechtomova, Olga

   In Proceedings of COLING 2020

   Abs Paper Poster BibTeX

   Generating relevant responses in a dialog is challenging, and requires not only proper modeling of context in the conversation, but also being able to generate fluent sentences during inference. In this paper, we propose a two-step framework based on generative adversarial nets for generating conditioned responses. Our model first learns a meaningful representation of sentences by autoencoding, and then learns to map an input query to the response representation, which is in turn decoded as a response sentence. Both quantitative and qualitative evaluations show that our model generates more fluent, relevant, and diverse responses than existing state-of-the-art methods.

2. ISMIR

   Generation of lyrics lines conditioned on music audio clips

   Vechtomova, Olga, Sahu, Gaurav, and Kumar, Dhruv

   In Proceedings of the First Workshop on NLP for Music and Audio (NLP4MusA) at ISMIR 2020

   Abs Paper BibTeX

   We present a system for generating novel lyrics lines conditioned on music audio. A bimodal neural network model learns to generate lines conditioned on any given short audio clip. The model consists of a spectrogram variational autoencoder (VAE) and a text VAE. Both automatic and human evaluations demonstrate effectiveness of our model in generating lines that have an emotional impact matching a given audio clip. The system is intended to serve as a creativity tool for songwriters.

3. UWSpace

   Adaptive Fusion Techniques for Effective Multimodal Deep Learning

   Sahu, Gaurav

   2020

   Abs Paper BibTeX

   Effective fusion of data from multiple modalities, such as video, speech, and text, is a challenging task due to the heterogeneous nature of multimodal data. In this work, we propose fusion techniques that aim to model context from different modalities effectively. Instead of defining a deterministic fusion operation, such as concatenation, for the network, we let the network decide “how” to combine given multimodal features more effectively. We propose two networks: 1) Auto-Fusion network, which aims to compress information from different modalities while preserving the context, and 2) GAN-Fusion, which regularizes the learned latent space given context from complementing modalities. A quantitative evaluation on the tasks of multimodal machine translation and emotion recognition suggests that our adaptive networks can better model context from other modalities than all existing methods, many of which employ massive transformer-based networks.

4. CE Journal

   Digital Literacy for Secondary School Students: Using Computer Technology to Educate about Credibility of Content Online

   Cohen, Robin, Parmentier, Alexandre, Melo, Glaucia, Sahu, Gaurav, Annamalai, Aswin, Chi, Sheldon, Clokie, Trevor, Farrag, Amir, Naik, Abdul, Naseem, Syed, and others,

   In Creative Education Journal special issue on Education and Information Technology 2020

   Abs Paper BibTeX

   This paper presents an approach to educate secondary school students in the province of Ontario about the credibility of online content. The critical focus here is on integrating computer technology into the teaching of the topic; how to introduce the material in classroom settings with respect to the current curriculum is also outlined. Contrast with an existing proposal for digital literacy developed by historians at Stanford University is provided at the outset. In all, the value of appealing to the current digital experiences of students, when revealing the potential for misinformation, is the critical message. Exploration of social media environments popular with youth and opportunities for game-based quizzes for interactive engagement are both advocated.

2019

1. arXiv

   Multimodal speech emotion recognition and ambiguity resolution

   Sahu, Gaurav

   arXiv preprint arXiv:1904.06022 2019

   Abs Paper BibTeX

   Identifying emotion from speech is a non-trivial task pertaining to the ambiguous definition of emotion itself. In this work, we adopt a feature-engineering based approach to tackle the task of speech emotion recognition. Formalizing our problem as a multi-class classification problem, we compare the performance of two categories of models. For both, we extract eight hand-crafted features from the audio signal. In the first approach, the extracted features are used to train six traditional machine learning classifiers, whereas the second approach is based on deep learning wherein a baseline feed-forward neural network and an LSTM-based classifier are trained over the same features. In order to resolve ambiguity in communication, we also include features from the text domain. We report accuracy, f-score, precision, and recall for the different experiment settings we evaluated our models in. Overall, we show that lighter machine learning based models trained over a few hand-crafted features are able to achieve performance comparable to the current deep learning based state-of-the-art method for emotion recognition.

2018

1. EMNLP

   Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit

   Krishna, Amrith, Santra, Bishal, Bandaru, Sasi Prasanth, Sahu, Gaurav, Sharma, Vishnu Dutt, Satuluri, Pavankumar, and Goyal, Pawan

   In Proceedings of EMNLP 2018

   Abs Paper Supp BibTeX

   The configurational information in sentences of a free word order language such as Sanskrit is of limited use. Thus, the context of the entire sentence will be desirable even for basic processing tasks such as word segmentation. We propose a structured prediction framework that jointly solves the word segmentation and morphological tagging tasks in Sanskrit. We build an energy based model where we adopt approaches generally employed in graph based parsing techniques (McDonald et al., 2005a; Carreras, 2007). Our model outperforms the state of the art with an F-Score of 96.92 (percentage improvement of 7.06%) while using less than one tenth of the task-specific training data. We find that the use of a graph based approach instead of a traditional lattice-based sequential labelling approach leads to a percentage gain of 12.6% in F-Score for the segmentation task.