Manaswi Saha

Knowledge gaps often arise during communication due to diverse backgrounds, knowledge bases, and vocabularies. With recent LLM developments, providing real-time knowledge support is increasingly viable, but is challenging due to shared and individual cognitive limitations (e.g., attention, memory, and comprehension) and the difficulty in understanding the user’s context and internal knowledge. To address these challenges, we explore the key question of understanding how people want to receive real-time knowledge support. We built StopGap—a prototype that provides real-time knowledge support for explaining jargon words in videos—to conduct a design probe study (N=24) that explored multiple visual knowledge representation formats. Our study revealed individual differences in preferred representations and highlighted the importance of user agency, personalization, and mixed-initiative assistance. Based on our findings, we map out six key design dimensions for real-time LLM knowledge support systems and offer insights for future research in this space.

Digital media platforms (e.g., social media, science blogs) offer opportunities to communicate scientific content to general audiences at scale. However, these audiences vary in their scientific expertise, literacy levels, and personal backgrounds, making effective science communication challenging. To address this challenge, we designed TranSlider, an AI-powered tool that generates personalized translations of scientific text based on individual user profiles (e.g., hobbies, location, and education). Our tool features an interactive slider that allows users to steer the degree of personalization from 0 (weakly relatable) to 100 (strongly relatable), leveraging LLMs to generate the translations with given degrees. Through an exploratory study with 15 participants, we investigated both the utility of these AI-personalized translations and how interactive reading features influenced users’ understanding and reading experiences. We found that participants who preferred higher degrees of personalization appreciated the relatable and contextual translations, while those who preferred lower degrees valued concise translations with subtle contextualization. Furthermore, participants reported the compounding effect of multiple translations on their understanding of scientific content. Given these findings, we discuss several implications of AI-personalized translation tools in facilitating communication in collaborative contexts.

Multimodal large language models have enabled a new generation of Conversational Agents (CA), leveraging language structure in human discourse to encode-decode multimedia formats (e.g., video-to-audio). These next-generation CAs can be useful in task guidance scenarios, where the user’s attention space is limited and verbal instructions can be overwhelming. In this paper, we explore the role of non-verbal conversational cues in identifying and recovering from errors while performing various assembly tasks. Findings from an exploratory Wizard-of-Oz study (N=8) indicate individual differences and preferences for auditory guidance. Combining these initial findings with our early exploration of the task monitoring system, we discuss implications for the emerging area of situated multimodal CAs for physical task guidance, where conversational interactions are based on inputting visual task actions and generating auditory feedback.