Forage: Understanding RAG-based Sensemaking for Community Conversations

Abstract

We introduce Forage, a RAG-based and LLM-augmented search engine, which we apply to the problem of sensemaking for community conversation data. We report on formative user studies introducing Forage to two distinct user groups: NPR journalists and municipal staff in the city of Durham, North Carolina. We taxonomize the query types users make with the tool, use cases that include synthesizing insights across conversations and finding content about a particular subject. We find that users tend to gravitate towards using the system for synthesis more than for pure search. We report on challenges and opportunities surfaced by performing sensemaking with an open-ended interface like Forage, such as the benefits of finding content quickly, but also the challenges users face interacting with a system in natural language. Insights from this formative study confirm the usefulness of Forage for sensemaking, but also make follow-up work, such as systematically evaluating system performance and developing appropriate design, urgent.