Ziyi (Selina) Xuan

Conducting human-computer interaction (HCI) experiments often requires extensive manual effort, including configuring environments, recruiting participants, and recording interactions. We introduce GIDEA, a generative AI-powered interactive design and evaluation platform for assistant agents to streamline and accelerate HCI research. Our platform employs a three-role interaction pipeline, where researchers define experiments, large language model-driven avatars simulated participants, and a smart assistant agent moderates interactions. This pipeline dynamically generates interaction scenarios, avatar profiles, and adaptive responses based on researcher input. By integrating with Unity, GIDEA enables real-time monitoring and control over simulated experiments, providing researchers with an interactive and adaptable evaluation environment. Through the replication of real-world case studies, we demonstrate that GIDEA reduces the time and effort required for HCI experiments while producing results that align with real studies. This capability has the potential to revolutionize HCI research by transforming traditionally lengthy and labor-intensive processes into a highly efficient, scalable, and adaptive methodology, accelerating innovation and broadening experimental possibilities.

Cardio exercise elevates both heart rate and respiration rate, resulting in distinct physiological changes that affect speech patterns, pitch, breathing sounds, and heart sounds. These variations, which occur post-exercise, are influenced by factors such as exercise intensity and individual fitness levels. A comprehensive audio dataset is critically needed to capture post-exercise physiological changes, as existing datasets focus mainly on resting speech, breathing, and heart sounds, neglecting the dynamic shifts following physical exertion. Current datasets fail to capture unique post-exercise variations like speech disfluencies, altered breathing patterns, and variable heart sound intensities, limiting model generalizability to post-exercise conditions. To address this gap, we recruited 59 subjects from diverse backgrounds to engage in cardio exercise, specifically running, reaching varied exertion levels to produce a rich dataset. Our dataset includes 250 sessions totaling 143 minutes of structured reading, 47 minutes of spontaneous speech, 71 minutes of breathing sounds, and 62.5 minutes of phonocardiogram (PCG) recordings. We designed and deployed preliminary case studies to show that speech changes post-cardio could serve as an indicator of exertion level. We envision this dataset as a foundational resource for designing models in speech and cardiorespiratory monitoring that are resilient to the physiological shifts induced by exercise. This dataset could advance natural language processing (NLP) applications, mobile health, and wearable sensing technologies by enabling resilient and accurate physiological monitoring in real-world conditions.