publications

2025

1. Preprint

   Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies

   Ziyi Xuan, Yiwen Wu, Xuhai Xu, and 3 more authors

   2025

   HTML
2. SenSys’25 Demo

   Demo Abstract: GIDEA: Generative AI-Powered Interactive Design and Evaluation Platform for Assistant Agent Research

   Ziyi Xuan, Yiwen Wu, and Yu Yang

   In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, UC Irvine Student Center., Irvine, CA, USA, 2025

   Abs DOI

   Conducting human-computer interaction (HCI) experiments often requires extensive manual effort, including configuring environments, recruiting participants, and recording interactions. We introduce GIDEA, a generative AI-powered interactive design and evaluation platform for assistant agents to streamline and accelerate HCI research. Our platform employs a three-role interaction pipeline, where researchers define experiments, large language model-driven avatars simulated participants, and a smart assistant agent moderates interactions. This pipeline dynamically generates interaction scenarios, avatar profiles, and adaptive responses based on researcher input. By integrating with Unity, GIDEA enables real-time monitoring and control over simulated experiments, providing researchers with an interactive and adaptable evaluation environment. Through the replication of real-world case studies, we demonstrate that GIDEA reduces the time and effort required for HCI experiments while producing results that align with real studies. This capability has the potential to revolutionize HCI research by transforming traditionally lengthy and labor-intensive processes into a highly efficient, scalable, and adaptive methodology, accelerating innovation and broadening experimental possibilities.

3. SenSys’25

   Multi-Modal Dataset Across Exertion Levels: Capturing Post-Exercise Speech, Breathing, and Phonocardiogram

   Jingping Nie, Yuang Fan, Minghui Zhao, and 4 more authors

   In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, UC Irvine Student Center., Irvine, CA, USA, 2025

   Abs DOI

   Cardio exercise elevates both heart rate and respiration rate, resulting in distinct physiological changes that affect speech patterns, pitch, breathing sounds, and heart sounds. These variations, which occur post-exercise, are influenced by factors such as exercise intensity and individual fitness levels. A comprehensive audio dataset is critically needed to capture post-exercise physiological changes, as existing datasets focus mainly on resting speech, breathing, and heart sounds, neglecting the dynamic shifts following physical exertion. Current datasets fail to capture unique post-exercise variations like speech disfluencies, altered breathing patterns, and variable heart sound intensities, limiting model generalizability to post-exercise conditions. To address this gap, we recruited 59 subjects from diverse backgrounds to engage in cardio exercise, specifically running, reaching varied exertion levels to produce a rich dataset. Our dataset includes 250 sessions totaling 143 minutes of structured reading, 47 minutes of spontaneous speech, 71 minutes of breathing sounds, and 62.5 minutes of phonocardiogram (PCG) recordings. We designed and deployed preliminary case studies to show that speech changes post-cardio could serve as an indicator of exertion level. We envision this dataset as a foundational resource for designing models in speech and cardiorespiratory monitoring that are resilient to the physiological shifts induced by exercise. This dataset could advance natural language processing (NLP) applications, mobile health, and wearable sensing technologies by enabling resilient and accurate physiological monitoring in real-world conditions.

2024

1. MobiCom’24 Poster

   Real-Time Non-Contact Estimation of Running Metrics on Treadmills using Smartphones

   Jingping Nie, Yuang Fan, Ziyi Xuan, and 2 more authors

   In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, Washington D.C., DC, USA, 2024

   Abs DOI

   Over half a trillion recreational runners worldwide engage in running for psychological, health, and social benefits. Running metrics are essential for motivation, goal setting, performance improvement, health management, and injury prevention. Although wearable devices like fitness trackers and smartwatches offer various metrics, they often perform poorly on treadmills and can be uncomfortable or restrictive. In this work, we propose a non-contact, real-time, smartphone-based approach to estimate running metrics, including cadence, ground contact time (GCT), and balance, using the sound produced during treadmill running. In collaboration with a licensed running coach, we recruited over 50 subjects with varying levels of running expertise. We collected treadmill running sounds and ground-truth running metrics in different environments. We designed and developed a multi-task learning (MTL) machine learning mobile system to capture the treadmill running sounds and estimate running metrics in situ. Our proposed method shows comparable accuracy in estimating running metrics to commercial off-the-shelf (COTS) wearable devices.

2023

1. IASA’23

   CaNRun: Non-Contact, Acoustic-based Cadence Estimation on Treadmills using Smartphones

   Ziyi Xuan, Ming Liu, Jingping Nie, and 3 more authors

   In Proceedings of Cyber-Physical Systems and Internet of Things Week 2023, San Antonio, TX, USA, 2023

   Abs DOI

   Running with a consistent cadence (number of steps per minute) is important for runners to help reduce risk of injury, improve running form, and enhance overall bio-mechanical efficiency. We introduce CaNRun, a non-contact and acoustic-based system that uses sound captured from a mobile device placed on a treadmill to predict and report running cadence. CaNRun obviates the need for runners to utilize wearable devices or carry a mobile device on their body while running on a treadmill. CaNRun leverages a long short-term memory (LSTM) network to extract steps observed from the microphone to robustly estimate cadence. Through an 8-person study, we demonstrate that CaNRun achieves cadence detection accuracy without calibration for individual users, which is comparable to the accuracy of the Apple Watch despite being non-contact.