EgoBalanceFormer：从自我中心姿态与VR原生运动预测人体平衡

文献信息

中文摘要

背景： 在沉浸式虚拟现实（VR）中，保持平衡对于安全交互十分重要，因为视觉-前庭冲突可能影响姿势稳定性。以往大多数基于VR的平衡分析主要依赖头戴式显示器（HMD）的运动信号，这些信号能捕捉头部运动，但对全身生物力学提供的信息有限。因此，仅凭VR传感数据直接预测生物力学平衡指标（如压力中心，CoP）仍然具有挑战性。方法：本文提出EgoBalanceFormer，一种多模态框架，通过将自我中心姿态重建作为一种身体感知的传感模态，并结合VR原生HMD运动信号，来预测VR交互过程中的人体平衡。与以往主要依赖HMD轨迹等稀疏设备运动的VR平衡方法不同，我们的方法利用自我中心视觉重建全身姿态，并推导出对平衡控制建模至关重要的生物力学线索，包括下肢运动学和质心（CoM）动力学。这些由姿态推导出的生物力学线索与VR原生HMD运动信号融合，并通过时间编码器处理以捕捉随时间变化的运动动态。我们进一步在下肢关节运动学、CoM动力学、旋转特征和HMD运动特征之间应用双向交叉注意力，以捕捉互补的生物力学关系。精炼后的多模态表征被融合，并使用带有可学习CoP查询的Transformer进行解码，以回归CoPX和CoPY。结果：我们将EgoBalanceFormer与多种机器学习和深度学习基线在CoP预测任务上进行比较。实验结果表明，该方法在不同参与者之间具有更高的准确性和更强的鲁棒性，证明与仅使用VR设备运动信号相比，自我中心姿态推导出的生物力学线索能显著增强平衡预测。结论：无

英文摘要

Maintaining balance is important for safe interaction in immersive virtual reality (VR), where visual-vestibular conflicts can affect postural stability. Most prior works on VR-based balance analysis rely primarily on motion signals from the head-mounted display (HMD), which capture head movement but provide only limited information about full-body biomechanics. As a result, directly predicting biomechanical balance indicators, such as the Center of Pressure (CoP), from VR sensing data remains challenging. In this paper, we present EgoBalanceFormer, a multimodal framework for predicting human balance during VR interaction by leveraging egocentric pose reconstruction as a body-aware sensing modality together with VR-native HMD motion signals. Unlike prior VR balance approaches that rely primarily on sparse device motion such as HMD trajectories, our approach uses egocentric vision to reconstruct full-body pose and derive biomechanical cues-including lower-body kinematics and center-of-mass (CoM) dynamics that are critical for modeling balance control. These pose-derived biomechanical cues are fused with VR-native HMD motion signals and processed through temporal encoders to capture motion dynamics over time. We further apply bidirectional cross-attention between lower-body joint kinematics, CoM dynamics, rotational features, and HMD motion features to capture complementary biomechanical relationships. The refined multimodal representations are fused and decoded using a transformer with a learnable CoP query to regress CoPX and CoPY . We evaluate EgoBalanceFormer against several machine learning and deep learning baselines for CoP prediction. Experimental results show improved accuracy and stronger robustness across participants, demonstrating that egocentric pose-derived biomechanical cues significantly enhance balance prediction compared with using VR-device motion signals alone.