EventPointMesh: Human Mesh Recovery Solely from Event Point Clouds

How much can we infer about human shape using an event camera that only detects the pixel position where the luminance changed and its timestamp? This neuromorphic vision technology captures changes in pixel values at ultra-high speeds, regardless of the variations in environmental lighting brightness. Existing methods for human mesh recovery (HMR) from event data need to utilize intensity images captured with a generic frame-based camera, rendering them vulnerable to low-light conditions, energy/memory constraints, and privacy issues. In contrast, we explore the potential of solely utilizing event data to alleviate these issues and ascertain whether it offers adequate cues for HMR, as illustrated in Fig.1. This is a quite challenging task due to the substantially limited information ensuing from the absence of intensity images. To this end, we propose EventPointMesh, a framework which treats event data as a three-dimensional (3D) spatio-temporal point cloud for reconstructing the human mesh. By employing a coarse-to-fine pose feature extraction strategy, we extract both global features and local features. The local features are derived by processing the spatio-temporally dispersed event points into groups associated with individual body segments. This combination of global and local features allows the framework to achieve a more accurate HMR, capturing subtle differences in human movements. Experiments demonstrate that our method with only sparse event data outperforms baseline methods.

Fig. 2. Pipeline of our event-based HMR method, EventPointMesh. It consists of four modules: Base Module, Keypoints Module, Anchor Points, and SMPL Module (Sec. III-A).

Fig. 4. The EventPointMesh Dataset (EPMD) encompasses diverse actions from daily activities to sports motions. Subjects performed the actions multiple times in both well-lit and poorly-lit environments. EPMD comprises synchronized event data, intensity images, SMPL meshes, and optical MoCap data.

This method's ability to accurately recover 3D human meshes from event data, even in challenging lighting conditions, makes it particularly suitable for VR applications, enabling seamless and real-time human motion representation in low-light environments, dynamic scenes, or privacy-sensitive settings. Moreover, as this method supports online processing, it allows interactive applications such as projecting the motion of a user wearing an HMD onto 3D characters in real time, making it applicable to VR environments. The following videos demonstrate examples of motions estimated from event data using EventPointMesh, applied to 3D characters.