A team led by Artur Schneider and Prof. Dr. Ilka Diester from the Institute of Biology III and Prof. Dr. Thomas Brox from the Computer Science Department of the University of Friborg have developed a new system that records the 3D movement of freely nameable objects body points: “FreiPose” uses multiple cameras and a special network architecture to recognize defined body points and track them. The method allows researchers to exclude movement influences from certain parts of the body in their analysis that are not of interest. The group presents its new method in the journal Neuron.
Body points are reconstructed directly in 3D using different camera perspectives and an adapted network architecture
“We wondered if it was possible to link neural activity in the brain to the movements of different parts of the body when living things move freely and only perform spontaneous movements,” explains Diester. According to the researchers, two conditions must be met for this to happen: first, detailed 3D tracking of movements at different parts of the body is required; second, the body part of interest must be isolated from the influence of other body parts.
Scientists have created a system with “FreiPose” that reconstructs detailed postures and movements of individual body points directly in 3D using various synchronized camera perspectives. Body points are freely selected beforehand.
“New to FreiPose is the projection of image features learned from individual camera views into a common 3D space, where a network architecture is able to combine all the information to draw conclusions about the position of body points. . Due to its multi-view approach and native 3D reconstruction, FreiPose is particularly suitable for free-moving creatures in various environments, including those with obstacles,” says Brox.
“FreiPose” could be used for epileptic patients
To see the neural representations of the movements of different body parts, the researchers reduced the contributions of each body part and movement. “This strategy makes it possible to analyze the behavior in question, for example, only the movements of the hand. The method could be used, for example, with epileptic patients with implanted electrodes. If neural activity is measured to verify electrodes, FreiPose could be used to track patient movement and eliminate the influence of unwanted movement,” explains Diester.
“This method can be used to better study the behavior of living things because they can move completely freely and naturally,” Brox adds. “Once established, our system can be applied to large datasets, minimizing the workload.”
- Original publication: Schneider, A., Zimmermann, C., Alyahyay, M., Steenbergen, F., Brox, T., Diester, I. (2022): 3D pose estimation enables virtual head fixation in free-moving rats. In: Neurone. DO I: 10.1016/j.neuron.2022.04.019
- Ilka Diester’s research focuses on the neural bases of motor control and cognitive control as well as the interactions between the prefrontal and motor cortex.
- Diester is a spokesperson and member of the BrainLinks-BrainTools center and spokesperson for the “Intelligent Machine-Brain Interfacing Technology” research building at the University of Fribourg.
- Thomas Brox leads the Computer Vision Group of the Computer Science Department and is part of the Pôle d’Excellence CIBSS – Center for Integrative Studies of Biological Signaling and the BrainLinks-BrainTools Center. Brox’s research includes deep learning with a focus on learning visual representations, video analytics, and learned 3D representations.