Automatical Audiovisual Analysis

Automatically Extracting Audiovisual Features to Analyze Videos in Educational Research

Affiliation: TU Darmstadt

External PartnerGraduate School Teaching & Learning Processes, Universität Koblenz-Landau

Financial Support by the Funding Agency : Dec 2015 – Nov 2017

Project Staff

Project Description

A significant amount of time in research is spent to pre-process data for subsequent qualitative analysis. This is not different for researchers in the educational domain, especially if their data contains video recordings of classroom situations. In that case, the preparation phase typically involves speech transcription, videos sampling, format conversion and other time-intense tasks. The manual process might involve subjective analysis, such as judging students due to their (potentially problematic) personality, or rating the teachers’ interaction with the students in the classroom. Like that, the initial analysis creates a bottleneck for scaling the research task as the amount of data increases and also potentially introduces a bias in the subsequent analysis of the data.

The main goal of this project is to bootstrap the initial analysis of video recordings using state-of-the-art machine learning techniques built on features extracted from the audiovisual signal. Our data is targeted towards the identification of personality traits of students based on short recordings of experiments in a physics class. The project specific goals thus consist of 1) implementing feature extractors for audiovisual features and 2) evaluating the implemented software for automatic personality perception of students using the annotations of psychology researchers as a ground truth. The technology implemented in this project will be publicly available for further usage by other researchers from Digital Humanities.


Associated Software Products

Associated Research Areas


  • Burkhardt, Paul Michael: Automatic prediction of students’ personality, self-concept and intelligence. On-going Bachelor thesis. 2016

Further Information

A system for training end-to-end neural models to automatically recognize the motivational levels from students, using only their facial expressions as input, can be found here.