Technology and methods

The BamLiD multi-sensory technology and its methodology at a glance

Extracting interpretable information from diverse data is a fundamental task in the Living Lab. What kind of behaviour is a person currently performing, what behavioural intention is being realised, and what emotional state is the person in? To ascertain this, hybrid artificial intelligence approaches combining machine learning and knowledge-based inference are used.

It is essential that multimodal derivation is possible while the person is moving freely in the BamLiD so that everyday activities and actions can be carried out unhindered. This multimodal derivation takes place via recordings:

1. Facial expressions

  • BamLiD is equipped with 12 (mobile) cameras so that lateral and frontal images of the face are assured at all times.
  • The cameras record synced videos (VideoSyncPro by Mangold International).
  • Facial reactions are manually analysed offline using the Facial Action Coding System.
  • Automatic facial expression recognition systems (Face Reader, Pain Face Reader, Open Face) are also used to analyse facial expression reactions.
  • Parameters: e.g. type of action unit (AU) shown, AU frequency, AU intensity, AU on and offset, AU duration

2. Voice

  • In BamLiD, 5 gooseneck microphones ensure optimal recording quality from any position in the room.
  • Audio is recorded in sync with the video recordings (VideoSyncPro by Mangold International).
  • The phonetic analyses of the audio tracks are carried out offline with the ProsodyPro program. 
  • Parameters: e.g. average fundamental frequency of phonation, frequency variation of phonation, average intensity and roughness

3. Gait analyses

  • BamLiD is fitted with an intelligent sensor floor (Smart Floor) (Fraunhofer IGD, Darmstadt). Sensors installed in the floor enable the location of a person to be determined with a high degree of accuracy based on passive electric field measurements.
  • Gait analyses are performed offline.
  • Parameters: e.g. speed, number of steps, step length, variance of speed/step length, asymmetry between left and right

4. Mobile biosignal recorder

  • BamLiD has a variety of mobile sensors (BioNomadix Wireless Wearable Physiology by BIOPAC, Biosignal Recorder NeXus-10 MKII by Hasomed/MindMedia) for recording blood pressure, heart rate, skin conductivity and actigraphy (motor activation).
  • These analyses are also performed offline. 
  • Parameters: e.g. heart rate, heart rate variability, sympathetic arousal, agitation/restlessness

5. Monitoring, trigger and storage technology in the control room

  • All data streams converge in the control room.
  • Trigger circuits ensure synchronous evaluation of the multi-sensor data.
  • The multi-sensor data is collected in the control room and camera settings are configured.

6. Lighting system

  • BamLiD is fitted with 42 luminaires that can be controlled individually.
  • This allows targeted control of light brightness and colour temperature (up to 3,000 lux).

7. Controlling ambient conditions

  • Temperature: BamLiD is air-conditioned so that temperatures can be monitored and set to the desired level.
  • Noise/noise level: BamLiD is equipped with loudspeakers so that any background noise can be played.