ADAPT DMS - 3D Data Collection

Purpose of the Research?

This study is part of a collaborative research project between Xperi and the ADAPT-2 SFI research center.

The aim of this research is to build an Artificial Intelligence Platform (AIP) for a new generation of driver and occupant monitoring systems (DMS/OMS) for automobiles. These systems will become mandatory across the EU over the next few years and it is important that they are designed to be inclusive and can adapt to all members of the population. Such systems are an important component of the transition to fully autonomous driving and will improve vehicular safety by checking for drowsiness of the driver. They will also monitor driver awareness and cognition, managing the switching between manual and semi-autonomous driving. The AIP does not send any personal data to a cloud service, thus ensuring that any speech or video data captured within the vehicle remains secure.

In order to build algorithms for DMS/OMS AIP training data is required, in particular advanced 3D face and body models are used to simulate a range of human actions, gestures, expressions and activities. Data from these simulations are then used to build large 2D datasets that are used to train the AIP. It is important to have a diverse range of 3D models and Xperi is currently gathering data at multiple locations globally.

NUIG is assisting in collecting a representative sample of the resident Irish population to contribute to the training data for these systems. 

What type(s) of data will be collected?

The 3D scanner used in this data acquisition is a photogrammetric scanner. It collects a series of high-resolution photographs of people or objects placed inside the scanner and processes these to build a sophisticated 3D model of the participant/object. This model can be further processed in a specialized 3D software framework to rig and animate the model. This data is referred to as ‘research data’ in this document.

Xperi, our industry partner in this project, has developed the software to process and create these 3D models and uses additional algorithms to animate them to perform a range of human actions and gestures. In addition to photographic data, we may also request to acquire some speech data which will be used for the same purposes.  

A picture showing the physical form of the scanner is shown in Fig 1. below and Fig 2. shows a number of views of the final 3D model generated by the scanner from the 2D photographic images captured. 

How is your data stored, processed and utilized?

Models and raw image data will be stored and pseudo anonymized on secure internal servers at Xperi following TISAX* compliance. It will be processed by researchers at the company using advanced 3D software tools to build training datasets for Xperi AIP platforms. Your data will not be used in products developed by Xperi. It is only used to improve and test the algorithms that enable these products to perform their required functionality.

Your personal contact information will be stored separately and is only linked with the research data you provide today through a subject ID number, such as “subject X23-12345”. This process is known as pseudo-anonymization of data.

The research data is stored in digital form and incudes a geometric mesh comprising many separate shapes, with associated vertex locations, color, texture and light information. A sophisticated computer with advanced GPU is required to process and construct a 3D model from this data and render correspoinding 2D images. 

Some participant-specific data such as age-range, gender, ethnicity, height, weight, and other physical attributes may be saved with the research data but is not considered personally identifiable data due to the pseudo-anonymization process applied. We only use this data to ensure the diversity of the training data - that means that we have a good representative sample of difference ages, heights and other physical attributes that is consistent with the general population. This helps to ensure that the AI algorithms developed from this data are non-discriminative and can cover the widest possible range of variations in human appearance. 

Note that we are required to retain your contact information by GDPR regulations in case you later decide to withdraw your consent, or it becomes necessary to inform you in the unlikely event of a data breach. Therefore, we are required to keep the indirect link between research data and personal contact data under EU GDPR regulations.

The data and models derived from 3D scanner data will only be used for internal research at Xperi. No data, personal or anonymized, will be made publicly available.

The storage and management of your data complies with TISAX – a data security regime applied in the Automotive sector. This is compliant with GDPR and, in fact applies a more sophisticated security and data management approach than is required under GDPR.

TISAX® stands for Trusted Information Security Assessment Exchange, an information security standard that was developed exclusively for the requirements of the automotive industry. Originally established by the German Association of the Automotive Industry (VDA), the standard has long been a mainstay across Europe – TISAX® has been a registered trademark of the ENX Association, an organisation consisting of European automotive manufacturers and suppliers (https://portal.enx.com/en-US/TISAX/) 

What are the health & safety risks if I participate in this study?

There are little to no risks to your health, safety or well-being through participation in this data acquisition. Data is acquired using conventional DSLR cameras. These are similar to the camera in your smartphone or in a consumer digital camera. Flash lighting is used during data acquisition, but it is indirect flash and less intrusive than a direct flash used in a professional photography studio. 

However, you should not participate in this project if you:

1. are sensitive to or uncomfortable with flash photography *
2. have photo-sensitive epilepsy or equivalent photo-sensitivity
3. are COVID-19 positive , have symptoms or are a close-contact of a person who recently tested positive
4. are not fully compliant with HSE and university’s current recommendations for preventing the spread of COVID-19.
5. suffer from neuro-motor disorders (eg. Parkinson) that preclude holding a fixed body pose or keeping your eyes open for several seconds, during data acquisition
6. your height including shoes and headgear (if for religious or personal reasons it is not removable) lies outside 1.6m – 1.85m range.
7. are under 18 years of age.

* Note that the flashes in the 3D scanner will fire only once when capturing the images for a 3D scan. The flashes are never directed directly at the participant, but the light is bounced off the internal white cover of the 3D scanner. Over the course of the 3D model capturing session up to 3 scans may be performed for a participant.

** The program operates within the confined space of 3D scanner room, meaning it cannot be undertaken unless all involved are fully compliant with HSE current recommendations (https://www2.hse.ie/conditions/covid19/preventing-the-spread/protect-yourself-and-others/).

What are the data security & privacy risks if I participate in this study?

Your identity will not be linked directly with the image or speech data we collect, or the 3D model that is generated from this data. As mentioned above, your personal contact information will be stored separately and is only linked with the research data you provide through a participant ID number, such as “subject X23-12345”.

You are providing personal biometric data, more specifically facial image data, that will be used to build a complex 3D model that requires an advanced software environment to access or process. This model is not directly linked with your personal contact information provided in the consent form.

We will only use the research data for training and validating improved AIP algorithms. Data or models will not be directly incorporated into products or software distributed by us thus the primary risk is due to a data breach by 3rd parties or equivalent criminal behaviour.

In the event of a data breach, it is unlikely, but possible that both the research data and your contact information data are obtained by a 3rd party. If such an event were to occur there is a risk that this 3rd party could employ this data in a spoofing attack. However, it would be easier and more efficient for a criminal 3rd party to obtain personally identifiable facial image data from a social media channel or personal web page to achieve the same end. As your data is stored in a secure private corporate network under data management protocols that comply with TISAX there are lower risks of a 3^rd party data breach.  

Note that a detailed Data Protection Impact Assessment (DPIA) has been carried out for the 3D scanner used in this study in consultation with the Data Protection Office at NUIG and legal representatives at both NUIG and Xperi.

How will I benefit from participating in this study?

You are compensatedfor your time spent in reviewing the documentation associated with this study and in participating in the data acquisition process with a token 'all-for-one' voucher.  Aside from this there is no direct benefit to you as an individual. 

It is expected that information obtained during the study will be useful to improve the capabilities of new AI-enabled occupant monitoring systems and will accelerate the introdictuon of fully autonomous vehicles.

Who is responsible for executing this study?

Principal Investigators:

Prof. Peter Corcoran, College of Engineering & Informatics, NUIG.

Email: Peter.Corcoran@nuigalway.ie

Other Investigators:

Ms. Claudia Costache, College of Engineering & Informatics, NUIG.

Email: claudia.iancucostache@nuigalway.ie

Mr. Adrian Pasvante, Director in ML Data Team, Xperi.

Email: adrian.pasvante@xperi.com

For any privacy concerns, you may also reach Xperi directly at <XperiPrivacy@xperi.com>.