Criminal charges against safety driver highlight risk of “automation complacency”

Those following the development of driverless vehicle technology may remember a tragic incident in March 2018, when a self-driving Uber test vehicle collided with a pedestrian, resulting in her death. Following in-depth analysis by the US National Transportation Safety Board (NTSB), it seems that the safety driver has now been charged with negligent homicide. Inward-facing camera footage and app data showed that the safety driver had been viewing a TV program on her mobile phone at the time of the incident.

The report concludes that, had the safety driver been paying attention, she would have been able to avoid or mitigate the collision. It notes that:

“an attentive driver would have been able stop the SUV before the impact location if applying maximum braking within 1.9 seconds of entering the straight section, or within 3.5 seconds of the ADS detecting the pedestrian.”

The NTSB noted that the vehicle’s software had not properly identified the pedestrian. It also criticised Uber for inadequate oversight of its safety drivers. Inward-facing camera footage was rarely reviewed. More vigilance here could have identified the need for great stringency and improved training. The NTSB noted that “maintaining operator attentiveness in the face of likely automation complacency” is an expected challenge.

The vehicle’s emergency braking system was temporarily suppressed at the time of the incident. This was a technical measure introduced “because the situation exceeded the ADS response design specifications for avoiding the collision (it required deceleration greater than 0.71 g to avoid the crash)”. Although this made sense in an experimental system, when the vehicle was under testing on public roads this reduced safety redundancy meaning that the level of risk was increased. Testing of the standard retail version of the Volvo XC90 that Uber was using indicated that it could have avoided or mitigated the incident with emergency braking enabled.

Two issues:

• driver focus. This incident shows how easy it is for a trained and monitored safety driver to be distracted from monitoring the vehicle. This is likely to be even more of a problem for members of the public.
• redundancy. Uber was criticised by the NTSB for suppressing an automated braking functionality in the test vehicle. Had this been operational, they concluded, the accident would have been less severe or avoided altogether.

In our view, vehicle developers and regulators must not rely on individual drivers to provide a fall-back emergency response. We consider this to be a particular danger as automation is gradually increased in vehicles, so that drivers learn to rely on it more and more. Failsafe emergency response features should be an essential element of increasing automation.

The UK Government is currently consulting on automated lane-keeping technology (Safe use of Automated Lane Keeping System on GB motorways: call for evidence). The consultation explores the issues arising from partial automation. While autonomous vehicles offer safety gains in the medium term, there are risks involved along the way. Getting this right is vital to maintain public trust in the driverless concept. The consultation is open for responses until 27 October.

Find out more about our automotive and technology services, and our thought leadership on autonomous vehicles.