Eye Tracking in Automotive use cases

While the general use of eye tracking systems is definitely interesting and leads to new and highly important insight, the main focus here is the automotive use. As mentioned in the introduction already, at least 90 % of the information a driver uses to safely steer and control the driven vehicle is obtained through the visual apparatus. Therefore, understanding what drivers look at and what they can perceive when driving is an obvious choice when inves-tigating driver behaviour. Additionally, this enables researchers and engineers to investigate or verify possible benefits with different driver assistance systems. Since headlamps are the main light source and are developed to help the driver to orientate himself and detect ob-stacles at night, using eye tracking systems in order to evaluate headlamp systems has been common in the community over the last couple of years.

The first remarkable study, in which eye tracking is used to identify driver’s performance is found in 1972, when Mourant used an eye tracker to identify the differences in orien-tation behaviour of drivers with different driving experiences. For this study, two cameras are used. One camera is set up in a way, that through filming a mirror, the driver’s eyes are recorded. The described cornea reflection method to calculate the gaze direction is used.

The secondary camera was equipped with a wide angle objective in order to record the road scene. Both cameras are synchronized and the gaze direction is marked on the road scene as a dot. The complete test setup is shown in figure4.13.

64 related work

Figure4.13– Test setup in order to monitor the driver’s gaze while recording the road scene (1972) to find differences in orientation behaviour between experienced and unexperienced drivers [129].

To reach the highest possible gap in driving experience between the test groups, the in-experienced drivers were test subjects who were actually doing their driver’s licence at the time and performed some of their practice drives during the study. The main differences were found in terms of the scanning patterns. More experienced drivers tend to use a wider area in which they search for possible hazards (±^{13° vs}±6° for novice drivers). Furthermore, they orientate themselves further away (1.5° further up) - closer to the horizon - and about 4°

further to the left than less experienced drivers. [129]

To further investigate the orientation behaviour of drivers, especially in cornering situations, Landsimultaneously recorded the horizontal gaze angle as well as the steering angle. Sim-ilar to the work of Mourant, a video based eye tracking system was used to measure both, the driver’s gaze as well as the road in front of the vehicle. Again, the calculated gaze direc-tion is transferred onto the recorded video using a computer based eye model to calculate the gaze direction depending on the pupillary direction. While this study only involved three test subjects, the findings indicate, that drivers use the tangent point of corners for their main orientation. However, it is also recorded, that the drivers tend to search besides the road and only frequently return back to the tangent point for orientation - the frequency in which this is done is dependent on the driver. [130]

The first study, that is directly on the same topic as this presented thesis is from 1998 by Damaskyand focuses on the influence of the light distribution of headlamps on driver’s fixa-tion behaviour at nighttime [131]. The work is divided in two parts. In the first part, different types of headlamps and the driver’s fixation behaviour with these headlamps were analysed.

The second part focused on the influence of the headlamps on the driving behaviour and the safety feeling of the drivers. For this, different headlamp systems were mounted on the testing vehicle using a metal rack. Different headlamps, including the tungsten halogen(H4, H7) andHID(D2R) headlamps with different reflector types were used and the testing vehicle

4.3 eye tracking 65 was set up with a commercial remote eye tracking system. The tests were performed on real roads with low levels of street illumination and each test drive (per headlamp) lasted about 4 h leading over different types of roads including country roads, motorways and federal highways. To identify differences in gaze-/orientation behaviour, areas of identical attention and duration are calculated. As an example, figure 4.14a shows the presented data for a halogen headlamp of the type H4 and figure 4.14b shows the same calculation for the H7 headlamp. The iso-areas of attention are shown by yellow lines and theoretical cut-off of the used headlamps is marked by the red line.

(a) (b)

Figure4.14–(a)Shows the iso areas of duration and attention (yellow) for driving with an H4headlamp for one test person and theoretical cut off (red line).(b)shows the same data recorded for an H7headlamp.[131]

The influence of the headlamp can clearly be seen in this example. For the H4headlamp, which has a higher emphasis on close-range illumination, the vision of the driver shifts closer to the vehicle. For theHID headlamps, the opposite is found. Since the usedHIDheadlamps deliver a lot of light close to the cut-off line, the 90 % iso-area of all gazes is lifted towards the cut-off and widened significantly. However, no significant influence of the headlamps on the driving performance is found with the recorded data. [131]

Brückmanninvestigated a similar case as Damaskywhere the influence from different head-lamp systems on gaze behaviour is tested [87]. While his findings in general reveal the same results as Damasky, he added new headlamp types and most importantly, determined the difference between the gaze behaviour at night compared to the gaze behaviour during the day. In order to investigate the orientation behaviour and the information gathering by the drivers, no instructions such as directions, but only the final destination, were given to the subjects. His findings regarding the differences between day and night show, that during the night the gaze distribution is significantly narrower than the distribution recorded during the day. He associates this with the limited information that is available during the night and concludes, that the driver only looks, where light is projected to by the headlamps. These re-sults are coherent with the findings regarding the different headlamp types he investigated and are similar to what Damaskypresented.

A completely different use case for remote eye tracking was developed by Jiwho used differ-ent characteristics of human facial features, including gaze oridiffer-entation, eye lid opening and blinking, or rather eye lid moving in general, to identify and objectively measure driver’s

66 related work

vigilance. For this, a setup including two remote cameras was developed and tested in a simulator. His test setup is shown in figure4.15aand4.15b.

(a) (b)

Figure4.15–(a)shows the camera placement in the vehicle simulator with two cameras set left and right of the optical axis of a potential driver.(b)Shows the camera setup for each of the two used cameras including the infra-red light source and the computer used to store and evaluate the data. [132]

To supply the cameras with an adequate amount of light during all simulated situations, day and night, infrared lighting is added to the system and the cameras are set up with an infrared bandpass filter. The infrared light sources are placed around the cameras to allow for bright pupil tracking. The main part of this work focuses on the used algorithm and the track-ing performances of the setup, which will not be discussed here, since in the presented thesis, a state-of-the-art commercial eye tracking system is used and no new algorithms regarding the eye tracking are developed. However, Jialso simulated drowsy driving and measured the gaze angles resulting in a significantly lowered gaze direction for tired drivers. [132]

Diem tackles a more general approach on the topic of gaze behaviour during night. In a larger scale than the previously described studies, he recorded gaze behaviour during auto-motive driving in real life situations during the day and the night. He then split the data sets according to different road types like urban, country road and motorway driving. These data sets were then split into further groups like single lane country roads vs. multi-lane country roads. To be able to measure this enormous data set, Diemdid not only develop his own eye tracking system that suited his needs for automotive use, but similar to the other publications presented here, recorded the road ahead of the vehicle as well. He then manually evaluated the data. His findings are vast but the most important findings are shortly summarized here.

The main difference between daytime driving and nighttime driving is, that during the day, Diem finds a much narrower field in which the driver orientates himself as shown by figure 4.16

4.3 eye tracking 67

Figure4.16– Gaze behaviour recorded by Diemon country roads for daytime driving (left) and nighttime driving (right) [4].

Differences between country roads and motorways are small but noticeable, however driv-ing in urban areas leads to a (compared to the other sets) very wide orientation behaviour.

Furthermore, influence by the number of available lanes is found, where different accumula-tion points are formed accoding to the number of available lanes, the quality of road mark-ings and for the brightness of the road surface, where darker roads lead to a wider, more searching gaze distribution. [4]

With the introduction of adaptive bending light, the research of the fixation distribution when driving through bends needed to be renewed. To investigate the benefit, respectively the influence of dynamic bending light on the gaze distribution, Shibataused an ^HID pro-jection system, that allows swivelling of±19° in both directions for both headlamps. The car was equipped with a remote eye tracking system and the gaze behaviour was recorded for driving through three different bends with radii of 65 m, 130 m and 210 m on country roads.

To investigate the effect of the bending light in larger detail, the headlamp was modified, so that four different modes were available: Both headlamps swivel, only the inner headlamp swivels, both headlamps swivel but the outer headlamp swivels only 50 % of the angle of the inner headlamp and both headlamps swivel with the outer headlamp swivelling 75 % of the inner headlamp. The results for the four extreme settings are shown in the figures4.17a to 4.17d.

68 related work

(a) (b)

(c) (d)

Figure4.17– Fixation areas (50 % area marked in blue, 90 % area marked in pink), and point of most gazes (red dot) as recorded (a)during the day and at night with (b) both headlamps fixed,(c) one headlamp swivelling,(d)both headlamps swivelling parallel [133].

Here the fixation areas are shown for 4.17a during the day, 4.17b with both headlamps fixed,4.17cone headlamp swivelling,4.17dboth headlamps swivelling parallel. The red dots marks the point with the highest amount of fixations, the blue line marks the 90 % iso-line and the pink line marks the 50 % area of fixations and all registered fixation points are shown in green circles. The findings show, that with theAFS, the fixations are more concentrated than during the day, which is explained by Shibataby the limited and more concentrated amount of light that is available in the areas where it is most needed. Furthermore, the data shows, that curves can be divided into three sections: entering a curve, the curve itself and exiting the curve and all three of the curve segments can be correlated with the gaze behaviour. [133]

Schulz uses a commercially available eye tracking system in combination with a scene cam-era to measure driver’s gaze behaviour and their orientation points when driving on country roads. While this research does not involve nighttime driving, the goal was to measure the gaze distribution in dependence of the available viewing distance to optimize street renewal and to allow drivers a comfortable driving experience with a high safety impression. The ex-periment was divided in three sections. In the first section, the participants were only asked to follow a given circuit in real life traffic with no further instructions. The gaze behaviour was measured to estimate the required information for the drivers to navigate the test vehicle.

For the second part of the study, a secondary task was introduced to the vehicle to verify, if the driver feels the need to focus more on the road in situations, where the viewing distance is limited. The third part was set in a driving simulator in which the same road geometry

4.3 eye tracking 69 as for the first two test parts was remodelled. In different situations, with different the view-ing distance was limited, different objects (potential hazards) were added at random. The findings of Schulzshow, that for viewing distances under 200 m an increased concentration and attention onto the road is needed. This increases with decreasing viewing distance and cannot be explained by any other type of road geometry. With viewing distances of below 150 m the driver’s gaze is focused nearly completely onto the road and for viewing distances even lower, an increase in breaking manoeuvres (even on straight roads) is recorded. The conclusion therefore is, that limited viewing distances lead to unsure driving behaviour and require a higher level of concentration and attention. [134]

A completely different goal for using eye tracking in a real life driving scenario is presented by Winter who uses the gaze cumulation of drivers in complex situations in combination with static luminance recordings to calculate the adaptation luminance in inner city driving therefore taking the approach presented by Heynderickx and applying it in a real driving situation [135]. Similar to the work presented by Mourant, the data is also analysed for differences in the experience of the drivers. The evaluated data was recorded in a different study with a head mounted eye tracking system in which 24 subjects drove on road in Berlin, Germany. The difference in experienced drivers and inexperienced drivers is set at 10 000 km of total driving experience. The luminance recordings are taken in the same roads as the driving test was performed from the inside of a car to ensure the effect of the windscreen is taken into account. The results in terms of orientation behaviour match the results previously found, in a way, that experienced drivers tend to use a significantly wider area that is located further up to orientate themselves in given road situations. This is visualized in figure4.18 where on the left side, the cumulative gaze for and inexperienced driver is shown, the mid-dle shows the much wider gaze distribution for an experienced driver and the image on the right shows the gaze distribution over all test subjects.

Figure4.18 – Differences in cummulative gaze vectors between an experienced driver (left) and inexperi-enced driver (middle) and cummalitve over all participants (right) [136].

The data presented shows, that the most primitive shape to describe the gaze is an 2°/10°

ellipse. No further assesment regarding the adaptation luminance is made due to the large scattering of the eye tracking data. [136]

In a similar work, Winter analyses the difference in gaze behaviour for main roads and residential areas in Berlin. His findings show, that for the main road, the gaze behaviour can be described by a circular shape while for residential areas, an ellipsoid is more accurate.

This also shows, that between these two road types, the vertical orientation does not differ significantly, while the horizontal orientation does. [137]

Obviously, this does not include all the research that has been done in terms of eye tracking or gaze behaviour in while driving. However, listing all available publications and shortly summarizing them would easily be enough to fill another complete book. Therefore, the

in-70 related work

terested reader is referred to the original publications on the topics general eye tracking in automotive use: [138,139], gaze distribution on motorways [140], orientation in curve driving [141–144], adaptation luminance in dependence of gaze behaviour [145] but the presented se-lection of research on eye tracking should showcase the different topics that can be tackled using eye tracking.

One very important fact that arises when comparing the different findings from previous research is, that there are areas where all previous studies come to the same conclusion. This is for example the case for the dependence of the orientation behaviour between experienced and inexperienced drivers or the general orientation behaviour when driving through curves.

Other very basic findings, the difference between day- and night-time driving however, lead to different results depending on which research one reads. While Brückmannand Damasky found a wider gaze distribution during the day, Diemrecorded the exact opposite.

Another remark is, that many of the presented studies were performed by a very limited amount of test subjects ranging from 3 subjects (Stahl), 4 subjects (Cengiz), 6 subjects (Shibata), 12 subjects (Winter), 20 subjects Schulz, to a maximum of 23 subjects (Winter), since eye tracking studies involve a time consuming calibration and (at least used to) include an even more time consuming evaluation of the generated data. However, due to the large variance shown in all data sets, it is inevitable, that a large number of subjects participates in eye tracking studies for statistical relevance.

The conclusion from the presented available data is, that due to the new and improved technology available for todays eye tracking systems, new data sets should be recorded and evaluated to today’s standards to compare them to the findings presented here. This is not only necessary due to the new technology in terms of eye tracking but new and improved headlamp technology as well. Furthermore, the traffic space in general has changed over the last couple of years with a much higher traffic density a lot more traffic happening in urban areas.