Design and Implementation of a Driver Drowsiness Detection System

A Practical Approach

Aleksandar Čolić, Oge Marques and Borko Furht

Computer & Electrical Engineering and Computer Science Department, Florida Atlantic University,
777 Glades Road, Florida, 33431, U.S.A.

Keywords: Driver Drowsiness Detection, Fatigue Detection, Drivers Assistance System, Eye-state Recognition, Head
Pose Estimation.

Abstract: This paper describes the steps involved in designing and implementing a driver drowsiness detection system
based on visual input (driver’s face and head). It combines off-the-shelf software components for face detec-
tion, human skin color detection, and eye state (open vs. closed) classification in a novel way. Preliminary
results show that the system is reliable and tolerant to many real-world constraints.

1 INTRODUCTION combination of computer vision and machine learn-

ing techniques to detect events of interest, measure
The interest in equipping vehicles with driver drowsi- them, and make a decision on whether the driver may
ness detection systems has been motivated by alarm- be drowsy or not. If the sequence of captured im-
ing statistics, such as the 2013 World Health Organi- ages and measured parameters (e.g., pattern of nod-
zation report (World Health Organization, 2013) stat- ding or time lapsed in “closed eye state”) suggest that
ing that: 1.24 million people die on the road ev- the driver is drowsy, an action – such as sounding an
ery year; approximately 6% of all the accidents are audible alarm – might be warranted.
caused by drivers driving in a drowsy state; and most The remainder of the paper is structured as fol-
of the accidents of this type result in fatalities. lows: Section 2 describes the purposes, context, and
Drowsiness (also referred to as sleepiness) can be general architecture of our work, Section 3 reports re-
defined as “the need to fall asleep”. This process is sults from preliminary experiments. Concluding re-
a result of normal human biological rhythm and its marks are presented in Section 4.
sleep-wake cycles. The longer the period of wakeful-
ness, the more pressure builds for sleep and the more
difficult it is to resist it (Akertedt et al., 2002). 2 OUR WORK
There are several different methodologies used
to detect, measure and predict driver drowsiness (or
This section describes the requirements, constraints,
sleepiness):
basic architecture, and selected algorithms associated
• Subjective methods with our driver drowsiness detection system. The
• Physiological methods hallmarks of the proposed system are its robustness,
accuracy, and overall simplicity.
• Vehicle-based methods
• Behavioral methods 2.1 Requirements and Constraints
• Hybrid methods
The driver drowsiness detection system described in
For the work described in this paper, we have
this paper must comply with the following main re-
adopted a subset of behavioral methods for driver
quirements:
drowsiness detection. These methods are based on the
detection of behavioral clues, e.g., closing of the eyes, • Algorithmically simple and easy to implement.
yawning and nodding of the head. They typically use We chose to rely on off-the-shelf solutions for
a video camera for image acquisition and rely on a most stages, based on the popularity and success

Čolić A., Marques O. and Furht B..

Design and Implementation of a Driver Drowsiness Detection System - A Practical Approach.
241
DOI: 10.5220/0005012302410247
In Proceedings of the 11th International Conference on Signal Processing and Multimedia Applications (SIGMAP-2014), pages 241-247
ISBN: 978-989-758-046-8
Copyright c 2014 SCITEPRESS (Science and Technology Publications, Lda.)
SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

of the associated algorithms (e.g., Viola-Jones If that comfort zone is left, performance can be
face detector, Support Vector Machine classifier). dropped significantly.
• Easily portable to different platforms. The ap- • Hardware and software limitations. Typical mo-
plication must run on a mobile device (e.g., bile devices have one or two processor cores,
Android-based smartphone) mounted on the ve- reduced working memory and tend to work on
hicle’s dashboard. Ideally, it should be easily lower clock frequencies, compared to their desk-
portable to other (e.g., iOS-based) mobile devices top counterparts. The reason for all of this is to
of comparable size and computational capabili- reduce the energy consumption but it creates a sig-
ties. nificant obstacle in designing this type of system.
• Computationally non-intensive. Since (near) real- • Driver cooperation. Last, but certainly not least,
time performance is required, algorithms must all driver drowsiness detection systems assume a
be optimized to ensure continuous monitoring of cooperative driver, who is willing to assist in the
driver’s state without excessive burdening of the setup steps, keep the monitoring system on at all
device’s main processor. As a side benefit, battery times, and take proper action when warned by the
consumption is reduced as well. system of potential risks due to detected drowsi-
• Accuracy. One of the main challenges of design- ness.
ing such a system is related to the fact that both
type I and type II errors are highly undesirable, 2.2 System Architecture
for different reasons: type I errors (false positives)
will annoy the driver and reduce their willingness Our driver drowsiness detection system consists of
to use the system (due to excessive false alarms), four main stages (Figure 1):
whereas type II errors (false negatives) can have
literally catastrophic consequences and defeat the
purpose of the entire system.
• Robustness. The system must be tolerant to mod-
est amounts of lighting variations, relative cam-
era motion (e.g. due to poor road conditions), Figure 1: Four stages of Drowsiness Detection System
changes to the driver’s visual appearance (even in
the course of a session, e.g., by wearing/removing 2.2.1 Detection Stage
a hat or sunglasses), camera resolution and frame
rates, and different computational capabilities of This is the initialization stage of the system. Every
the device’s processors. time the system is started it needs to be set up and op-
timized for current user and conditions. The key step
Some of the anticipated constraints and limitations in this stage is successful head detection (Figure 2). If
faced by the proposed system include: the driver’s head is correctly located we can proceed
• Lighting conditions. Frequent and drastic change to extract the features necessary for setting up the sys-
in darkness or brightness of a scene (or part of it), tem. Setup steps include: (i) extracting driver’s skin
which may happen even during the shortest driv- color and using that information to create custom skin
ing intervals, have been proven to be a significant color model and (ii) collecting a set of open/closed
challenge for many computer vision algorithms. eyes samples, along with driver’s normal head posi-
• Camera motion. Poor road conditions as well as tion.
a more aggressive style of driving can introduce To help achieve these goals, user interaction might
significant amount of vibrations and discomfort be required. The driver might be asked to sit comfort-
to the driving experience. Those vibrations can ably in its normal driving position so that system can
be passed onto the camera and cause distortion in determine upper and lower thresholds needed for de-
the images which can significantly skew the re- tecting potential nodding. The driver might also be
sults and decrease the overall performance of the asked to hold their eyes closed and then open for a
system. matter of few seconds each time. This is enough to
get the system started. Over time, the system will ex-
• Relative positioning of device. The camera must pand the dataset of obtained images and will become
be positioned within a certain range from the more error resistant and overall more robust.
driver and within a certain viewing angle. Every
computer vision algorithm has a “comfort zone”
in which it performs the best and most reliably.

242
 DesignandImplementationofaDriverDrowsinessDetectionSystem-APracticalApproach

proactive and alert the driver of potential dangers that

can arise. Combination of audio/visual alerts are used
to attract the driver’s attention and raise their alertness
level. Alerting has to be implemented in such a way as
not to cause the opposite effect of intended and startle
the driver into causing an accident.

2.3 Selected Algorithms for Feature

Detection and Image Classification
Figure 2: Successful Initial Face and Eyes Detection. The proposed system relies on three main features of a
driver: face, eyes and skin. We have chosen the popu-
2.2.2 Tracking Stage lar Viola-Jones algorithm for baseline face detection,
due to its wide availability and overall simplicity. We
Once the driver’s head and eyes are properly located enhanced the face detection process to make use of
and all the necessary features are extracted, the sys- skin color information. Human skin color has unique
tem enters the regular tracking (monitoring) stage. A features. These features can be best expressed and de-
key step in this stage is the continuous monitoring of scribed by breaking given color into its basic chroma
the driver’s eyes within a dynamically allocated track- components (red, green, blue), and defining compo-
ing area. More specifically, in order to save some nents’ ranges. It has been shown that the vast major-
processing time, the system will determine the size ity of skin color types have their red chroma compo-
of the tracking area based on the previous history of nent in a range between 133 and 173, and their blue
eye movements. For example, if the eyes were mov- chroma component in a range of 77 – 127. One of the
ing horizontally to the left for a number of frames it is contributions of this work is the adoption of a user-
to be expected that that trend with continue in the fol- specific red chroma range. Figure 3 shows differ-
lowing frame also. So it is logical to expand the track- ences between using generalized range and custom,
ing area towards the expected direction of the eyes user specific range.
and shrink the area in other three directions. During In the initialization stage of the system, when the
this stage, the system must also determine the state of face of the driver is detected, the area containing the
the eyes. All these tasks must be carried out in real- face is used to analyze the specific red chroma range
time; depending on the processor’s abilities and cur- in which current driver’s skin color falls in. Chroma
rent load, it might be necessary to occasionally skip a values can fall in the range between 0 and 255. His-
few frames, without sacrificing algorithmic accuracy. togram of red chroma component of the area con-
taining driver’s face is created to give us that specific
2.2.3 Warning Stage range. This specific range is much narrower then the
generalized range. We can extract the upper and lower
If the driver keeps his eyes closed for prolonged pe- boundary from the histogram and use it in the follow-
riod of time or starts to nod, alertness has to be raised. ing stages of the system. So, when the system tracks
The key step within this stage is close monitoring of the eyes or face in the following frames, by analyz-
drivers eyes. The system must determine whether the ing the red chroma component histogram we can con-
eyes are still closed, and what is the eyes’ position rel- firm that the tracked object really is pair of eyes or a
ative to previously established thresholds. We cannot driver’s face.
afford to skip frames in this stage. In practice, track- Once our detection algorithm has successfully de-
ing of eyes is performed much in the same way as in tected a face and, subsequently, the eyes, it focuses
the tracking stage with the addition of the following on determining in what state the eyes are (closed or
processes: calculation of velocity and trajectory of the open). The proposed system monitors if the driver’s
eyes and threshold monitoring. These additional com- eyes are being closed for prolonged period of time. If
putations are required to improve the system’s ability that is the case, we can conclude that the driver might
to determine whether the driver is drowsy or not. be experiencing signs of drowsiness. The classifi-
cation method implemented in our system uses data
2.2.4 Alert Stage from the (most recent) setup stage as training data that
is applied to a 2-class Support Vector Machine (SVM)
Once it has been determined that the driver appears to classifier whose job is to distinguish the difference be-
be in an abnormal driving state, the system has to be tween open and closed eyes.

243
SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

(a) Eyes above the upper (b) Head tilting down - Eyes be-
threshold - nodding did not start neath the upper threshold
(a) Generalized skin color chroma range
yet

(c) Head tilting down - Eyes be- (d) Head recovering up - eyes
neath the lower threshold above lower threshold
(b) User-specific skin color chroma range

Figure 3: Chroma-based skin detection comparison.

In order to increase the overall robustness of the

system, we have adopted a classification approach
that relies on both head position and eye state. Our
approach is based on the following premises:
• Driver’s head position does not deviate a lot when
fully awake.
(e) Head recovering up - eyes
• When sleepy, head position changes drastically. above upper threshold: Nod is
complete
Our system implements a dynamic two-threshold
approach that is simple and effective way of detect- Figure 4: Nodding detection method and its stages.
ing abnormal head behavior of a driver. The upper
threshold is positioned slightly beneath the eyes. As As long as the eyes are held above the upper
long as the eyes are located above this threshold sys- threshold, the system is considered to be in a nor-
tem considers this to be wide awake, active state of mal state. Once the head start tilting vertically down
the driver in which he is looking ahead towards the and crosses the upper threshold, it is considered a be-
road. When the eyes start going vertically down and ginning of the nodding sequence. From here, if the
cross the upper threshold the system is preparing for head recovers back up, the nodding sequence is can-
potential nodding. If the head continues to nod for- celed; but if the head continues down and crosses the
ward and the eyes eventually cross the lower thresh- lower threshold, we know that the driver is certainly
old we know that driver is certainly not focused on the not monitoring the road anymore. Also, while mon-
road. Human nodding is very specific and consists of itoring the position of the eyes relative to the thresh-
head slowly dropping down followed by rapid recov- olds, the system also monitors the state of the eyes.
ering right back into original position. Our system If at anytime the driver opens his eyes while going
takes this into account. The speed of each stage of down in his nodding sequence, the sequence is can-
the nodding is measured in order to filter out potential celed. For a “true nod”, the driver has to keep his
false positives such as driver nodding in agreement eyes closed while moving his head downwards. Once
to something. Figure 4 shows the nodding detection the lower threshold is crossed, the system can expect
method. rapid vertical recovery of the driver’s normal driv-

244
 DesignandImplementationofaDriverDrowsinessDetectionSystem-APracticalApproach

ing position with potential opening of the eyes while same distance throughout; the camera starts pointed
moving upward. The system also monitors the ve- at driver’s left profile and ends pointing at driver’s
locity of the nod since it is essential in distinguishing right profile. The driver’s head stays stationary, only
a true sleepy nodding from other potentially similar the camera moves around it while keeping constant
head movements that are not caused by nodding. distance from it. Video used for testing contains 310
frames of driver’s head from various angles. Figure 5
shows the process and the results. We can deduct that
the detection algorithm performs consistently. If we
3 PRELIMINARY EXPERIMENTS define an angle of 0 degrees to be when the camera
is directly facing the driver, we can conclude that the
In order to fulfill all of the challenging prerequisites
range in which it was reliably detecting drivers eyes
we have to test out limitations of our chosen off-the-
falls approximately within 25 degree viewing angle
shelf components. Some of our preliminary tests were
in both direction from a driver. This result is very
devised to help us out with that. We started by setting
encouraging since this means that we can position the
our work environment in MATLAB. For the purposes
camera of the system conveniently on car’s dashboard
of testing, we simplified our dynamic solution to its
(usually has available slot for attaching devices such
core, into a horizontal, linear solution that can per-
as phones etc.) which is within driver’s reach.
form basic face/eye detection as well as simple classi-
fication task of determining eye state. The goal of the
devised set of experiments is to test the basic perfor-
mance in various ways.
From the very beginning we decided to use Viola-
Jones algorithm, Haar-like feature based face/eyes de-
tection algorithm available in MATLAB (Viola and
Jones, 2001). It is known for being stable and compu- (a) Test setup (b) Angle range results
tationally non-intensive algorithm. Haar-like feature
can be described as a set of two or more adjacent rect-
angular regions organized in a specific way in a given
detection window. This algorithm proves to be com-
pute non-intensive since it’s core operation is simple
summing up of the pixel intensities in a given regions,
followed by calculation of differences between them.
That difference is then used to categorize subsections
of an image to be face/eyes or not.
Also, for differentiating between open and closed
eyes we decided to go with a proven two-class Sup-
port Vector Machine (SVM) classifier (Cortes and
Vapnik, 1995). SVM is a non-probabilistic binary lin-
ear classifier. It takes a set of input data and predicts,
for each given input, which of two possible classes (c) Consistency results
forms the output. This is perfect since our basic prob-
lem can be defined as two class problem: open eyes Figure 5: Camera rotating: angle change limitations.
vs. closed eyes.
3.2 Head Rotation Test
3.1 Camera Rotation Test
In this test the camera stays directly facing the driver
The aim of this test is to find out how does move- while the driver rotates his head to the right, returns
ment of the camera and change in the viewing angle to the starting position and then rotates the head to the
relative to the driver influences the performance of left and back to center again. We are again looking to
used face/eye detection algorithm as well as the con- see what is the angle of breaking for our head/eye de-
sistency of its detection. The experiment was setup in tection algorithm and what is its consistency across
following manner: the driver is sitting inside the car, the frames. Figure 6 is showing the tests setup and re-
looking towards the road; the camera, approximately sults. Difference in detection angles is much smaller
50 centimeters away from the driver’s head, will make than in previous test, only around 5 degrees. It seems
a semi-circle around the driver’s head, keeping the that the algorithm is consistently and reliably detect-

245
SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

ing drivers eyes as long as the driver is facing towards

the camera and his gaze does not deviate more then 35
degrees in both directions from the front facing posi-
tion. Even though the difference in detection angles is
much smaller, there is an obvious false positive pro-
duced when drivers head turns almost 90 degrees to
the left. Though one eye is visible the other eye is
completely occluded. Background influenced the al-
gorithm into wrong conclusion.

Figure 7: Real-World test results.

(a) Test setup (b) Angle range results

SVM classification algorithm. A special video con-
taining 571 frames of a driver sitting in a drivers posi-
tion is used. Driver is moving minimally. Every frame
is manually marked as containing open or closed eyes.
Open and closed eye samples are roughly the same in
number, 304 containing open eyes and 267 contain-
ing closed eyes. 70 % of randomly selected samples
were used to train the SVM model on which the re-
maining 30% of the samples were tested. Using avail-
able SVM MATLAB version, our preliminary results
showed 96 % success rate. We are acknowledging
the fact that the sample size is very small and that we
only had samples relating to one subject so instead
on focusing on high accuracy rate we concluded that
building a system in which an SVM eye model is user
specific is encouragingly good idea. It can simplify
(c) Consistency results
significantly our system as well as provide a dose of
Figure 6: Head rotating: angle change limitations. robustness and reliability to it. Such an eye model can
always be upgraded, thus increase its quality, through
frequent use of the system. Some of the classification
3.3 Real-world Test results are shown on Figure 8.
For pure test of consistency of the detection algo-
rithm we recorded an approximately 900 frame video
sequence of a driver behaving naturally behind the
wheel of the car. He is moving around, adjusting his (a) Closed (b) Open
seat, mirror, turning to face the passengers. We want
to see if there are false positives and in what volume
and how reliable or basis detection algorithm really
is. Figure 7 is showing quite good consistency: there (c) Called closed (d) Called open (e) Called closed

are no sudden changes except with one spike for the Figure 8: Support Vector Machine result examples: (a) &
duration of one frame. This spike actually is a true (b) correct; (c), (d) & (e) incorrect.
positive.

3.4 Open vs. Closed Eyes Test

4 CONCLUDING REMARKS
To test the behavior of our chosen classification al-
gorithm we decided to go with the simple solution of In this paper, we have described the process of design-
using gray-scaled and cropped image regions contain- ing and implementing a driver drowsiness detection
ing eyes and to feed that set of pixels as an input to the system by combining some off-the-shelf algorithms

246
 DesignandImplementationofaDriverDrowsinessDetectionSystem-APracticalApproach

with some of the novel approaches in a clever man-

ner. The system is dynamic, it can update and mod-
ify its components like current drivers eye model and
skin model throughout its life cycle. Constantly up-
grading baseline models used can increase the over-
all resilience towards errors. Moreover, the system
is user specific. All the feature models created are
solely based on the current users features instead of
using generalized parameters. Such an approach sim-
plifies the system while providing solid performance.
Each of the algorithms is performing solidly by itself.
But they do have their limitations. To increase the
reliability and accuracy of the system, both baseline
detection/tracking algorithm and eye-state classifica-
tion algorithm are complimented with simple, but ef-
ficient, custom algorithms.

REFERENCES
Akertedt, T., Fredlung, P., Gillberg, M., and Jansson, B.
(2002). A prospective study of fatal occupational ac-
cidents – relationship to sleeping difficulties and occu-
pational factors. Journal of Sleep Research, 11(1):69–
71.
Cortes, C. and Vapnik, V. (1995). Support-vector networks.
Machine Learning, 20(3):273–297.
Viola, P. and Jones, M. (2001). Robust real-time object de-
tection. In International Journal of Computer Vision.
World Health Organization (2013). Global Status Report
on Road Safety 2013: Supporting a Decade of Action
: Summary. World Health Organization.

247