Advanced Facial Attributes Detection
Last updated: Sep-02-2024
Cloudinary provides an add-on for advanced face attribute detection capabilities, fully integrated into Cloudinary's image management and transformation pipeline. With the Advanced Facial Attribute Detection add-on, you can extend the Cloudinary features that involve semantic photo data extraction, image cropping and the positioning of image overlays. When using the Advanced Facial Attribute Detection add-on, your images are further processed and many advanced face attributes are automatically extracted. Cloudinary can then use these additional details to smartly crop, position, rotate and overlay images.
Cloudinary is a cloud-based service that provides an end-to-end image management solution including uploads, storage, transformations, optimizations and delivery, and offers a rich set of image management, transformation, analysis and cropping capabilities.
The Advanced Facial Attribute Detection add-on is an integrated face detection solution that utilizes Microsoft Cognitive Services. Microsoft Cognitive Services provides high precision face location detection with state-of-the-art cloud-based algorithms that can detect up to 64 human faces in an image. The detected faces are returned with rectangles (left, top, width and height) indicating the location of faces in the image in pixels, the exact position details of the eyes, mouth, eyebrows, nose and lips, as well as a series of face related attributes from each face such as pose, gender and age.
As of June 30th 2023, Microsoft Cognitive Services will be retiring facial recognition capabilities that can be used to try to infer emotional states and identity attributes which, if misused, can subject people to stereotyping, discrimination or unfair denial of services.
The following attributes will no longer be returned - emotion
, gender
, age
, smile
, facial_hair
, hair
and makeup
.
You can read more about the change and potential impact here.
Getting started
Before you can use the Advanced Facial Attributes Detection add-on:
You must have a Cloudinary account. If you don't already have one, you can sign up for a free account.
Register for the add-on: make sure you're logged in to your account and then go to the Add-ons page. For more information about add-on registrations, see Registering for add-ons.
Keep in mind that many of the examples on this page use our SDKs. For SDK installation and configuration details, see the relevant SDK guide.
If you are new to Cloudinary, you may want to take a look at How to integrate Cloudinary in your app for a walk through on the basics of creating and setting up your account, working with SDKs, and then uploading, transforming and delivering assets.
Face attribute detection on upload
The Advanced Facial Attribute Detection add-on is very simple to use. The extensive list of face attributes can be extracted from any photo when calling Cloudinary's upload API and setting the detection
parameter to adv_face
. For example, take a look at the following photo that we want to upload to Cloudinary's demo product environment:
When calling Cloudinary's API to upload the photo, the detection
parameter is added to extract the face attributes in the uploaded photo:
Learn more: Upload presets
The JSON snippet below contains an excerpt of the results of applying automatic face attribute detection on the uploaded image. The response includes very detailed information regarding each of the faces that were automatically detected in the photo:
-
bounding_box
- The bounding box surrounding a detected face. -
smile
- The likelihood that the person is smiling. -
head_pose
- The way the face is positioned in 3D. -
gender
- Whether the person is a male or a female. -
age
- Estimated age of the person. -
facial_hair
- The likelihood that the person has a moustache, beard, and/or sideburns, respectively. -
glasses
- Whether the person is wearing glasses. -
emotion
- The likelihood that the person's expression implies certain emotions. -
blur
- The amount of blur on the face. -
exposure
- The photo exposure on the face. -
noise
- A general and numeric evaluation of the amount of photographic noise on the face. -
makeup
- Whether the person is wearing eye and/or lip makeup. -
accessories
- An array of accessory items (if any) detected around the face area, such as headwear, glasses, or mask. -
occlusion
- Whether the forehead, eyes, or mouth is blocked, for example by a scarf, hat, or mask. -
hair
- Information about the hair, including whether the hair is visible in the photo, the likelihood that the person is bald, and the likelihood that the person's hair is a particular color. -
facial_landmarks
- Exact position details of specific elements of the mouth, eybrows, eyes, and nose.
Face attribute detection after uploading
The example above shows how to automatically detect face attributes of photos during their upload process. Alternatively, you can use Cloudinary's Admin API to apply automatic face attribute detection to already uploaded images, based on their public IDs. The following code sample uses Cloudinary's update API to apply face attribute detection to the uploaded image that has the 'coupled' public ID.
The face attributes that were previously extracted either using the 'upload' or 'update' API, are also available using the Admin API's show resource details method:
Face detection based cropping
Cloudinary provides a large set of image transformation and cropping options. Based on the position of facial attributes detected by the Advanced Facial Attribute Detection add-on, Cloudinary can crop your images to focus on the detected faces, while providing a large set of image transformation and cropping options when using a Cloudinary delivery URL or calling Cloudinary's image API.
To focus an automatic crop on the detected faces, simply set the crop
parameter to thumb
, fill
or crop
and the gravity
parameter to adv_faces
(set gravity to adv_face
for focusing on the single largest detected face in the image).
The following code sample creates a 150x150 thumbnail of the coupled
image that is generated using face-detection based cropping. The gravity
parameter is set to adv_faces
while the crop
mode is set to thumb
.
![150x150 thumbnail of faces cropped with the Advanced Facial Attribute Detection add-on 150x150 thumbnail of faces cropped with the Advanced Facial Attribute Detection add-on](https://res.cloudinary.com/demo/image/upload/c_thumb,g_adv_faces,h_150,w_150/coupled.jpg)
Alternatively, you can crop an image based on the single most visible face in a photo. The following code sample generates and delivers a 100x100 thumbnail containing a single face. The gravity
parameter is set to adv_face
while the crop
mode is set to thumb
.
![100x100 thumbnail of a single face cropped with the Advanced Facial Attribute Detection add-on 100x100 thumbnail of a single face cropped with the Advanced Facial Attribute Detection add-on](https://res.cloudinary.com/demo/image/upload/c_thumb,g_adv_face,h_100,w_100/coupled.jpg)
Face detection with signed URLs
Cloudinary's dynamic image transformation URLs are quite powerful, however, due to the potential costs of users accessing unplanned dynamic URLs with the Advanced Facial Attribute Detection cropping directives, image transformation add-on URLs are required (by default) to be signed using Cloudinary's authenticated API and with the sign_url
parameter set to true
.
- You can create dynamic unsigned image transformation URLs based on Advanced Facial Attribute Detection as long as you first extract the face attributes during upload or by using the Admin API.
- You can optionally remove this default requirement for a particular add-on by selecting it in the Allow unsigned add-on transformations section of the Security Settings in the Cloudinary Console.
For example, a signed URL and the code to generate it:
As you can see, the generated Cloudinary URL includes a signature component (s--vnhlc4WH--
). Only URLs with a valid signature (specific to that transformation) will be approved for on-the-fly image transformation using advanced facial attribute detection.
Eyes detection based cropping
In addition to face-detection based cropping, Cloudinary can dynamically crop your images based on the position of detected eyes. Simply set the gravity
parameter to adv_eyes
(g_adv_eyes
for URLs) to center the image on the detected eyes. The example below delivers a 100x40 thumbnail centered on the eyes of the woman in the photo.
![Thumbnail centered on eyes using Advanced Facial Attribute Detection Thumbnail centered on eyes using Advanced Facial Attribute Detection](https://res.cloudinary.com/demo/image/upload/c_thumb,g_adv_eyes,h_40,w_100/face1.jpg)
Eyes detection for accurate red eye removal
Cloudinary's rich transformation capabilities already allow you to automate red eye removal by setting the effect
parameter to redeye
(e_redeye
for URLs). This enables smart red eye removal algorithms to be automatically applied on the fly to uploaded images.
In order to get even better quality results, you can use Cloudinary's Advanced Facial Attribute Detection add-on for eye detection together with the red eye removal effect. The add-on can automatically detect where eyes are located in a photo and then the red eye removal algorithm can be applied in a more precise way: simply set the effect
parameter to adv_redeye
(e_adv_redeye
for URLs).
For example, an image named redeye
was uploaded to Cloudinary. A cropped 300x80 thumbnail of the photo centered on the eyes only for illustration is displayed below:
![redeye example photo redeye example photo](https://res.cloudinary.com/demo/image/upload/c_thumb,g_adv_eyes,h_80,w_300/redeye.jpg)
Adding the redeye effect parameter and setting its value to adv_redeye
delivers the following image:
![photo after applying the adv_redeye effect photo after applying the adv_redeye effect](https://res.cloudinary.com/demo/image/upload/e_adv_redeye/c_thumb,g_adv_eyes,h_80,w_300/redeye.jpg)
Face overlays
The extracted facial attribute details also include the pose of the detected face, which allows Cloudinary to not only position overlays on top of detected faces, but also to rotate and scale the overlays accordingly for exact positioning over the underlying face. This is easily accomplished by setting the gravity
parameter of the added overlay to adv_faces
.
For example, the following PNG image named silver_face_mask
was uploaded to Cloudinary:
To smartly overlay the mask image on top of all detected faces in the coupled
image, the overlay
parameter is set to the ID of the mask image and the gravity
parameter is set to adv_faces
. In addition, we set the region_relative
flag together with a 1.1
width, which means that each overlay will be scaled to 110% of the width of the detected face.
![mask overlaid coupled.jpg mask overlaid coupled.jpg](https://res.cloudinary.com/demo/image/upload/l_silver_face_mask/c_scale,fl_region_relative,w_1.1/fl_layer_apply,g_adv_faces/coupled.jpg)
Eyes overlays
As described above, the Advanced Facial Attribute Detection add-on detects specific facial attributes, including the exact position of the eyes of each face in a photo. Based on this information, Cloudinary can position overlays on top of all the detected eye pairs in an image.
For example, the following PNG image named glasses
was uploaded to Cloudinary:
In order to smartly overlay the glasses on top of all detected eye pairs in an uploaded image, the overlay
parameter is set to the ID of the glasses image and the gravity
parameter is set to adv_eyes
. In addition, we set the region_relative
flag together with a 1.5
width, which means that each overlay should be scaled to 150% of the detected eyes.
![glasses added to auto detected eyes of all faces in the image glasses added to auto detected eyes of all faces in the image](https://res.cloudinary.com/demo/image/upload/l_glasses/c_scale,fl_region_relative,w_1.5/fl_layer_apply,g_adv_eyes/coupled.jpg)
In another example, the following PNG image named harlequinmask
was uploaded to Cloudinary:
To smartly overlay the mask on top of all the detected eye pairs in the uploaded image cloudinary_team
, the overlay
parameter is set to the ID of the harlequinmask
image and the gravity
parameter is set to adv_eyes
. We also set the region_relative
flag together with a 1.7
width to scale the overlay to 170% of the width of the detected eyes, and resize the image to an oval thumbnail with a width of 700 pixels.
![add a mask overlay on the eyes of everyone in the image add a mask overlay on the eyes of everyone in the image](https://res.cloudinary.com/demo/image/upload/c_scale,w_700/r_max/l_harlequinmask/c_scale,fl_region_relative,w_1.7/fl_layer_apply,g_adv_eyes/cloudinary_team.jpg)
For a full list of additional Cloudinary's image transformation options, see Image transformations.