Chapter 5 Image data

In this class, we learn how to deal with digit images. A digital image is a numeric representation of a two-dimensional image such as a photo. A typical image is a raster image whose quality would change after scaling. Another type is called a vector image which can be scaled without losing details.

x <- rnorm(100)
y <- rnorm(100)

## raster iamge
png("figures/rasterimage.png")
plot(x,y)
dev.off()
## png 
##   2

## vector image
svg('figures/vectorimage.svg')
plot(x,y)
dev.off()
## png 
##   2

Typical images created from a camera or scanner are raster images. In this class, we mostly concern such images.

Raster images are made of pixels. A pixel is a single point or the smallest single element in a display device. If you zoom in to a raster image you may start to see a lot of little tiny squares. Each square is a pixel. Each pixel is made of combinations of primary colors, e.g., for photos from a digital camera – red, green and blue – called RBG channels.

RGB is an additive color model to add red, green and blue light together reproduce different colors. The amount of light can be quantified from 0 to a maximum number. For example, most software uses a number from 0 to 255 to represent the amount of different light. In this case, we can use a triplet (R, G, B) to get a certain color. For example, (0,0,0) is black, (255, 0, 0) is red, and (255, 255, 255) is white. Try out the RGB explorer.

R actually use a value between [0, 1] to represent RGB, which can be viewed as a rescale of the values between [0, 255].

Therefore, for a given image, it can be viewed as a collection of pixels with each having an RGB color. See the example below (from https://takundagambino.files.wordpress.com/2014/06/pixelzoom.png).

Terminology

Channel. Each of the three primary colors is called a channel.
Bit per channel. Most digital cameras allow 8-bits per channel so they allow for \(2^8 = 256\) different intensity values for each primary color.
Bit per pixel. With three 8-bits channel, there are a total of \(2^8 \times 2^8 \times 2^8 = 2^{24}\) different colors, that’s 24 bits. If each pixel allows for this number of different colors, it is called “true color.”

In addition to the image information, an image such as a photo taken by a camera often includes a lot of metadata, additional information to describe the image. For example, the typical information includes the size of the image and the creation time.

5.1 Read image data into R

5.1.1 Read metadata

The R package exifr can be used to get the metadata of an image file and the package imager can read and manipulate the image data.

library(exifr)
meta <- read_exif('figures/ND.jpg')
str(meta)
## Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  107 variables:
##  $ SourceFile               : chr "figures/ND.jpg"
##  $ ExifToolVersion          : num 11.2
##  $ FileName                 : chr "ND.jpg"
##  $ Directory                : chr "figures"
##  $ FileSize                 : int 2165315
##  $ FileModifyDate           : chr "2019:01:23 15:00:50-05:00"
##  $ FileAccessDate           : chr "2019:04:08 12:34:04-04:00"
##  $ FileCreateDate           : chr "2019:04:08 12:33:53-04:00"
##  $ FilePermissions          : int 666
##  $ FileType                 : chr "JPEG"
##  $ FileTypeExtension        : chr "JPG"
##  $ MIMEType                 : chr "image/jpeg"
##  $ ExifByteOrder            : chr "MM"
##  $ ModifyDate               : chr "2018:11:11 14:13:31"
##  $ GPSDateStamp             : chr "2018:11:11"
##  $ GPSAltitudeRef           : int 0
##  $ GPSLongitudeRef          : chr "W"
##  $ GPSLatitudeRef           : chr "N"
##  $ GPSTimeStamp             : chr "19:13:24"
##  $ Model                    : chr "Nexus 5"
##  $ YCbCrPositioning         : int 1
##  $ ResolutionUnit           : int 2
##  $ YResolution              : int 72
##  $ Orientation              : int 1
##  $ ColorSpace               : int 1
##  $ CreateDate               : chr "2018:11:11 14:13:31"
##  $ FNumber                  : num 2.4
##  $ FocalLength              : num 3.97
##  $ ApertureValue            : num 2.39
##  $ WhiteBalance             : int 0
##  $ ExifImageWidth           : int 3264
##  $ SubSecTime               : int 954220
##  $ ShutterSpeedValue        : num 1641
##  $ DateTimeOriginal         : chr "2018:11:11 14:13:31"
##  $ SubSecTimeDigitized      : int 954220
##  $ ComponentsConfiguration  : chr "1 2 3 0"
##  $ ExifImageHeight          : int 2448
##  $ Flash                    : int 0
##  $ ExifVersion              : chr "0220"
##  $ InteropIndex             : chr "R98"
##  $ InteropVersion           : chr "0100"
##  $ ISO                      : int 101
##  $ FlashpixVersion          : chr "0100"
##  $ SubSecTimeOriginal       : int 954220
##  $ ExposureTime             : num 0.000609
##  $ XResolution              : int 72
##  $ Make                     : chr "LGE"
##  $ ThumbnailLength          : int 25537
##  $ ThumbnailOffset          : int 878
##  $ Compression              : int 6
##  $ ProfileCMMType           : chr ""
##  $ ProfileVersion           : int 512
##  $ ProfileClass             : chr "mntr"
##  $ ColorSpaceData           : chr "RGB "
##  $ ProfileConnectionSpace   : chr "XYZ "
##  $ ProfileDateTime          : chr "2009:03:27 21:36:31"
##  $ ProfileFileSignature     : chr "acsp"
##  $ PrimaryPlatform          : chr ""
##  $ CMMFlags                 : int 0
##  $ DeviceManufacturer       : chr ""
##  $ DeviceModel              : chr ""
##  $ DeviceAttributes         : chr "1 0"
##  $ RenderingIntent          : int 0
##  $ ConnectionSpaceIlluminant: chr "0.9642 1 0.82491"
##  $ ProfileCreator           : chr ""
##  $ ProfileID                : chr "41 248 61 222 175 242 85 174 120 66 250 228 202 131 57 13"
##  $ ProfileDescription       : chr "sRGB IEC61966-2-1 black scaled"
##  $ BlueMatrixColumn         : chr "0.14307 0.06061 0.7141"
##  $ BlueTRC                  : chr "base64:Y3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCf"| __truncated__
##  $ DeviceModelDesc          : chr "IEC 61966-2-1 Default RGB Colour Space - sRGB"
##  $ GreenMatrixColumn        : chr "0.38515 0.71687 0.09708"
##  $ GreenTRC                 : chr "base64:Y3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCf"| __truncated__
##  $ Luminance                : chr "0 80 0"
##  $ MeasurementObserver      : int 1
##  $ MeasurementBacking       : chr "0 0 0"
##  $ MeasurementGeometry      : int 0
##  $ MeasurementFlare         : int 0
##  $ MeasurementIlluminant    : int 2
##  $ MediaBlackPoint          : chr "0.01205 0.0125 0.01031"
##  $ RedMatrixColumn          : chr "0.43607 0.22249 0.01392"
##  $ RedTRC                   : chr "base64:Y3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkAXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCf"| __truncated__
##  $ Technology               : chr "CRT "
##  $ ViewingCondDesc          : chr "Reference Viewing Condition in IEC 61966-2-1"
##  $ MediaWhitePoint          : chr "0.9642 1 0.82491"
##  $ ProfileCopyright         : chr "Copyright International Color Consortium, 2009"
##  $ ChromaticAdaptation      : chr "1.04791 0.02293 -0.0502 0.0296 0.99046 -0.01707 -0.00925 0.01506 0.75179"
##  $ ImageWidth               : int 3264
##  $ ImageHeight              : int 2448
##  $ EncodingProcess          : int 0
##  $ BitsPerSample            : int 8
##  $ ColorComponents          : int 3
##  $ YCbCrSubSampling         : chr "2 2"
##  $ Aperture                 : num 2.4
##  $ GPSAltitude              : int 169
##  $ GPSDateTime              : chr "2018:11:11 19:13:24Z"
##  $ GPSLatitude              : num 41.7
##  $ GPSLongitude             : num -86.2
##  $ GPSPosition              : chr "41.7013416666667 -86.2440694444444"
##  $ ImageSize                : chr "3264x2448"
##   [list output truncated]

Note that metadata can include as many as 107 pieces of information. For example, for this particular photo, it even included the GPS location information. With the GPS information, we can easily reverse geocode a location, e.g., through Google service. For example, using the code below, we can see that the photo was taken at Notre Dame.

library(jsonlite)

gps <- paste0(meta$GPSLatitude, ",", meta$GPSLongitude)

gps <- "41.7,-86.2"

fromJSON(paste0("https://maps.googleapis.com/maps/api/geocode/json?latlng=", gps, "&key=AIzaSyCyf5osRyuvRj1hVzYvH7bPhrqQVmXqYXo"))
## $plus_code
## $plus_code$compound_code
## [1] "MQXX+XX South Bend, IN, USA"
## 
## $plus_code$global_code
## [1] "86HMMQXX+XX"
## 
## 
## $results
##                                                                                                                                                                                                                                                                                                                                                                                                  address_components
## 1                         1709, North Hickory Road, South Bend, Clay Township, St. Joseph County, Indiana, United States, 46635, 1709, N Hickory Rd, South Bend, Clay Township, St Joseph County, IN, US, 46635, street_number, route, locality, political, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political, postal_code
## 2 1799, Rerick Street, South Bend, Clay Township, St. Joseph County, Indiana, United States, 46635, 2029, 1799, Rerick St, South Bend, Clay Township, St Joseph County, IN, US, 46635, 2029, street_number, route, locality, political, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political, postal_code, postal_code_suffix
## 3                                                            Rerick Street, South Bend, Clay Township, St. Joseph County, Indiana, United States, 46635, Rerick St, South Bend, Clay Township, St Joseph County, IN, US, 46635, route, locality, political, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political, postal_code
## 4                                                                                             46635, South Bend, Clay Township, St. Joseph County, Indiana, United States, 46635, South Bend, Clay Township, St Joseph County, IN, US, postal_code, locality, political, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political
## 5                                                                                                                                                                     Clay Township, St. Joseph County, Indiana, United States, Clay Township, St Joseph County, IN, US, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political
## 6                                                                                                                  South Bend, Portage Township, St. Joseph County, Indiana, United States, South Bend, Portage Township, St Joseph County, IN, US, locality, political, administrative_area_level_3, political, administrative_area_level_2, political, administrative_area_level_1, political, country, political
## 7                                                                                                                                                                                                                                           St. Joseph County, Indiana, United States, St Joseph County, IN, US, administrative_area_level_2, political, administrative_area_level_1, political, country, political
## 8                                                                                                                                                                                                                                                                                                                        Indiana, United States, IN, US, administrative_area_level_1, political, country, political
## 9                                                                                                                                                                                                                                                                                                                                                                             United States, US, country, political
##                              formatted_address geometry.location.lat
## 1 1709 N Hickory Rd, South Bend, IN 46635, USA              41.69925
## 2    1799 Rerick St, South Bend, IN 46635, USA              41.69972
## 3         Rerick St, South Bend, IN 46635, USA              41.69978
## 4                    South Bend, IN 46635, USA              41.71328
## 5                       Clay Township, IN, USA              41.72870
## 6                          South Bend, IN, USA              41.67635
## 7                    St Joseph County, IN, USA              41.62281
## 8                                 Indiana, USA              40.26719
## 9                                United States              37.09024
##   geometry.location.lng geometry.location_type geometry.viewport.northeast.lat
## 1             -86.19994                ROOFTOP                        41.70060
## 2             -86.20197     RANGE_INTERPOLATED                        41.70107
## 3             -86.20197       GEOMETRIC_CENTER                        41.70113
## 4             -86.21077            APPROXIMATE                        41.74545
## 5             -86.22601            APPROXIMATE                        41.76055
## 6             -86.25199            APPROXIMATE                        41.75210
## 7             -86.33768            APPROXIMATE                        41.76070
## 8             -86.13490            APPROXIMATE                        41.76137
## 9             -95.71289            APPROXIMATE                        71.53880
##   geometry.viewport.northeast.lng geometry.viewport.southwest.lat
## 1                       -86.19859                        41.69790
## 2                       -86.20062                        41.69837
## 3                       -86.20062                        41.69843
## 4                       -86.19107                        41.69447
## 5                       -86.17714                        41.69421
## 6                       -86.19129                        41.59734
## 7                       -86.05944                        41.43279
## 8                       -84.78466                        37.77174
## 9                       -66.88542                        18.77630
##   geometry.viewport.southwest.lng geometry.bounds.northeast.lat
## 1                       -86.20129                            NA
## 2                       -86.20332                            NA
## 3                       -86.20332                      41.69983
## 4                       -86.23165                      41.74545
## 5                       -86.27364                      41.76055
## 6                       -86.36048                      41.75210
## 7                       -86.52668                      41.76070
## 8                       -88.09789                      41.76137
## 9                       170.59570                      71.53880
##   geometry.bounds.northeast.lng geometry.bounds.southwest.lat
## 1                            NA                            NA
## 2                            NA                            NA
## 3                     -86.20197                      41.69974
## 4                     -86.19107                      41.69447
## 5                     -86.17714                      41.69421
## 6                     -86.19129                      41.59734
## 7                     -86.05944                      41.43279
## 8                     -84.78466                      37.77174
## 9                     -66.88542                      18.77630
##   geometry.bounds.southwest.lng
## 1                            NA
## 2                            NA
## 3                     -86.20197
## 4                     -86.23165
## 5                     -86.27364
## 6                     -86.36048
## 7                     -86.52668
## 8                     -88.09789
## 9                     170.59570
##                                                                                           place_id
## 1                                                                      ChIJYQcC6HXSFogRy6IwHU3B0-k
## 2 EikxNzk5IFJlcmljayBTdCwgU291dGggQmVuZCwgSU4gNDY2MzUsIFVTQSIbEhkKFAoSCcs9az520haIEaR5jOQzTO1gEIcO
## 3                                                                      ChIJl-L1QnbSFogRBEU89hre-vI
## 4                                                                      ChIJQbGK50LSFogRjBQ_5EYkz9s
## 5                                                                      ChIJiYQ80O_SFogR0gBc7-ZiN6g
## 6                                                                      ChIJE9NhSsQyEYgRBDKjb7PZSpc
## 7                                                                      ChIJ8YY3te1LEYgRvxj0oskckKU
## 8                                                                      ChIJHRv42bxQa4gRcuwyy84vEH4
## 9                                                                      ChIJCzYy5IS16lQRQrfeQ5K5Oxw
##                                   plus_code.compound_code plus_code.global_code
## 1 MRX2+M2 South Bend, Portage Township, IN, United States           86HMMRX2+M2
## 2                                                    <NA>                  <NA>
## 3                                                    <NA>                  <NA>
## 4                                                    <NA>                  <NA>
## 5                                                    <NA>                  <NA>
## 6                                                    <NA>                  <NA>
## 7                                                    <NA>                  <NA>
## 8                                                    <NA>                  <NA>
## 9                                                    <NA>                  <NA>
##                                    types
## 1                         street_address
## 2                         street_address
## 3                                  route
## 4                            postal_code
## 5 administrative_area_level_3, political
## 6                    locality, political
## 7 administrative_area_level_2, political
## 8 administrative_area_level_1, political
## 9                     country, political
## 
## $status
## [1] "OK"

The creation time, or the time when the photo was taken, can also be useful.

strptime(meta$CreateDate, '%Y:%m:%d %H:%M:%S')
## [1] "2018-11-11 14:13:31 EST"

5.1.2 Read image data

The function load.image from the package imager can be used to read the image data.

ND <- load.image('figures/ND.jpg')
ND
## Image. Width: 3264 pix Height: 2448 pix Depth: 1 Colour channels: 3

For this photo, the width is 3,264 pixels and the height is 2,448 pixels. Three primary color channels, RBG, are used in the photo. The data include the information of each pixel. For example, we can get all the RBG information in a data frame.

ND.data <- as.data.frame(ND, wide='c')
str(ND.data)
## 'data.frame':    7990272 obs. of  5 variables:
##  $ x  : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ y  : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ c.1: num  0.624 0.627 0.596 0.573 0.58 ...
##  $ c.2: num  0.663 0.667 0.631 0.616 0.624 ...
##  $ c.3: num  0.769 0.773 0.745 0.725 0.733 ...

x and y are coordinates of the pixels. The top left corner is (1,1) and the bottom right corner is (3264, 2448). In this specific photo, there are a total of 7990272 pixels.
c.1, c.2, c.3 represents the amount of RBG lights.

5.2 Manipulate an image in R

Resize the image. An image can be scaled down or scaled up. The way to do it is to recalculate the pixels information.

ND4 <- imresize(ND, 1/4)
ND4
## Image. Width: 816 pix Height: 612 pix Depth: 1 Colour channels: 3

plot(ND4)

Warning! The code takes some time to run.

ND8 <- imresize(ND, 1/8)
ND8.data <- as.data.frame(ND8) 

x <- y <- cc <- value <- NULL
for (i in 1:204){
  for (j in 1:153){
    for (k in 1:3){
      x <- c(x, i)
      y <- c(y, j)
      cc <- c(cc, k)
      value <- c(value, 
         mean(c(ND8.data[ND8.data$x==2*i-1 & ND8.data$y==2*j-1 & ND8.data$cc==k, 4],
              ND8.data[ND8.data$x==2*i-1 & ND8.data$y==2*j & ND8.data$cc==k, 4],
              ND8.data[ND8.data$x==2*i & ND8.data$y==2*j-1 & ND8.data$cc==k, 4],
              ND8.data[ND8.data$x==2*i & ND8.data$y==2*j & ND8.data$cc==k, 4]
                             )))
    }
  }
}

We can construct an image using the information.

ND16 <- as.cimg(data.frame(x=x,y=y,cc=cc,value=value))
plot(ND16)

Convert it to a grayscale image

grayscale(ND8)
## Image. Width: 408 pix Height: 306 pix Depth: 1 Colour channels: 1
plot(grayscale(ND8))

Change the color

plot(ND8/2, rescale=FALSE)


plot(ND8/2, rescale=TRUE)

Color distribution

ND8.data %>%
  mutate(channel=factor(cc,labels=c('R','G','B'))) %>%
  ggplot(aes(value, col=channel)) + 
  geom_histogram() + facet_wrap(~ channel)

Histogram equalization

Histogram equalization is a method in image processing of contrast adjustment using the image’s histogram. This can be viewed as a data transformation process. One way to do it is to use the empirical cdf of the data.

hist.eq <- function(im) as.cimg(ecdf(im)(im),dim=dim(im))

ND8.eq <- iiply(ND8, "c", hist.eq) 

par(mfrow=c(1,2))
plot(ND8)
plot(ND8.eq)

Edge detection

The edge is the places with quick changes, denoted by the derivatives.

ND8.gr <- imgradient(ND8)
plot(ND8.gr, layout="row")

ND8.gr %>% enorm %>% plot()

Blob detection

ND8.he <- imhessian(ND8)

Hdet <- with(ND8.he,(xx*yy - xy^2))

plot(Hdet)


threshold(Hdet, "99%") %>% plot()

5.3 Face recognition

One typical application of image analysis is face recognition, e.g., to recognize faces in a photo. Many methods are available such as the traditional principal component analysis and the latest convolutional neural network method.

As for the sentiment analysis, we use the Microsoft Face API to conduct the analysis. With the API, we can find the position of faces in a photo. It can also quantify the emotion based on a face.

To use the Face API, we again need to get an API key from Microsoft. For testing, the API key 5b336abc1ae94484a21cead03baedccc can be used. The free account allows 20 transactions per minutes.

5.3.1 Use the API to detect the face and related information

The following information is needed
- The image file to be used. Use upload_file to specify it. This file will be uploaded to the server of Microsoft.
- Your API key
- URL to the API in which we can specify what kind of information to get
  - returnFaceId=true: Unique faceId of the detected face, created by detection API and it will expire 24 hours after the detection call.
  - returnFaceLandmarks=true
  - returnFaceAttributes=age,gender,smile,hair,makeup,glasses,emotion
- Content-Type = application/octet-stream
To process the output, use the content() function.

facekey <- "5b336abc1ae94484a21cead03baedccc"
kelly1 <- upload_file('figures/kelly1.jpg')
apiurl <- "https://westus.api.cognitive.microsoft.com/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true&returnFaceAttributes=age,gender,smile,hair,makeup,glasses,emotion"

kelly1.json.out <- POST(apiurl, 
              add_headers(`Ocp-Apim-Subscription-Key` = facekey, 
                          `Content-Type` = "application/octet-stream"),
                          body = kelly1
                  )

kelly1.out <- kelly1.json.out %>%
  content()

Plot the face landmarks

kelly1.face <- load.image('figures/kelly1.jpg')
plot(kelly1.face)

x <- c(427, 427, 427+218, 427+218, 427)
y <- c(104, 104+218, 104+218, 104, 104)

lines(x, y, type='o', col='red')

We can also write a function to simplify the process.

# A function to plot the square
face.landmark <- function(face, ...){
  face.top <- face[[1]]$faceRectangle$top
  face.left <- face[[1]]$faceRectangle$left
  face.width <- face[[1]]$faceRectangle$width
  face.height <- face[[1]]$faceRectangle$height
  x <- c(face.left, face.left, face.left + face.width, face.left + face.width, face.left)
  y <- c(face.top, face.top + face.height, face.top + face.height, face.top,  face.top)
  lines(x, y, type='o', col='red', ...)
  points(face[[1]]$faceLandmarks$pupilLeft$x, face[[1]]$faceLandmarks$pupilLeft$y, pch=17, col='red')
  points(face[[1]]$faceLandmarks$pupilRight$x, face[[1]]$faceLandmarks$pupilRight$y, pch=17, col='red')
  points(face[[1]]$faceLandmarks$mouthLeft$x, face[[1]]$faceLandmarks$mouthLeft$y, pch=17, col='red')
  points(face[[1]]$faceLandmarks$mouthRight$x, face[[1]]$faceLandmarks$mouthRight$y, pch=17, col='red')
}

plot(kelly1.face)
face.landmark(kelly1.out)

Facial emotion

A particular useful output of the API is the detected emotions.

unlist(kelly1.out[[1]]$faceAttributes$emotion)
##     anger  contempt   disgust      fear happiness   neutral   sadness  surprise 
##     0.055     0.019     0.000     0.000     0.000     0.828     0.098     0.000

Other information

unlist(kelly1.out[[1]]$faceAttributes)
##                     smile                    gender                       age 
##                       "0"                    "male"                      "59" 
##                   glasses             emotion.anger          emotion.contempt 
##               "NoGlasses"                   "0.055"                   "0.019" 
##           emotion.disgust              emotion.fear         emotion.happiness 
##                       "0"                       "0"                       "0" 
##           emotion.neutral           emotion.sadness          emotion.surprise 
##                   "0.828"                   "0.098"                       "0" 
##          makeup.eyeMakeup          makeup.lipMakeup                 hair.bald 
##                   "FALSE"                   "FALSE"                     "0.1" 
##            hair.invisible      hair.hairColor.color hair.hairColor.confidence 
##                   "FALSE"                    "gray"                    "0.97" 
##      hair.hairColor.color hair.hairColor.confidence      hair.hairColor.color 
##                   "black"                    "0.62"                   "blond" 
## hair.hairColor.confidence      hair.hairColor.color hair.hairColor.confidence 
##                    "0.61"                   "brown"                    "0.53" 
##      hair.hairColor.color hair.hairColor.confidence      hair.hairColor.color 
##                   "other"                    "0.29"                     "red" 
## hair.hairColor.confidence 
##                    "0.02"

kelly1.out %>% melt()
##                                   value                  L3        L4
## 1  f22761bc-a0c0-4cd2-b994-1e812c9eb35a                <NA>      <NA>
## 2                                   104                 top      <NA>
## 3                                   427                left      <NA>
## 4                                   218               width      <NA>
## 5                                   218              height      <NA>
## 6                                 492.2           pupilLeft         x
## 7                                   165           pupilLeft         y
## 8                                 581.4          pupilRight         x
## 9                                 158.9          pupilRight         y
## 10                                526.9             noseTip         x
## 11                                211.4             noseTip         y
## 12                                498.4           mouthLeft         x
## 13                                271.2           mouthLeft         y
## 14                                578.5          mouthRight         x
## 15                                266.9          mouthRight         y
## 16                                  461    eyebrowLeftOuter         x
## 17                                  147    eyebrowLeftOuter         y
## 18                                505.3    eyebrowLeftInner         x
## 19                                144.7    eyebrowLeftInner         y
## 20                                477.7        eyeLeftOuter         x
## 21                                168.8        eyeLeftOuter         y
## 22                                491.2          eyeLeftTop         x
## 23                                160.6          eyeLeftTop         y
## 24                                491.9       eyeLeftBottom         x
## 25                                171.2       eyeLeftBottom         y
## 26                                506.4        eyeLeftInner         x
## 27                                165.7        eyeLeftInner         y
## 28                                550.6   eyebrowRightInner         x
## 29                                144.5   eyebrowRightInner         y
## 30                                614.3   eyebrowRightOuter         x
## 31                                140.9   eyebrowRightOuter         y
## 32                                566.3       eyeRightInner         x
## 33                                161.8       eyeRightInner         y
## 34                                  581         eyeRightTop         x
## 35                                154.8         eyeRightTop         y
## 36                                581.6      eyeRightBottom         x
## 37                                165.5      eyeRightBottom         y
## 38                                597.6       eyeRightOuter         x
## 39                                  161       eyeRightOuter         y
## 40                                519.8        noseRootLeft         x
## 41                                165.1        noseRootLeft         y
## 42                                543.8       noseRootRight         x
## 43                                163.9       noseRootRight         y
## 44                                512.3     noseLeftAlarTop         x
## 45                                195.9     noseLeftAlarTop         y
## 46                                551.5    noseRightAlarTop         x
## 47                                193.5    noseRightAlarTop         y
## 48                                503.9  noseLeftAlarOutTip         x
## 49                                  216  noseLeftAlarOutTip         y
## 50                                564.4 noseRightAlarOutTip         x
## 51                                214.8 noseRightAlarOutTip         y
## 52                                532.3         upperLipTop         x
## 53                                253.8         upperLipTop         y
## 54                                532.9      upperLipBottom         x
## 55                                261.7      upperLipBottom         y
## 56                                534.1         underLipTop         x
## 57                                264.5         underLipTop         y
## 58                                533.9      underLipBottom         x
## 59                                278.3      underLipBottom         y
## 60                                    0               smile      <NA>
## 61                                    1              gender      <NA>
## 62                                   59                 age      <NA>
## 63                                    1             glasses      <NA>
## 64                                0.055             emotion     anger
## 65                                0.019             emotion  contempt
## 66                                    0             emotion   disgust
## 67                                    0             emotion      fear
## 68                                    0             emotion happiness
## 69                                0.828             emotion   neutral
## 70                                0.098             emotion   sadness
## 71                                    0             emotion  surprise
## 72                                    0              makeup eyeMakeup
## 73                                    0              makeup lipMakeup
## 74                                  0.1                hair      bald
## 75                                    0                hair invisible
## 76                                 gray                hair hairColor
## 77                                 0.97                hair hairColor
## 78                                black                hair hairColor
## 79                                 0.62                hair hairColor
## 80                                blond                hair hairColor
## 81                                 0.61                hair hairColor
## 82                                brown                hair hairColor
## 83                                 0.53                hair hairColor
## 84                                other                hair hairColor
## 85                                 0.29                hair hairColor
## 86                                  red                hair hairColor
## 87                                 0.02                hair hairColor
##            L6 L5             L2 L1
## 1        <NA> NA         faceId  1
## 2        <NA> NA  faceRectangle  1
## 3        <NA> NA  faceRectangle  1
## 4        <NA> NA  faceRectangle  1
## 5        <NA> NA  faceRectangle  1
## 6        <NA> NA  faceLandmarks  1
## 7        <NA> NA  faceLandmarks  1
## 8        <NA> NA  faceLandmarks  1
## 9        <NA> NA  faceLandmarks  1
## 10       <NA> NA  faceLandmarks  1
## 11       <NA> NA  faceLandmarks  1
## 12       <NA> NA  faceLandmarks  1
## 13       <NA> NA  faceLandmarks  1
## 14       <NA> NA  faceLandmarks  1
## 15       <NA> NA  faceLandmarks  1
## 16       <NA> NA  faceLandmarks  1
## 17       <NA> NA  faceLandmarks  1
## 18       <NA> NA  faceLandmarks  1
## 19       <NA> NA  faceLandmarks  1
## 20       <NA> NA  faceLandmarks  1
## 21       <NA> NA  faceLandmarks  1
## 22       <NA> NA  faceLandmarks  1
## 23       <NA> NA  faceLandmarks  1
## 24       <NA> NA  faceLandmarks  1
## 25       <NA> NA  faceLandmarks  1
## 26       <NA> NA  faceLandmarks  1
## 27       <NA> NA  faceLandmarks  1
## 28       <NA> NA  faceLandmarks  1
## 29       <NA> NA  faceLandmarks  1
## 30       <NA> NA  faceLandmarks  1
## 31       <NA> NA  faceLandmarks  1
## 32       <NA> NA  faceLandmarks  1
## 33       <NA> NA  faceLandmarks  1
## 34       <NA> NA  faceLandmarks  1
## 35       <NA> NA  faceLandmarks  1
## 36       <NA> NA  faceLandmarks  1
## 37       <NA> NA  faceLandmarks  1
## 38       <NA> NA  faceLandmarks  1
## 39       <NA> NA  faceLandmarks  1
## 40       <NA> NA  faceLandmarks  1
## 41       <NA> NA  faceLandmarks  1
## 42       <NA> NA  faceLandmarks  1
## 43       <NA> NA  faceLandmarks  1
## 44       <NA> NA  faceLandmarks  1
## 45       <NA> NA  faceLandmarks  1
## 46       <NA> NA  faceLandmarks  1
## 47       <NA> NA  faceLandmarks  1
## 48       <NA> NA  faceLandmarks  1
## 49       <NA> NA  faceLandmarks  1
## 50       <NA> NA  faceLandmarks  1
## 51       <NA> NA  faceLandmarks  1
## 52       <NA> NA  faceLandmarks  1
## 53       <NA> NA  faceLandmarks  1
## 54       <NA> NA  faceLandmarks  1
## 55       <NA> NA  faceLandmarks  1
## 56       <NA> NA  faceLandmarks  1
## 57       <NA> NA  faceLandmarks  1
## 58       <NA> NA  faceLandmarks  1
## 59       <NA> NA  faceLandmarks  1
## 60       <NA> NA faceAttributes  1
## 61       <NA> NA faceAttributes  1
## 62       <NA> NA faceAttributes  1
## 63       <NA> NA faceAttributes  1
## 64       <NA> NA faceAttributes  1
## 65       <NA> NA faceAttributes  1
## 66       <NA> NA faceAttributes  1
## 67       <NA> NA faceAttributes  1
## 68       <NA> NA faceAttributes  1
## 69       <NA> NA faceAttributes  1
## 70       <NA> NA faceAttributes  1
## 71       <NA> NA faceAttributes  1
## 72       <NA> NA faceAttributes  1
## 73       <NA> NA faceAttributes  1
## 74       <NA> NA faceAttributes  1
## 75       <NA> NA faceAttributes  1
## 76      color  1 faceAttributes  1
## 77 confidence  1 faceAttributes  1
## 78      color  2 faceAttributes  1
## 79 confidence  2 faceAttributes  1
## 80      color  3 faceAttributes  1
## 81 confidence  3 faceAttributes  1
## 82      color  4 faceAttributes  1
## 83 confidence  4 faceAttributes  1
## 84      color  5 faceAttributes  1
## 85 confidence  5 faceAttributes  1
## 86      color  6 faceAttributes  1
## 87 confidence  6 faceAttributes  1

5.3.2 Find similar faces

First, run each face. Note that a face is saved for 24 hours.

## first face
kelly1 <- upload_file('figures/kelly1.jpg')
kelly1.json.out <- POST(apiurl, 
              add_headers(`Ocp-Apim-Subscription-Key` = facekey, 
                          `Content-Type` = "application/octet-stream"),
                          body = kelly1
                  )

kelly1.out <- kelly1.json.out %>%
  content()

kelly2 <- upload_file('figures/kelly2.png')

kelly2.json.out <- POST(apiurl, 
              add_headers(`Ocp-Apim-Subscription-Key` = facekey, 
                          `Content-Type` = "application/octet-stream"),
                          body = kelly2
                  )

kelly2.out <- kelly2.json.out %>%
  content()

Second, verify if the two faces are the same. It only requires the two face ids in the json format.

apiurl.verify <- "https://westus.api.cognitive.microsoft.com/face/v1.0/verify"

body <- list(faceId1 = kelly1.out[[1]]$faceId, faceId2 = kelly2.out[[1]]$faceId)
body <- toJSON(body, auto_unbox = TRUE)

kelly.compare.out <- POST(apiurl.verify, 
              add_headers(`Ocp-Apim-Subscription-Key` = facekey, 
                          `Content-Type` = "application/json"),
                          body = body
                  )

kelly.compare.out %>%
  content()
## $isIdentical
## [1] TRUE
## 
## $confidence
## [1] 0.78278

For the comparison of the two photos from Kelly, there is a 78% confidence that they are the same.

5.4 Text/handwriting detection

A useful application of image analysis is to detect texts. This can be done using the Vision API of Microsoft. As an example, we recognize the text on the stick note below.

The API actually uses a two-step procedure. In the first step, we upload the image file for analysis. The output includes an operation id that can be used to retrieve the detected text.

Step 1.

visionkey <- 'ad2bd2caa95f4c9885712bcd25074f25'

url.vision <- "https://westus.api.cognitive.microsoft.com/vision/v1.0/recognizeText?handwriting=true"

#url.vision <- "https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Categories"

hw <- upload_file('figures/handwriting.jpg')
hw.json.out <- POST(url.vision, 
              add_headers(`Ocp-Apim-Subscription-Key` = visionkey, 
                          `Content-Type` = "application/octet-stream"),
                          body = hw
                  )

str(hw.json.out)
## List of 10
##  $ url        : chr "https://westus.api.cognitive.microsoft.com/vision/v1.0/recognizeText?handwriting=true"
##  $ status_code: int 202
##  $ headers    :List of 11
##   ..$ cache-control            : chr "no-cache"
##   ..$ pragma                   : chr "no-cache"
##   ..$ content-length           : chr "0"
##   ..$ expires                  : chr "-1"
##   ..$ operation-location       : chr "https://westus.api.cognitive.microsoft.com/vision/v1.0/textOperations/87ca3f6e-292a-427c-8fbf-0be47f6510cf"
##   ..$ x-aspnet-version         : chr "4.0.30319"
##   ..$ x-powered-by             : chr "ASP.NET"
##   ..$ apim-request-id          : chr "677663ac-b40a-424a-a538-bdc46f3c272a"
##   ..$ strict-transport-security: chr "max-age=31536000; includeSubDomains; preload"
##   ..$ x-content-type-options   : chr "nosniff"
##   ..$ date                     : chr "Mon, 08 Apr 2019 21:09:46 GMT"
##   ..- attr(*, "class")= chr [1:2] "insensitive" "list"
##  $ all_headers:List of 1
##   ..$ :List of 3
##   .. ..$ status : int 202
##   .. ..$ version: chr "HTTP/1.1"
##   .. ..$ headers:List of 11
##   .. .. ..$ cache-control            : chr "no-cache"
##   .. .. ..$ pragma                   : chr "no-cache"
##   .. .. ..$ content-length           : chr "0"
##   .. .. ..$ expires                  : chr "-1"
##   .. .. ..$ operation-location       : chr "https://westus.api.cognitive.microsoft.com/vision/v1.0/textOperations/87ca3f6e-292a-427c-8fbf-0be47f6510cf"
##   .. .. ..$ x-aspnet-version         : chr "4.0.30319"
##   .. .. ..$ x-powered-by             : chr "ASP.NET"
##   .. .. ..$ apim-request-id          : chr "677663ac-b40a-424a-a538-bdc46f3c272a"
##   .. .. ..$ strict-transport-security: chr "max-age=31536000; includeSubDomains; preload"
##   .. .. ..$ x-content-type-options   : chr "nosniff"
##   .. .. ..$ date                     : chr "Mon, 08 Apr 2019 21:09:46 GMT"
##   .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
##  $ cookies    :'data.frame': 0 obs. of  7 variables:
##   ..$ domain    : logi(0) 
##   ..$ flag      : logi(0) 
##   ..$ path      : logi(0) 
##   ..$ secure    : logi(0) 
##   ..$ expiration: 'POSIXct' num(0) 
##   ..$ name      : logi(0) 
##   ..$ value     : logi(0) 
##  $ content    : raw(0) 
##  $ date       : POSIXct[1:1], format: "2019-04-08 21:09:46"
##  $ times      : Named num [1:6] 0e+00 1e-06 1e-06 1e-06 1e-06 ...
##   ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
##  $ request    :List of 7
##   ..$ method    : chr "POST"
##   ..$ url       : chr "https://westus.api.cognitive.microsoft.com/vision/v1.0/recognizeText?handwriting=true"
##   ..$ headers   : Named chr [1:3] "application/json, text/xml, application/xml, */*" "ad2bd2caa95f4c9885712bcd25074f25" "application/octet-stream"
##   .. ..- attr(*, "names")= chr [1:3] "Accept" "Ocp-Apim-Subscription-Key" "Content-Type"
##   ..$ fields    : NULL
##   ..$ options   :List of 6
##   .. ..$ useragent          : chr "libcurl/7.59.0 r-curl/3.3 httr/1.4.0"
##   .. ..$ cainfo             : chr "C:/PROGRA~1/R/R-35~1.2/etc/curl-ca-bundle.crt"
##   .. ..$ http_version       : num 0
##   .. ..$ post               : logi TRUE
##   .. ..$ readfunction       :function (nbytes, ...)  
##   .. ..$ postfieldsize_large: num 860357
##   ..$ auth_token: NULL
##   ..$ output    : list()
##   .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
##   ..- attr(*, "class")= chr "request"
##  $ handle     :Class 'curl_handle' <externalptr> 
##  - attr(*, "class")= chr "response"

Step 2. We get the text in this step.


## now call again to get the output
operationId.url <- hw.json.out$headers$`operation-location`

hw.json.text <- GET(operationId.url, 
              add_headers(`Ocp-Apim-Subscription-Key` = visionkey)
                  )

hw.text <- hw.json.text %>% content()

all.text <- NULL

for (i in 1:length(hw.text$recognitionResult$lines)){
  all.text <- paste(all.text, 
                    hw.text$recognitionResult$lines[[i]]$text)
}

all.text
## character(0)

5.5 Practice

Run and understand the code and output below. The photo used here is:

5.5.1 1

facekey <- "5b336abc1ae94484a21cead03baedccc"
jenkins <- upload_file('figures/jenkins.jpg')
apiurl <- "https://westus.api.cognitive.microsoft.com/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true&returnFaceAttributes=age,gender,smile,hair,makeup,glasses,emotion"

jenkins.json.out <- POST(apiurl, 
              add_headers(`Ocp-Apim-Subscription-Key` = facekey, 
                          `Content-Type` = "application/octet-stream"),
                          body = jenkins
                  )

jenkins.out <- jenkins.json.out %>%
  content()

str(jenkins.out)
## List of 3
##  $ :List of 4
##   ..$ faceId        : chr "557d6bb7-ab40-47be-a82a-94ab97442054"
##   ..$ faceRectangle :List of 4
##   .. ..$ top   : int 118
##   .. ..$ left  : int 301
##   .. ..$ width : int 51
##   .. ..$ height: int 51
##   ..$ faceLandmarks :List of 27
##   .. ..$ pupilLeft          :List of 2
##   .. .. ..$ x: num 315
##   .. .. ..$ y: num 134
##   .. ..$ pupilRight         :List of 2
##   .. .. ..$ x: num 339
##   .. .. ..$ y: num 134
##   .. ..$ noseTip            :List of 2
##   .. .. ..$ x: num 325
##   .. .. ..$ y: num 144
##   .. ..$ mouthLeft          :List of 2
##   .. .. ..$ x: num 316
##   .. .. ..$ y: num 156
##   .. ..$ mouthRight         :List of 2
##   .. .. ..$ x: num 340
##   .. .. ..$ y: num 156
##   .. ..$ eyebrowLeftOuter   :List of 2
##   .. .. ..$ x: num 308
##   .. .. ..$ y: num 128
##   .. ..$ eyebrowLeftInner   :List of 2
##   .. .. ..$ x: num 321
##   .. .. ..$ y: num 129
##   .. ..$ eyeLeftOuter       :List of 2
##   .. .. ..$ x: num 312
##   .. .. ..$ y: num 134
##   .. ..$ eyeLeftTop         :List of 2
##   .. .. ..$ x: num 316
##   .. .. ..$ y: num 133
##   .. ..$ eyeLeftBottom      :List of 2
##   .. .. ..$ x: num 316
##   .. .. ..$ y: num 135
##   .. ..$ eyeLeftInner       :List of 2
##   .. .. ..$ x: num 319
##   .. .. ..$ y: num 134
##   .. ..$ eyebrowRightInner  :List of 2
##   .. .. ..$ x: num 332
##   .. .. ..$ y: num 129
##   .. ..$ eyebrowRightOuter  :List of 2
##   .. .. ..$ x: num 349
##   .. .. ..$ y: num 130
##   .. ..$ eyeRightInner      :List of 2
##   .. .. ..$ x: num 335
##   .. .. ..$ y: num 134
##   .. ..$ eyeRightTop        :List of 2
##   .. .. ..$ x: num 340
##   .. .. ..$ y: num 133
##   .. ..$ eyeRightBottom     :List of 2
##   .. .. ..$ x: num 339
##   .. .. ..$ y: num 135
##   .. ..$ eyeRightOuter      :List of 2
##   .. .. ..$ x: num 344
##   .. .. ..$ y: num 135
##   .. ..$ noseRootLeft       :List of 2
##   .. .. ..$ x: num 323
##   .. .. ..$ y: num 134
##   .. ..$ noseRootRight      :List of 2
##   .. .. ..$ x: num 330
##   .. .. ..$ y: num 134
##   .. ..$ noseLeftAlarTop    :List of 2
##   .. .. ..$ x: num 320
##   .. .. ..$ y: num 140
##   .. ..$ noseRightAlarTop   :List of 2
##   .. .. ..$ x: num 332
##   .. .. ..$ y: num 141
##   .. ..$ noseLeftAlarOutTip :List of 2
##   .. .. ..$ x: num 316
##   .. .. ..$ y: num 145
##   .. ..$ noseRightAlarOutTip:List of 2
##   .. .. ..$ x: num 336
##   .. .. ..$ y: num 146
##   .. ..$ upperLipTop        :List of 2
##   .. .. ..$ x: num 326
##   .. .. ..$ y: num 153
##   .. ..$ upperLipBottom     :List of 2
##   .. .. ..$ x: num 326
##   .. .. ..$ y: num 155
##   .. ..$ underLipTop        :List of 2
##   .. .. ..$ x: num 326
##   .. .. ..$ y: num 160
##   .. ..$ underLipBottom     :List of 2
##   .. .. ..$ x: num 326
##   .. .. ..$ y: num 164
##   ..$ faceAttributes:List of 7
##   .. ..$ smile  : num 1
##   .. ..$ gender : chr "male"
##   .. ..$ age    : num 43
##   .. ..$ glasses: chr "NoGlasses"
##   .. ..$ emotion:List of 8
##   .. .. ..$ anger    : num 0
##   .. .. ..$ contempt : num 0
##   .. .. ..$ disgust  : num 0
##   .. .. ..$ fear     : num 0
##   .. .. ..$ happiness: num 1
##   .. .. ..$ neutral  : num 0
##   .. .. ..$ sadness  : num 0
##   .. .. ..$ surprise : num 0
##   .. ..$ makeup :List of 2
##   .. .. ..$ eyeMakeup: logi FALSE
##   .. .. ..$ lipMakeup: logi FALSE
##   .. ..$ hair   :List of 3
##   .. .. ..$ bald     : num 0.56
##   .. .. ..$ invisible: logi FALSE
##   .. .. ..$ hairColor:List of 6
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "gray"
##   .. .. .. .. ..$ confidence: num 0.98
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "black"
##   .. .. .. .. ..$ confidence: num 0.97
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "other"
##   .. .. .. .. ..$ confidence: num 0.51
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "blond"
##   .. .. .. .. ..$ confidence: num 0.44
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "brown"
##   .. .. .. .. ..$ confidence: num 0.14
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "red"
##   .. .. .. .. ..$ confidence: num 0
##  $ :List of 4
##   ..$ faceId        : chr "eab161da-2228-4433-99cd-219388a4ee66"
##   ..$ faceRectangle :List of 4
##   .. ..$ top   : int 165
##   .. ..$ left  : int 115
##   .. ..$ width : int 50
##   .. ..$ height: int 50
##   ..$ faceLandmarks :List of 27
##   .. ..$ pupilLeft          :List of 2
##   .. .. ..$ x: num 129
##   .. .. ..$ y: num 180
##   .. ..$ pupilRight         :List of 2
##   .. .. ..$ x: num 151
##   .. .. ..$ y: num 179
##   .. ..$ noseTip            :List of 2
##   .. .. ..$ x: num 140
##   .. .. ..$ y: num 190
##   .. ..$ mouthLeft          :List of 2
##   .. .. ..$ x: num 128
##   .. .. ..$ y: num 201
##   .. ..$ mouthRight         :List of 2
##   .. .. ..$ x: num 152
##   .. .. ..$ y: num 201
##   .. ..$ eyebrowLeftOuter   :List of 2
##   .. .. ..$ x: num 120
##   .. .. ..$ y: num 176
##   .. ..$ eyebrowLeftInner   :List of 2
##   .. .. ..$ x: num 135
##   .. .. ..$ y: num 175
##   .. ..$ eyeLeftOuter       :List of 2
##   .. .. ..$ x: num 126
##   .. .. ..$ y: num 180
##   .. ..$ eyeLeftTop         :List of 2
##   .. .. ..$ x: num 129
##   .. .. ..$ y: num 178
##   .. ..$ eyeLeftBottom      :List of 2
##   .. .. ..$ x: num 129
##   .. .. ..$ y: num 181
##   .. ..$ eyeLeftInner       :List of 2
##   .. .. ..$ x: num 133
##   .. .. ..$ y: num 180
##   .. ..$ eyebrowRightInner  :List of 2
##   .. .. ..$ x: num 145
##   .. .. ..$ y: num 176
##   .. ..$ eyebrowRightOuter  :List of 2
##   .. .. ..$ x: num 159
##   .. .. ..$ y: num 177
##   .. ..$ eyeRightInner      :List of 2
##   .. .. ..$ x: num 148
##   .. .. ..$ y: num 180
##   .. ..$ eyeRightTop        :List of 2
##   .. .. ..$ x: num 150
##   .. .. ..$ y: num 178
##   .. ..$ eyeRightBottom     :List of 2
##   .. .. ..$ x: num 151
##   .. .. ..$ y: num 180
##   .. ..$ eyeRightOuter      :List of 2
##   .. .. ..$ x: num 154
##   .. .. ..$ y: num 180
##   .. ..$ noseRootLeft       :List of 2
##   .. .. ..$ x: num 137
##   .. .. ..$ y: num 181
##   .. ..$ noseRootRight      :List of 2
##   .. .. ..$ x: num 143
##   .. .. ..$ y: num 181
##   .. ..$ noseLeftAlarTop    :List of 2
##   .. .. ..$ x: num 135
##   .. .. ..$ y: num 186
##   .. ..$ noseRightAlarTop   :List of 2
##   .. .. ..$ x: num 144
##   .. .. ..$ y: num 186
##   .. ..$ noseLeftAlarOutTip :List of 2
##   .. .. ..$ x: num 132
##   .. .. ..$ y: num 190
##   .. ..$ noseRightAlarOutTip:List of 2
##   .. .. ..$ x: num 148
##   .. .. ..$ y: num 190
##   .. ..$ upperLipTop        :List of 2
##   .. .. ..$ x: num 140
##   .. .. ..$ y: num 198
##   .. ..$ upperLipBottom     :List of 2
##   .. .. ..$ x: num 140
##   .. .. ..$ y: num 200
##   .. ..$ underLipTop        :List of 2
##   .. .. ..$ x: num 140
##   .. .. ..$ y: num 202
##   .. ..$ underLipBottom     :List of 2
##   .. .. ..$ x: num 140
##   .. .. ..$ y: num 205
##   ..$ faceAttributes:List of 7
##   .. ..$ smile  : num 1
##   .. ..$ gender : chr "male"
##   .. ..$ age    : num 66
##   .. ..$ glasses: chr "ReadingGlasses"
##   .. ..$ emotion:List of 8
##   .. .. ..$ anger    : num 0
##   .. .. ..$ contempt : num 0
##   .. .. ..$ disgust  : num 0
##   .. .. ..$ fear     : num 0
##   .. .. ..$ happiness: num 1
##   .. .. ..$ neutral  : num 0
##   .. .. ..$ sadness  : num 0
##   .. .. ..$ surprise : num 0
##   .. ..$ makeup :List of 2
##   .. .. ..$ eyeMakeup: logi FALSE
##   .. .. ..$ lipMakeup: logi TRUE
##   .. ..$ hair   :List of 3
##   .. .. ..$ bald     : num 0.04
##   .. .. ..$ invisible: logi FALSE
##   .. .. ..$ hairColor:List of 6
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "gray"
##   .. .. .. .. ..$ confidence: num 0.99
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "other"
##   .. .. .. .. ..$ confidence: num 0.74
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "black"
##   .. .. .. .. ..$ confidence: num 0.59
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "blond"
##   .. .. .. .. ..$ confidence: num 0.5
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "brown"
##   .. .. .. .. ..$ confidence: num 0.05
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "red"
##   .. .. .. .. ..$ confidence: num 0.03
##  $ :List of 4
##   ..$ faceId        : chr "a26ff819-0dcc-4a7a-a5ed-4f7de87dd597"
##   ..$ faceRectangle :List of 4
##   .. ..$ top   : int 128
##   .. ..$ left  : int 437
##   .. ..$ width : int 50
##   .. ..$ height: int 50
##   ..$ faceLandmarks :List of 27
##   .. ..$ pupilLeft          :List of 2
##   .. .. ..$ x: num 450
##   .. .. ..$ y: num 144
##   .. ..$ pupilRight         :List of 2
##   .. .. ..$ x: num 472
##   .. .. ..$ y: num 140
##   .. ..$ noseTip            :List of 2
##   .. .. ..$ x: num 461
##   .. .. ..$ y: num 152
##   .. ..$ mouthLeft          :List of 2
##   .. .. ..$ x: num 454
##   .. .. ..$ y: num 167
##   .. ..$ mouthRight         :List of 2
##   .. .. ..$ x: num 475
##   .. .. ..$ y: num 163
##   .. ..$ eyebrowLeftOuter   :List of 2
##   .. .. ..$ x: num 442
##   .. .. ..$ y: num 141
##   .. ..$ eyebrowLeftInner   :List of 2
##   .. .. ..$ x: num 454
##   .. .. ..$ y: num 138
##   .. ..$ eyeLeftOuter       :List of 2
##   .. .. ..$ x: num 447
##   .. .. ..$ y: num 146
##   .. ..$ eyeLeftTop         :List of 2
##   .. .. ..$ x: num 449
##   .. .. ..$ y: num 143
##   .. ..$ eyeLeftBottom      :List of 2
##   .. .. ..$ x: num 450
##   .. .. ..$ y: num 146
##   .. ..$ eyeLeftInner       :List of 2
##   .. .. ..$ x: num 454
##   .. .. ..$ y: num 144
##   .. ..$ eyebrowRightInner  :List of 2
##   .. .. ..$ x: num 467
##   .. .. ..$ y: num 135
##   .. ..$ eyebrowRightOuter  :List of 2
##   .. .. ..$ x: num 479
##   .. .. ..$ y: num 134
##   .. ..$ eyeRightInner      :List of 2
##   .. .. ..$ x: num 468
##   .. .. ..$ y: num 141
##   .. ..$ eyeRightTop        :List of 2
##   .. .. ..$ x: num 471
##   .. .. ..$ y: num 138
##   .. ..$ eyeRightBottom     :List of 2
##   .. .. ..$ x: num 472
##   .. .. ..$ y: num 141
##   .. ..$ eyeRightOuter      :List of 2
##   .. .. ..$ x: num 475
##   .. .. ..$ y: num 140
##   .. ..$ noseRootLeft       :List of 2
##   .. .. ..$ x: num 457
##   .. .. ..$ y: num 144
##   .. ..$ noseRootRight      :List of 2
##   .. .. ..$ x: num 464
##   .. .. ..$ y: num 142
##   .. ..$ noseLeftAlarTop    :List of 2
##   .. .. ..$ x: num 456
##   .. .. ..$ y: num 150
##   .. ..$ noseRightAlarTop   :List of 2
##   .. .. ..$ x: num 466
##   .. .. ..$ y: num 149
##   .. ..$ noseLeftAlarOutTip :List of 2
##   .. .. ..$ x: num 454
##   .. .. ..$ y: num 155
##   .. ..$ noseRightAlarOutTip:List of 2
##   .. .. ..$ x: num 469
##   .. .. ..$ y: num 152
##   .. ..$ upperLipTop        :List of 2
##   .. .. ..$ x: num 463
##   .. .. ..$ y: num 162
##   .. ..$ upperLipBottom     :List of 2
##   .. .. ..$ x: num 463
##   .. .. ..$ y: num 164
##   .. ..$ underLipTop        :List of 2
##   .. .. ..$ x: num 463
##   .. .. ..$ y: num 166
##   .. ..$ underLipBottom     :List of 2
##   .. .. ..$ x: num 463
##   .. .. ..$ y: num 169
##   ..$ faceAttributes:List of 7
##   .. ..$ smile  : num 0.79
##   .. ..$ gender : chr "male"
##   .. ..$ age    : num 55
##   .. ..$ glasses: chr "ReadingGlasses"
##   .. ..$ emotion:List of 8
##   .. .. ..$ anger    : num 0
##   .. .. ..$ contempt : num 0.001
##   .. .. ..$ disgust  : num 0
##   .. .. ..$ fear     : num 0
##   .. .. ..$ happiness: num 0.79
##   .. .. ..$ neutral  : num 0.209
##   .. .. ..$ sadness  : num 0.001
##   .. .. ..$ surprise : num 0
##   .. ..$ makeup :List of 2
##   .. .. ..$ eyeMakeup: logi FALSE
##   .. .. ..$ lipMakeup: logi FALSE
##   .. ..$ hair   :List of 3
##   .. .. ..$ bald     : num 0.04
##   .. .. ..$ invisible: logi FALSE
##   .. .. ..$ hairColor:List of 6
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "gray"
##   .. .. .. .. ..$ confidence: num 0.96
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "brown"
##   .. .. .. .. ..$ confidence: num 0.89
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "blond"
##   .. .. .. .. ..$ confidence: num 0.51
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "black"
##   .. .. .. .. ..$ confidence: num 0.49
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "other"
##   .. .. .. .. ..$ confidence: num 0.14
##   .. .. .. ..$ :List of 2
##   .. .. .. .. ..$ color     : chr "red"
##   .. .. .. .. ..$ confidence: num 0.03

5.5.2 2 Try a new analysis

visionkey <- 'ad2bd2caa95f4c9885712bcd25074f25'

url.vision <- "https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Categories,Tags,Description,Faces&details=Celebrities"

jenkins2.json.out <- POST(url.vision, 
              add_headers(`Ocp-Apim-Subscription-Key` = visionkey, 
                          `Content-Type` = "application/octet-stream"),
                          body = jenkins
                  )

jenkins2.json.out %>% 
  content() %>%
  str()
## List of 6
##  $ categories :List of 2
##   ..$ :List of 3
##   .. ..$ name  : chr "people_group"
##   .. ..$ score : num 0.23
##   .. ..$ detail:List of 1
##   .. .. ..$ celebrities:List of 2
##   .. .. .. ..$ :List of 3
##   .. .. .. .. ..$ name         : chr "Barack Obama"
##   .. .. .. .. ..$ confidence   : num 1
##   .. .. .. .. ..$ faceRectangle:List of 4
##   .. .. .. .. .. ..$ left  : int 301
##   .. .. .. .. .. ..$ top   : int 118
##   .. .. .. .. .. ..$ width : int 51
##   .. .. .. .. .. ..$ height: int 51
##   .. .. .. ..$ :List of 3
##   .. .. .. .. ..$ name         : chr "John I. Jenkins"
##   .. .. .. .. ..$ confidence   : num 1
##   .. .. .. .. ..$ faceRectangle:List of 4
##   .. .. .. .. .. ..$ left  : int 437
##   .. .. .. .. .. ..$ top   : int 128
##   .. .. .. .. .. ..$ width : int 50
##   .. .. .. .. .. ..$ height: int 50
##   ..$ :List of 3
##   .. ..$ name  : chr "people_show"
##   .. ..$ score : num 0.469
##   .. ..$ detail:List of 1
##   .. .. ..$ celebrities:List of 2
##   .. .. .. ..$ :List of 3
##   .. .. .. .. ..$ name         : chr "Barack Obama"
##   .. .. .. .. ..$ confidence   : num 1
##   .. .. .. .. ..$ faceRectangle:List of 4
##   .. .. .. .. .. ..$ left  : int 301
##   .. .. .. .. .. ..$ top   : int 118
##   .. .. .. .. .. ..$ width : int 51
##   .. .. .. .. .. ..$ height: int 51
##   .. .. .. ..$ :List of 3
##   .. .. .. .. ..$ name         : chr "John I. Jenkins"
##   .. .. .. .. ..$ confidence   : num 1
##   .. .. .. .. ..$ faceRectangle:List of 4
##   .. .. .. .. .. ..$ left  : int 437
##   .. .. .. .. .. ..$ top   : int 128
##   .. .. .. .. .. ..$ width : int 50
##   .. .. .. .. .. ..$ height: int 50
##  $ tags       :List of 2
##   ..$ :List of 2
##   .. ..$ name      : chr "person"
##   .. ..$ confidence: num 0.988
##   ..$ :List of 2
##   .. ..$ name      : chr "commencement"
##   .. ..$ confidence: num 0.988
##  $ description:List of 2
##   ..$ tags    :List of 19
##   .. ..$ : chr "person"
##   .. ..$ : chr "building"
##   .. ..$ : chr "man"
##   .. ..$ : chr "uniform"
##   .. ..$ : chr "group"
##   .. ..$ : chr "standing"
##   .. ..$ : chr "blue"
##   .. ..$ : chr "military"
##   .. ..$ : chr "people"
##   .. ..$ : chr "woman"
##   .. ..$ : chr "walking"
##   .. ..$ : chr "large"
##   .. ..$ : chr "soccer"
##   .. ..$ : chr "wearing"
##   .. ..$ : chr "street"
##   .. ..$ : chr "city"
##   .. ..$ : chr "holding"
##   .. ..$ : chr "shirt"
##   .. ..$ : chr "game"
##   ..$ captions:List of 1
##   .. ..$ :List of 2
##   .. .. ..$ text      : chr "Barack Obama, John I. Jenkins in uniform"
##   .. .. ..$ confidence: num 0.861
##  $ faces      :List of 3
##   ..$ :List of 3
##   .. ..$ age          : int 43
##   .. ..$ gender       : chr "Male"
##   .. ..$ faceRectangle:List of 4
##   .. .. ..$ left  : int 301
##   .. .. ..$ top   : int 118
##   .. .. ..$ width : int 51
##   .. .. ..$ height: int 51
##   ..$ :List of 3
##   .. ..$ age          : int 66
##   .. ..$ gender       : chr "Male"
##   .. ..$ faceRectangle:List of 4
##   .. .. ..$ left  : int 115
##   .. .. ..$ top   : int 165
##   .. .. ..$ width : int 50
##   .. .. ..$ height: int 50
##   ..$ :List of 3
##   .. ..$ age          : int 55
##   .. ..$ gender       : chr "Male"
##   .. ..$ faceRectangle:List of 4
##   .. .. ..$ left  : int 437
##   .. .. ..$ top   : int 128
##   .. .. ..$ width : int 50
##   .. .. ..$ height: int 50
##  $ requestId  : chr "415845af-df44-47f7-bb00-caf6b0e193b7"
##  $ metadata   :List of 3
##   ..$ width : int 650
##   ..$ height: int 489
##   ..$ format: chr "Jpeg"

Practical Data Processing for Social and Behavioral Research Using R