Using Python for Pixel-Based Analysis

Pixel-based analysis capitalizes on the fact that images are in fact numerical arrays and can thus be statistically analyzed. This example uses images from AT&T Labortories’ Olivetti dataset 1, 10 images from different perspectives of 40 people (N = 400). The gray-scale images are encoded on black-to-white luminosity scale from \(0 - 255\).

Pearson Correlation with Python

Python is similar to MATLAB in many respects (e.g., specializes in matrix operations), though it is an open-source software that makes it appealing to a wide variety of audiences and programmers. In this example we will use the Olivetti dataset in order to compare the Pearson correlation between two images from the same person (though from different perspectives) and the correlation with that of a different person but from the same perspective. The hypothesis is that the images from the same person but from different perspectives will have a higher correlation than images from different people from the same perspective.

Setting up Python Environment

If you are familiar with R, Python works along the same lines in that the relevant packages need to be installed and loaded before you can utilize the functions. In the code we first import the functions in the ‘preamble’ and then procede with the rest of the code. I will present the code in chunks for easier explanation/analysis. 

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from sklearn import linear_model, datasets, metrics
from sklearn.datasets import fetch_olivetti_faces
import numpy as np
from PIL import Image
from scipy import signal
from scipy import misc

#fetch the Olivetti dataset. Save data in X (64 x 64 pixels) and targets in y
Df = fetch_olivetti_faces(return_X_y=False) #
X, y = fetch_olivetti_faces(return_X_y=True)
X.shape#the appende

The first part of the code here imports the packages and the functions within the packages. Notice the dot ‘.’ formatting in the import statements. This is equivalent to the ‘$’ formatting in R and is a way of calling the sub-directories or functions within a package. The ‘from’ statement preceding the ‘import’ statement indicates the package from which we want to draw or extract the functions. For example, the statement ‘from sklearn.datasets import fetch_olivetti_faces’ tells us that from the sklearn package we will access the datasets subdirectory, and within that directory, access the fetch_olivetti_faces dataset. Here we create a dataframe ‘Df’ composed of X, the n x n pixel-matrix of the images, and the y vector of targets or labels (the outcome in traditional statistics).The .shape command is similar to the dim() command in R, which provides the dimensions of the object. Here, X.shape = (400, 4096), 400 rows (or separate images) with 4096 columns. In each entry in these columns is a pixel luminosity-scale value for the image. This image is a 64 x 64 pixel image, flattened into one row (64 x 64 = 4096). Now we will move on to the second segment of code. 

#extract first image from dataset
X1 = X[1,:]
X1.shape
X2 = np.reshape(X1,(64, 64))
X2.shape

#plot 2D array as image with matplotlib.pyplot
plt.imshow(X2, cmap="gray")
plt.xticks(())
plt.yticks(())
plt.savefig("Olive1.png", bbox_inches='tight')

#Call up image and save as object 'img1'. Not necessary for correlation
img1 = plt.imread(r'C:\Users\Mark V. Brow\Desktop\PythonDir\Olive1.png')

In the code above we are extracting the first image from this dataset. Recall, that each row of the dataset represents one image. Here, we are creating a subset of the original dataset with only the first row (i.e., the first image): X1 = X[1,:]. Next, we reshape the 4096 row vector into a 64 x 64 matrix so that the matplotlib command imshow can read it. Here is the first image:

Now let’s take a look at the second image. This image is of the same person, but from a different perspective. The code below is to extract the second image. 

#extract first image from dataset
X3 = X[2,:]
X3.shape
X4 = np.reshape(X3,(64, 64))
X4.shape

#plot 2D array as image with matplotlib.pyplot
plt.imshow(X4, cmap="gray")
plt.xticks(())
plt.yticks(())
plt.savefig("Olive2.png", bbox_inches='tight')

#Call up image and save as object 'img1'. Not necessary for correlation
img2 = plt.imread(r'C:\Users\Mark V. Brow\Desktop\PythonDir\Olive2.png')

Here is the second image:

Pearson Correlation Coefficient for Images

Now we will calculate the Pearson Correlation coefficient for this two images. Recall, that this image is simply composed of an array of numbers in which each pixel in the image corresponds to a value on the luminosity scale. The mean Pearson correlation coefficinet is .51, with the value of the first quartile .30 and the third quartile .73. Now let’s get an image from a different person but the same perspective as the first image presented. The code for extracting this image is presented below: 

#extract image from a different person in the dataset
X5 = X[101,:]
X5.shape
X6 = np.reshape(X5,(64, 64))
X6.shape

#plot 2D array as image with matplotlib.pyplot
plt.imshow(X6, cmap="gray")
plt.xticks(())
plt.yticks(())
plt.savefig("Olive3.png", bbox_inches='tight')

#Call up image and save as object 'img1'. Not necessary for correlation
img3 = plt.imread(r'C:\Users\Mark V. Brow\Desktop\PythonDir\Olive3.png')

Here is the image for that person:

Now let’s calculate the Pearson correlation coefficient for the first image and the second image. The code is presented below: 

#Now calculate Pearson correlation coefficient between both images from different people
corrIm = np.corrcoef(X2, X6)
corrImAbs = np.absolute(corrIm)
np.mean(corrImAbs)
np.max(corrImAbs)
np.median(corrImAbs)
# First quartile (Q1) 
Q1 = np.percentile(corrImAbs, 25, interpolation = 'midpoint') 
print(Q1)
# Third quartile (Q3) 
Q3 = np.percentile(corrImAbs, 75, interpolation = 'midpoint') 
print(Q3)
# Interquaritle range (IQR) 
IQR = Q3 - Q1 
print(IQR)

The mean Pearson correlation coefficient is .42, with the first quartile as .16 and the third quartile as .65. We see a lower correlation between these images (even though the perspectives are the same) than between the images from the same person from different perspectives.

Adding Filters

There is also a way of adding filters to the images in order to make features of the image more pronounced or distinct. Here we will scale the data from \(0-1\) and create a binary filter. 

#scale data
X = np.asarray( X, 'float32')
X = (X - np.min(X, 0)) / (np.max(X, 0) + 0.0001)  # 0-1 scaling

# Convert image array to binary with threshold. This will create black-and-white images
X = X > 0.5

Here are the images with the filer:

In this case, the filter does not improve the separation of the correlation coefficients and is meant merely for example.

References
  1. AT&T Laborties, Cambridge. (n.d.).