Last week we saw that image duplicacy and cropping the headlines does not seems to work, from this week onwards we will start with the neural network methods . But one of the interesting thing we noticed that the features used in the image duplicacy seems to be worked, thus My mentor Peter sir and I decided to go through the feature extraction algorithms and extract those features to see if that works in our case .
Generally there are various and interesting feature extraction algorithms that are been widely used in the domain of the computer vison
1) Edge Features:
A edge is generally the sharp change in the color , there are various types of edge detection algorithms which are:
A) perwit
B)sobel
C)canny
2)The other class of feature descriptors are :
A) HOG: Histogram of Oriented Gradients: It provide the edge directions in addition to the information provided by the edge features . It works by extracting the gradient and orientation .
B)FAST : It is a corner detection method, which could be used to extract feature points and later used to track and map objects in many computer vision tasks
C)BRIEF:
D)ORB:
The features are visualized in this notebook below
Feature_Extraction_without_pretrained.html The feature extraction code can be found in the github link as
https://github.com/Himani2000/GSOC_2020/blob/master/Feature_extraction/Images/image_feature_extraction.py I extracted the above features for both the cropped and the uncropped version
I ran these features through the agglomerative and the density based methods
The text file(please look into the previous week blog posts if you are not familiar about the text file being refereed there )
the number of files which cross the filter is 2 here:
concatenated_image_results_conventional_text_file.txt
The sample look at the pdf files:
1.ideology__hog_-density-tsne_dbscan_labels.pdf
2.ideology__hog_-density-umap_dbscan_labels.pdf
the number of files which cross the filter is 3 here :
concatenated_image_results_crop_conventional_text_file.txt
The sample look at the pdf files is:
1.crop_ideology__fast_-density-pca-tsne_dbscan_labels.pdf
2.crop_ideology__hog_-density-umap_dbscan_labels.pdf
3.crop_ideology__sobel_-density-tsne_dbscan_labels.pdf
Starting with Deep learning clustering techniques
1. Self Organizing map
Self Organizing map is a type of the artificial neural network based technique. It differs from other artificial network techniques as they apply competitive learning as opposed to the error-correction learning (backpropagation with gradient descent ). SOM starts by initializing the weight vectors , from there a sample vector is randomly selected, and a weight vector is searched with represents the sample best. The chosen weight vectors along with the neighbours of the weight vector are rewarded by being able to become more like the chosen sample vector .
More details can be found in here :
https://towardsdatascience.com/self-organizing-maps-ff5853a118d4
the code for the SOM can be found here
https://github.com/Himani2000/GSOC_2020/blob/master/clustering-models/som_clustering.py Results
1. Pretrained features - without crop
a) Text files
concatenated_som_pretrained.txt
b)Pdf files
ideology_model_resnet50-som-pca-tsne_som_.pdf
ideology_model_resnet50-som-tsne_som_.pdf
2. Conventional features --without crop

3. Pretrained features --with crop
a)Text files
concatenate_crop_som_pretrained.txt
b)Pdf files
ideology_crop_model_densenet169-som-tsne_som_.pdf
ideology_crop_model_mobilenet-som-tsne_som_.pdf
ideology_crop_model_resnet50-som-tsne_som_.pdf
ideology_crop_model_vgg16-som-pca-tsne_som_.pdf
ideology_crop_model_vgg19-som-pca-tsne_som_.pdf
4. Conventioanl features --without crop
a)Text files
concatenated_crop_som_conventional.txt
b)Pdf files
crop_ideology__hog_-som-pca-tsne_som_.pdf
And again to my surprise none of them passed the filter (again the details of the filter are provided in the previous week blog posts :) )
Now the few next possible steps seems to be
a)Using autoencoders as the dimensionality reduction technique and applying the hierarchical and density based algorithm above the reduced transformation .
b)Using the combined deep clustering methods such as Deep Embedded Clustering (DEC),Deep Clustering Network (DCN) and etc.
c)Extract the region of faces with some surrounding pixedl and then apply any of the clustering algoithm
d) Ensemble techniques to combine different clustering algorithms .
e)using attention based mechanisms - To find the person , the 0/1 weights will result into the class activation map( we can use the generalization of CAM known as Grad-Cam), then use a class-based clustering algorithm to divide the training data to several clusters with respective members belonging to the same class.
Also I will be integrating the bio metric clustering along with the fellow Gsoc mate in the redhen rapid annotator
See you next week