Can Human Intuition be infused with that of a machine?
We often see different Machine and Deep Learning models struggling to mimic human level performance. For example, we have not been able to create Vision Models capable of detecting objects, understanding relationships between them and summarizing the events in the image that equal any human’s capacity to perform such tasks. Have you ever wondered why?
Even after giving humongous amounts of data to our model, with different permutations of Hyperparameters, we still fail to achieve human-level performance. Its very clear to us that the model fails to capture the nuances that our brain is capable of capturing almost subconsciously. This happens because of the inclusion of prior multi faceted knowledge, a human gathers over time. This is termed as “Human Knowledge”. Even after trying out different tactics with our Machine and Deep Learning model, we still feel as if we are missing something. That something is the absence of Human Knowledge in our model. We humans are who we are because of what we have faced in our lives. We are us maybe because of how our lives have been shaped by our childhood, by our mistakes, by our experiences and by our way of looking at life.
Do you think a computer or any model that we have artificially manufactured is ever going to be capable of performing tasks like us? Do you think a machine can ever hope to reach our level of understanding?. The answer is neither a sound Yes nor a sound No. Let me explain why.
It is not a sound No, because there have been so many people trying to infuse the idea of context into our Machine and Deep learning models in order to make a machine learn the what, why and who in any scenario[8]. It is not a sound Yes because this is still a work in progress. Now, the question that you might be having is how do you make a machine grasp the idea of context in any given situation.
The answer is not so straightforward. There are many approaches towards introducing the idea of context to a model. One of the most well-known once is usage of multi-modal Knowledge Graphs. Knowledge graphs are representations that tries to mimic the real world relationships keeping their semantics intact. For example, by using <Subject, predicate, Object> triplet, a sentence like “man rides bicycle” is denoted as <“man”, “rides”, “bicycle”>.
Knowledge Graphs: Graph Based Knowledge Representations of real world relationships.
“A knowledge graph (i) mainly describes real world entities and their interrelations, organized in a graph, (ii) defines possible classes and relations of entities in a schema, (iii) allows for potentially interrelating arbitrary entities with each other and (iv) covers various topical domains.[1]”
There has been many Knowledge Graphs that have been constructed with due diligence and are quite famous because of their large scale interdisciplinary applicability. Some of the most well known ones are DBPedia, ConceptNet by MIT Media Lab, Google’s Knowledge Graph, etc. Some Knowledge Graphs are also multi-lingual which held add multi-faceted Knowledge to the Knowledge Graphs.
“Knowledge graphs could be envisaged as a network of all kind things which are relevant to a specific domain or to an organization. They are not limited to abstract concepts and relations but can also contain instances of things like documents and datasets.[2]”
One limitation of Knowledge Graphs is that they are not dynamic[9]. At least to the best of my knowledge, I have not come across a Knowledge Graph which is dynamic and automatically updates new information as it gains them. All Knowledge Graphs are static and need to be updated using human intervention. This is specially tricky in most cases because one needs to be careful about what kind of information are we adding to our graph. Specially in today’s world where fake data and misinformation is a huge problem, verifying the authenticity and technical soundness is extremely important. Thus, human intervention becomes a necessity and Knowledge Graphs remain static and are unable to acquire dynamic nature.
Knowledge Graphs are static and need human intervention to check authenticity of data added to it!
Knowledge Inspired learning need not be just Knowledge Graphs though. So many researchers are coming up with so many new and exciting ideas to infuse contextual information to our Machine and Deep learning Models.
Let us consider a computer vision task like Image Inpainting. (If you don’t know what is Image Inpainting then please have a look at my article “Different Computer Vision Tasks” to understand it better). For a task like Image Inpainting or Image Reconstruction, some people have come up with ideas that consider the image as a whole and try to grasp what is missing in the image. While others have focused on predicting the detailed context of the missing region of the image based on context of only surrounding pixels near the missing region by associating important weights to them along with priors[4]. Few others have tried to create a context encoder which has an Encoder-Decoder like framework where encoder captures the context of Image and projects it to a latent feature representation and the Decoder uses these representations to produce the missing image context[5]. Some people took a multi-modal approach which considers the language representation and language attributes of the image annotations along with visual representations and visual attributes of the image and proposed a joint model that combines knowledge from Image and text to understand context[6].
Few others have experimented with the idea of usage of Language Priors to add information[3] in the case of other computer vision tasks such as Visual Relationship Detection. One other idea that I came across is the idea of combining knowledge gained from Concept Net (Knowledge Graph) to a Neural Image Captioning Model, known as Concept Net Enhanced Neural Image captioning (CNET-NIC)[7] to help augment the information that the model has.
These were just few examples of how people have come up with novel approaches towards bridging the cap between human intuition and machine understanding.
I hope you found this article helpful.
Till next time!
Thank you for reading!
Resources:
[1] H. Paulheim. Knowledge Graph Refinement: A Survey of Approaches and Evaluation Methods. Semantic Web Journal, (Preprint):1–20, 2016.
[2] A. Blumauer. From Taxonomies over Ontologies to Knowledge Graphs, July 2014. https://blog.semanticweb.at/2014/07/15/from-taxonomies-over-ontologiesto-knowledge-graphs [August, 2016].
[3]Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016, October). Visual relationship detection with language priors. In European Conference on Computer Vision (pp. 852–869). Springer, Cham.
[4]Yeh, R. A., Chen, C., Yian Lim, T., Schwing, A. G., Hasegawa-Johnson, M., & Do, M. N. (2017). Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5485–5493).DBPedia: https://wiki.dbpedia.org/
[5]Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2536–2544).
[6]Garcia, N., Renoust, B., & Nakashima, Y. (2019). Understanding Art through Multi-Modal Retrieval in Paintings. arXiv preprint arXiv:1904.10615.
[7]Zhou, Y., Sun, Y., & Honavar, V. (2019, January). Improving image captioning by leveraging knowledge graphs. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 283–293). IEEE.
[8]Sheth, A., Perera, S., Wijeratne, S., & Thirunarayan, K. (2017, August). Knowledge will propel machine understanding of content: extrapolating from current examples. In Proceedings of the International Conference on Web Intelligence (pp. 1–9). ACM.
[9]Padhee, S., Lalithsena, S., & Sheth, A. P. (2018). Creating Real-Time Dynamic Knowledge Graphs.
More Resources:
ConceptNet: http://conceptnet.io/
NELL: Never-Ending Language Learning : http://rtw.ml.cmu.edu/rtw/index.php?
Knowledge Graph Definition: https://www.researchgate.net/profile/Wolfram_Woess/publication/323316736_Towards_a_Definition_of_Knowledge_Graphs/links/5a8d6e8f0f7e9b27c5b4b1c3/Towards-a-Definition-of-Knowledge-Graphs.pdf
Learning More about Knowledge Graphs:https://medium.com/@sderymail/challenges-of-knowledge-graph-part-1-d9ffe9e35214
