Applying deep learning for language-independent recognition of sentences representing reasons and grounds for opinions in text data

Tokyo, June 2, 2016 --- Hitachi, Ltd. (TSE: 6501; “Hitachi”) today announced that it has developed a basic artificial intelligence (AI) technology that analyzes huge volumes of Japanese text data on issues that are subject to debate, and presents in Japanese both affirmative and negative opinions on those issues together with reasons and grounds. In this research, Hitachi applied deep learning1 to the process of distinguishing sentences representing reasons and grounds for opinions, eliminating the need for a dedicated program to be prepared for each language and thus enabling the creation of a general-purpose system analyzing text data in any language. Previously, Hitachi developed a basic AI technology which analyzed huge volumes of English text data and presented opinions in English.2 This time, Hitachi incorporated this technology into a new AI technology for the Japanese language to meet the needs of Japanese enterprises.

Today, the social landscape changes rapidly and customer needs are becoming increasingly diversified. Companies are expected to continuously create new services and values. Further, driven by recent advancements in information & telecommunication and analytics technologies, interest is growing in technology that can extract valuable insight from big data which is generated on a daily basis.

Hitachi has been developing a basic AI technology that analyzes huge volumes of English text data and presents opinions in English to help enterprises make business decisions. The original technology required rules of grammar specific to the English language to be programmed, to extract sentences representing reasons and grounds for opinions. This process represented a hurdle in applying system to Japanese or any other language as it required dedicated programs correlated to the linguistic rules of the target language.

By applying deep learning, this issue was eliminated thus enabling the new technology to recognize sentences that have high probability of being reasons and grounds without relying on linguistic rules. More specifically, the AI system is presented with sentences which represent reasons and grounds extracted from thousands of articles. Learning from the rules and patterns, the system becomes discriminating of sentences which represent reasons and grounds in new articles. Hitachi added an "attention mechanism" which support deep learning to estimate which words and phrases are worthy of attention in texts like news articles and research reports. The "attention mechanism" helps the system to grasp the points that require attention, including words and phrases related to topics and values. This method enables the system to distinguish sentences which have a high probability of being reasons and grounds from text data in any language.

The technology developed will be core technology in achieving a multi-lingual AI system capable of offering opinion. Hitachi will pursue further research to realize AI systems supporting business decision making by enterprises worldwide.

This research achievement will be presented at the 30^th National Convention of the Japanese Society for Artificial Intelligence (JSAI 2016) to be held in Kitakyushu-shi, Japan from 6-9 June 2016. 1Deep Learning: A neural network machine learning model based on the mechanism of nerve cells. The structure of a neural network is comprised of 3 layers: an input layer, an intermediate layer and an output layer. In Deep Learning, the intermediate layer is increased to enable the expression of even more complex models than previously possible, achieving higher recognition rates in the field of voice and image recognition.2Hitachi News Release on July 22 2015: “Hitachi Developed Basic Artificial Intelligence Technology That Enables Logical Dialogue”<http://www.hitachi.com/New/cnews/month/2015/07/150722.html>