Expected Likelihood: Not a Good Metric for Gaussian Embeddings

The success of neural network methods in Natural Language Processing (NLP) tasks is fairly related to the widely use of word embeddings in which a word is represented as a point in a vector space. Previous research has shown that high quality word embeddings can improve the performance of NLP models. Since natural language is intrinsically ambiguous, different senses of a word should be in different positions of the vector space. So that representing a word as a single point in the vector space is problematic. Some recent work proposed to use Gaussian distributions to represent a word which is named Gaussian embeddings. This idea is very natural and proved to be useful in tasks like word similarity and word entailment. However, Gaussian embeddings are very difficult to train, especially for the commonly used loss function based on Expected Likelihood (EL). In this paper, we study the training process of Gaussian embeddings based on EL. Supported by our experimental results, we find that EL is not suitable for training Gaussian embeddings. Besides, we propose an effective algorithm to train Gaussian embeddings and improve the performance on word similarity tasks.