Why Meta Embeddings?

Representing the meanings of individual words is arguably one of the fundamental tasks in Natural Language Processing (NLP). If we can accurately represent the meanings of individual words then we can use those representations to compose the meanings of larger lexical units such as phrases, sentences, paragraphs or entire documents for that matter.

Fortunately, we live in era where numerous different approaches have been proposed in the NLP community to learn word representations (embeddings). However, word embeddings learnt using different methods have reported different degrees of performance on different tasks. Simply put, there is no one silver bullet when it comes to word embeddings that solves all NLP problems. More importantly the quest for better word embeddings continues and new methods for learning word embeddings that capture various aspects of word semantics are being proposed.

From the NLP practitioners point of view however, this plethora of choices can become a bit of a problem. How do you pick the best word embeddings to train your NLP application? Of course you could try them all and pick the best one. But that is often not a solution because of the time constrants or sheer numnber of different word embedding methods available.

Meta embedding comes to your help!

Instead of picking one word embedding why not use them all! At least use few good ones all at once thereby covering various aspects of word semantics and a larger vocabulary.

The purpose of this web site is to share information about such meta embedding learning methods, publications and pre-trained meta embeddings.

What makes Meta Embedding Learning Different from Word Embedding Learning?

Meta embedding learning is defined as the process of producing a single (meta) word embedding from a given set of pre-trained input (source) word embeddings.

There are several key points in this definition that makes meta embedding learning problem different from that of learning source embeddings.

Meta Embedding Learning Methods and Pre-trained Meta Embeddings

Here is a list of papers that propose methods for learning meta embeddings. Where pre-trained meta embeddings produced using those methods are publicly available we provide those links too. If you have a paper on meta embedding learning and would like to get it listed here please let me know.

This site is maintained by Danushka Bollegala