Bangkok, Thailand — photo by @joshrh19

Vector Space for Information Retrieval

Notes from a lecture in IN2110 at the University of Oslo about document vectors

This article is a set of notes on one of the lecture in IN2110 at the University of Oslo about document vectors. The original title is: “IN2110: Språkteknologiske metoder Vektorrom for IR.” That is: “Vector space models for Information Retrieval (IR).”

In the lecture it was said:

In this case it can be about modelling the similarity of documents, queries, and topics.

  1. The first step is to represent our data in a way that the model can take as input.

→ Very often done by describing the data by a set of features.

In this manner the properties of the data.

What is observable?

What is relevant?

The object modelled can be described as tuple of d feature values.

We can record features of the data.

Every feature has a numerical value.

To bring in the explanation of tuple.

→ object x

→ can be described as a finite ordered list of elements = d feature values

Features are also numerically indexed.

Vector space model is based on a spatial metaphor.

Features correspond to dimensions or coordinate axes in the ‘space’.

This helpful video can explain in more detail.

Features correspond to dimensions or coordinate axes in this space.

The objects correspond to points in this feature space.

It is possible to measure geometrical distance/closeness.

According to the lecture:

“High-dimensional spaces where d is in the thousands or even millions are not uncommon in ML/NLP.”

You can see why this may complicate matters ever so slightly.

There is often a wish to quantify similarity of different documents.

How relevant is a document to a certain query?

Imagine searching on Google and getting the wrong thing.

This may have happened, but often you find what you were searching for.

How relevant is a certain document to a given query?

It is possible to do a frequency count of all the words (so-called bag-of-words).

Each word type corresponds to a dimension.

Conceptually, a vector space is often thought of as a matrix.

It is often called a co-occurrence matrix.

Here is another useful video to explain this as well.

We can assume that documents that share many of the same words are semantically similar in terms of their content.

This may of course not be right at all — however, these inquiries work by making a set of assumptions.

We can also view the rows as vectors representing words.

Words that co-occur in the same documents may be semantically related.

Distributional hypothesis is important in this regard.

The distributional hypothesis is the basis for statistical semantics.

Distributional semantics is a research area that develops and studies theories and methods for quantifying and categorising semantic similarities between linguistic items based on their distributional properties in large samples of language data.

If vectors are defined the one can now compute similarity (of either words or documents) based on distance in the space.

There are several possibilities for this. This is listed in the lecture.

Euclidean distance

Slide from IN2110: Språkteknologiske metoder [1ex] Vektorrom for IR

Euclidean distance and length bias.

Slide from IN2110: Språkteknologiske metoder [1ex] Vektorrom for IR

It is possible to reduce frequency effects.

One can first normalise all vectors to have unit length.

Can be achieved by simply dividing each element by the length.

Model from IN2110: Språkteknologiske metoder [1ex] Vektorrom for IR

It is possible to measure (cosine) proximity rather than (Euclidean) distance.

Computes similarity as a function of the angle between the vectors.

Avoids the arbitrary scaling caused by dimensionality, frequency, etc.

Cosine similarity — model from IN2110: Språkteknologiske metoder [1ex] Vektorrom for IR

Why do this?

We want to know how to find something and whether it is relevant.

AKA queries and relevance making.

Information retrieval (IR).

A central task in information retrieval:

  • “Identifying relevant documents for a given query (i.e. search terms).
  • Treat the query as a short document: I Represent it as a vector and find its nearest neighbors.
  • I.e. rank the documents based on the distance between the document vectors and the query vector.”
Model from IN2110: Språkteknologiske metoder [1ex] Vektorrom for IR

“Problem: Raw frequency counts not always good indicators of relevance.”

The most frequent words will typically not be very discriminative.

One can therefore apply weighting:

A weighting function is usually applied to the raw counts.

The most commonly used weighting function is tf-idf:

The term frequency tf(ti , dj ) denotes the number of times the term ti occurs in document dj.

The document frequency df(ti) denotes the total number of documents in the collection that the term occurs in.

Model from IN2110

N is the total number of documents in the collection.

The weight given to term ti in document dj is then computed as

Model from IN2110

A high tf-idf: is obtained if a term has a high frequency in the given document.

A low tf-idf: frequency in the document collection as whole.

The weights hence tend to filter out common terms.

It is possible to use different processes to further define what a word is.

Model from IN2110

Tokenization: Splitting a text into sentences and words or other units.

To use supervised learning, requiring labeled training data.

One can train a classifier to automatically assign new instances to predefined classes.

This requires labeled training data.

However, one can train a classifier to automatically assign new instances to predefined classes, given some set of training examples.

One can perform clustering. Unsupervised learning from unlabeled data, automatically group similar objects together.

“Classification amounts to computing the boundaries in the space that separate the classes; the decision boundaries.”

Model from IN2110

If you want to read more it is recommended to read about information retrieval here:

Hope you enjoyed this article.

Again, this article is a set of notes on one of the lecture in IN2110 at the University of Oslo about document vectors. For me this is a learning process, and I think it is interesting to get to know more about programming and language.

This is #500daysofAI and you are reading article 446. I am writing one new article about or related to artificial intelligence every day for 500 days.

AI Policy and Ethics at www.nora.ai. Student at University of Copenhagen MSc in Social Data Science. All views are my own. twitter.com/AlexMoltzau