Why quantifying ?

Figure 1: The evolution of data storage and computational power. Source: Bit by Bit: Social Science Research in the Digital Age, Salganik (2017)

The rise of publications

Figure 2: The evolution of documents in the Econlit database

New methods to analyze texts

• A lot of progress in Natural Language Processing:
  • Translation model (e.g., Deepl)
  • Conversational model (e.g., Chat GPT)

Quantitative and qualitative methods

“everything can be counted”

Defining a corpus

• A corpus \(C\) is set of documents \(d\).
• A document \(d\) is a general unit, it can be a book, an article, a speech, a paragraph, an abstract, a tweet, etc.

Illustration of a typical corpus

Web scraping

Regular Expression (RegEx)

To make our corpus operational for quantitative analysis, we often need to “clean” it:

• For manipulating a large amount of textual data, we use Regular Expressions (RegEx), i.e. a set of rules you can use in a program such as R to find patterns in a text.

What is representation ?

An example

• We have a corpus with two documents:

The measure tf-idf

• In the tf-idf, the tf stands for Term Frequency.
• It estimates the relative frequency a word in a document:

\[tf(w, d) = \frac{f_{w,d}}{\sum\limits_{w \in d } f_{w, d}}\]

With, \(f_{w,d}\) the frequency of a word in the document \(d\), and \(\sum\limits_{w \in d } f_{w, d}\) the number of words in the document \(d\).

The measure tf-idf

• In the tf-idf, the idf stands for Inverse Document Frequency.

• It is the ratio between the number of documents in the corpus noted \(N\) and the number of documents that contain the word \(w\):

\[idf = \log{(\frac{N}{d\in C:w \in d})}\]

The measure tf-idf

The TF-IDF combines the two distinct measures.

\[\frac{f_{w,d}}{\sum\limits_{w \in d } f_{w, d}} * \log{(\frac{N}{d\in C:w \in d})}\]

• The TF-IDF gives more weight to words that appear several times in a document (tf) and to words that are specific to a document compared to other documents (idf).

• It is therefore a measure of both the importance and the specificity of a word within a document relative to the corpus.

The measure tf-idf

And the representation ?

We did not apply the tf-idf to the corpus \(C\), but to a another table, \(w\), which associates a textual data to a numeric value:

Selection

The limits of the representation bag-of-words

• Beyond the curse of dimensionality, BoW has important shortcomings.
• For instance, the two following sentences have the exact same BoW representation but opposite meanings:

“Hayek is wrong and Keynes is right !”

“Hayek is right and Keynes is wrong !”

The limits of the representation bag-of-words

The BoW representation does not capture information about:

The importance of the context

• Natural languages are inherently ambiguous. The meaning of a word crucially depends of the context on which it is used:

You know a word by the company it keeps (Firth 1957)

• Yet, the bag-of-words approach removes all information about the context.

• A simple solution is to look at word co-occurences.

The co-occurence

• Imagine we want to predict what is the missing word of this sentence, what words are the most likely ?

My research contributes to the field of [ ? ] economics

The co-occurence: an example

Imagine a new corpus:

The co-occurence frequency

Word embedding representation

• In 2013, a new method for representing texts emerged in NLP called word embedding, or word vectorization.

  • Word2vec (Mikolov et al. 2013)
  • Glove (Pennington et al. 2014)
  • FastText (Bojanowski et al. 2016)
  • and now transformers: BERT, GPT …

A look at word2vec

Semantic similarities

• This representation associates each word of the vocabulary to a vector of real numbers (generally \(N = 300\)).

• These real numbers are arbitrary, but the words that are semantically similar have close vectors

Intuitions in a 3 dimensions space

Intuition with vectors

Word semantic proximity

• The standard mesure to estimate the proximity between word vectors is the cosine similarity:

• \[\text{Cosine Similarity}(\vec{A}, \vec{B}) = \frac{\vec{A} \cdot \vec{B}}{\|\vec{A}\| \|\vec{B}\|} = \frac{\sum_{i=1}^{n} A_i B_i}{\sqrt{\sum_{i=1}^{n} A_i^2} \sqrt{\sum_{i=1}^{n} B_i^2}}\]

• Score between -1 (opposite meanings) and +1 (similar meanings).

• \(\text{Cosine Similarity}(\vec{price}, \vec{inflation}) = 0.7210443\) (source: Whatever it takes project)

Manipulating concepts and ideas

• Word embedding models capture conceptual and analological relationship between words:

\[\overrightarrow{king} - \overrightarrow{man} + \overrightarrow{woman} = \overrightarrow{queen}\]

\[\overrightarrow{paris} - \overrightarrow{france} + \overrightarrow{england} = \overrightarrow{london}\]

Word embeddings advantages and issues

What is analysis ?

• Once we build \(W\), a representation of the corpus \(C\), we can finally analyze the data to answer a research question \(Y\).
• Formally, we can think the analysis step as the application of a measure \(f\) to \(W\) to produce an estimation of a research question \(\hat{Y})\).

Lexicon analysis

• The dictionary approach is a supervised method based on a dictionary \(D\) made up of a set of pairs \((w_i,s_i)\) where \(w_i\) is a token associated with a measure \(s_i\).

• A popular dictionary approach is the sentiment analysis, where \(s_i \in [-1,1]\) represents a sentiment score.

  • Words that reveal a negative sentiment have a score close to -1 (“awful”, -0.9)
  • Words that reveal a positive sentiment are close to 1 (“good”, 0.8).

An example

Boyer et al. (2020) applies this method to study the evolution of the “gilets jaunes” movement:

A word embedding augmented lexicon analysis

• Standard lexicon is simple but depends of the quality of the dictionary.
• Lexicon analysis can be augmented with word embeddings:
  • You don’t need a complete dictionary, just a good set of word embeddings
• Two examples: Ash, Chen, and Naidu (2022) and Goutsmedt et al. (2024)

Ash, Chen, and Naidu (2022): “Ideas Have Consequences”

Comparing the usage of a word by different groups

Comparing the evolution of meaning over time

Document embedded vectors

Topic modelling

Source: Six Decades of Economic Research at the Bank of England (Acosta et al. 2023)