Text Analysis on rdata.lu Blog | Data science with R

Analysis of the Renert - Part 3: Visualizations

Fri, 26 Jan 2018 00:00:00 +0000

This is part 3 of a 3 part blog post. This post uses the data that was scraped in part 1 and prepared in part 2.

Now that we have the data in a nice format, let’s make a frequency plot! First let’s load the data and the packages:

library("tidyverse")
library("ggthemes") # To use different themes and colors
renert_tokenized = readRDS("renert_tokenized.rds")

Using the ggplot2 package, I can produce a plot of the most frequent words.

renert_tokenized %>%
  count(word, sort = TRUE) %>%
  filter(n > 50) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip() +
  theme_minimal()

So, the most frequent word is kinnek, meaning King! kinnek is mentioned more times than renert, the name of the hero. Next are här and wollef meaning mister and wolf. In fifth position we have fuuss, for fox. I’ll let you use Google Translate for the other words 😄.

Now, I’m also doing sentiment analysis by using the AFINN list of words. This list of words have a score that gives its sentiment. You can download the original list from here.

Because such a list is not available in Luxembourguish, I have translated it using Google’s translate api. Here is the code to do that:

library("tidyverse")
library("translate") # google translate api
library("tidytext") # to load the AFINN dictionary

api_key = "api_key_goes_here"

set.key(api_key)

afinn = get_sentiments("afinn")

# I wrap the `translate()` function around `purrr::possibly()` so that in case of an
# error, I get the translations that worked back.

possibly_translate = purrr::possibly(translate::translate, otherwise = "error")

afinn_lux = afinn %>%
  mutate(lux = map(word, possibly_translate, source = "en", target = "lb")) %>%
  mutate(lux = unlist(lux))

write_csv(afinn_lux, "afinn_lux.csv")

For the above code to work, you need to have a Google cloud account, which you can create for free.

I did not check the quality of the translations, and I’m sure it’s far from perfect. It’s also available on the Github repository here. Again, contributions more than welcome!

Now, I need to merge the dictionary with the data from each song. First, let’s load the dictionary:

afinn_lux = read.csv("afinn_lux.csv")

# I only keep the `lux` column (and rename it to word) and the `score column`
afinn_lux = afinn_lux %>%
  select(word = lux, score)

How does this dictionary look like? Let’s see:

head(afinn_lux)

##             word score
## 1       opzeginn    -2
## 2     verloossen    -2
## 3       opzeginn    -2
## 4 entfouert ginn    -2
## 5       entlooss    -2
## 6      entfouert    -2

Let’s load the tokenized songs, and merge them with the dictionary:

renert_songs_tokenized = readRDS("renert_songs_tokenized.rds")
  
renert_songs_sentiment = map(renert_songs_tokenized, ~full_join(., afinn_lux))

I can now merge the data in a single data frame and do some further cleaning:

renert_songs_sentiment = renert_songs_sentiment %>%
  bind_rows() %>%
  filter(!is.na(score)) %>%
  filter(!is.na(gesank))

What does the final data look like? Here it is:

head(renert_songs_sentiment)

## # A tibble: 6 x 3
##   word  gesank  score
##   <chr> <chr>   <int>
## 1 rifft éischte    -2
## 2 léiw  éischte    -3
## 3 fest  éischte     2
## 4 fest  éischte     2
## 5 räich éischte     2
## 6 räich éischte     3

We see that there are words that are the same, but with different scores. That’s because the translation of the dictionary was most probably not very good. Oh well, let’s do a boxplot of the sentiment for each song:

order =  c("éischte", "zwete", "drëtte", "véierte", "fënnefte", "sechste", "siwente",
                   "aachte", "néngte", "zéngte", "elefte", "zwielefte", "dräizengte", "véierzengte")

renert_songs_sentiment %>%
  ggplot(aes(gesank, score)) + 
  scale_x_discrete(limits = order) + 
  geom_boxplot() + 
  theme_minimal() + 
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

As we can see, there is no discernible pattern. This can mean two things; either the general sentiment inside each song is fairly neutral, or the the quality of the translation was too bad for the results to make any sense.

That’s it for this series of posts! I hope you enjoyed reading it as much as I enjoyed writing it and analyzing the data!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Analysis of the Renert - Part 2: Data Processing

Wed, 24 Jan 2018 00:00:00 +0000

This is part 2 of a 3 part blog post. This post uses the data that we scraped in part 1 and prepares it for further analysis, which is quite technical. If you’re only interested in the results of the analysis, skip to part 3!

First, let’s load the data that we prepared in part 1. Let’s start with the full text:

library("tidyverse")
library("tidytext")
renert = readRDS("renert_full.rds")

I want to study the frequencies of words, so for this, I will use a function from the tidytext package called unnest_tokens() which breaks the text down into tokens. Each token is a word, which will then make it possible to compute the frequencies of words.

So, let’s unnest the tokens:

renert = renert %>%
  unnest_tokens(word, text)

We still need to do some cleaning before continuing. In Luxembourgish, the is written d’ for feminine nouns. For example d’Kaz for the the cat. There’s also a bunch of ’ts in the text, which is it. For example, the second line of the first song:

’T stung Alles an der Bléi,

Everything (it) was on flower,

We can remove these with a couple lines of code:

renert_tokenized = renert %>%
  mutate(word = str_replace_all(word, "d'", "")) %>%
  mutate(word = str_replace_all(word, "'t", ""))

But that’s not all! We still need to remove so called stop words. Stop words are words that are very frequent, such as “and”, and these words usually do not add anything to the analysis. There are no set rules for defining a list of stop words, so I took inspiration for the stop words in English and German, and created my own, which you can get on Github.

stopwords = read.csv("stopwords_lu.csv", header = TRUE)

For my Luxembourgish-speaking compatriots, I’d be glad to get help to make this list better! This list is far from perfect, certainly contains typos, or even words that have no reason to be there! Please help 😅.

Using this list of stop words, I can remove words that don’t add anything to the analysis. Creating a list of stop words for the Luxembourgish language is very challenging, because there might be stop words that come from German, such as “awer”, from the German “aber”, meaning but, but you could also use “mä”, from the French mais, meaning also but. Plus, as a kid, we never really learned how to write Luxembourgish. Actually, most Luxembourguians don’t know how to write Luxembourgish 100% correctly. This is because for a very long time, Luxembourgish was used for oral communication, and French for formal written correspondence. This is changing, and more and more people are learning how to write correctly. I definitely have a lot to learn! Thus, I have certainly missed a lot of stop words in the list, but I am hopeful that others will contribute to the list and make it better! In the meantime, that’s what I’m going to use.

Let’s take a look at some lines of the stop words data frame:

head(stopwords, 20)

##          word
## 1           a
## 2           à
## 3         äis
## 4          är
## 5         ärt
## 6        äert
## 7        ären
## 8         all
## 9       allem
## 10      alles
## 11   alleguer
## 12        als
## 13       also
## 14         am
## 15         an
## 16 anerefalls
## 17        ass
## 18        aus
## 19       awer
## 20        bei

We can remove the stop words from our tokens using an anti_join():

renert_tokenized = renert_tokenized %>%
  anti_join(stopwords)

## Joining, by = "word"

## Warning: Column `word` joining character vector and factor, coercing into
## character vector

Let’s save this for later use:

saveRDS(renert_tokenized, "renert_tokenized.rds")

I now have to do the same for the data that is stored by song. Because this is a list where each element is a data frame, I have to use purrr::map() to map each of the functions I used before to each data frame:

renert_songs = readRDS("renert_songs_df.rds")
renert_songs = map(renert_songs, ~unnest_tokens(., word, text))
renert_songs = map(renert_songs, ~anti_join(., stopwords))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, "d'", "")))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, "'t", "")))

Let’s take a look at the object we have:

head(renert_songs[[1]])

## # A tibble: 6 x 1
##   word     
##   <chr>    
## 1 éischte  
## 2 gesank   
## 3 edit     
## 4 päischten
## 5 stung    
## 6 bléi

Looks pretty nice! But I can make it nicer by adding a column containing which song the data refers to. Indeed, the first line of each data frame contains the number of the song. I can extract this information and add it to each data set:

renert_songs = map(renert_songs, ~mutate(., gesank = pull(.[1,1])))

Let’s take a look again:

head(renert_songs[[1]])

## # A tibble: 6 x 2
##   word      gesank 
##   <chr>     <chr>  
## 1 éischte   éischte
## 2 gesank    éischte
## 3 edit      éischte
## 4 päischten éischte
## 5 stung     éischte
## 6 bléi      éischte

Now I can save this object for later use:

saveRDS(renert_songs, "renert_songs_tokenized.rds")

In the final part of this series, I will use the tokenized data as well as the list of songs to create a couple of visualizations!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Analysis of the Renert - Part 1: Scraping

Mon, 22 Jan 2018 00:00:00 +0000

This is part 1 of a 3 part blog post. This post presents the Luxembourgish language as well as the literary work I am going to analyze using the R programming language. Part 2 deals with preparing the data for analysis, and finally part 3 is the analysis. Hope you enjoy!

Luxembourg and the Luxembourgish language

Luxembourg is a small European country, squeezed between France, Belgium and Germany. Over the course of its history, it’s been invaded over and over by either France or Prussia (later Germany). It eventually became a state under the personal possession of William I of the Netherlands in 1815, with a… Prussian garrison to guard its capital, Luxembourg City, from further French invasions. After the Belgian revolution of 1839, the purely French-speaking part of the country was ceded to Belgium and the Luxembourgish-speaking part became what is known today as the Grand-Duchy of Luxembourg. What’s a Grand-Duchy you might wonder?

Luxembourg is the only remaining Grand-Duchy in the world. A Grand-Duchy is like a Kingdom, but instead of a King, we have a Grand Duke. The current monarch is Henri, which means that Luxembourg is a constitutional monarchy with the head of state being the prime minister, Xavier Bettel. As you can imagine, Luxembourg’s history has had a very important impact on the languages we speak today in the country; there are three official languages, French, German, and Luxembourgish. Unlike other countries with several official languages, in Luxembourg, there is not a French, or German, or Luxembourgish speaking part. In Luxembourg, you use one of the three languages based on context.

For example, the laws are all written in French, and French is mostly the language used for official or formal written correspondence.German has traditionally been the language of the press and the police. And finally Luxembourgish is the language Luxembourguians use to speak with one another. This means that on a given day, most people here might switch between these three languages; of course, add English to the pile, which is rapidly growing in the country due to all the English speaking expats that come here to work (coughbrexitcough).

There is also a sizable Portuguese community in Luxembourg, so you’ll hear a lot of Portuguese on the streets too, as well as Italian. Around 50% of the inhabitants of Luxembourg are foreign born, mostly from other EU countries. The Italians, Portuguese and a lot of others have emigrated to Luxembourg starting in the 60s to work in the metallurgic sector, and later, in the construction sector. The children of these emigrants usually speak five languages; their mother tongue, say, Portuguese, the three official languages of the country, and finally English.

You might wonder what Luxembourgish sounds like? Here is a video of our Prime Minister talking in Luxembourgish:

Here is another video of him speaking French:

Here he’s speaking German :

And here English :

On the English video, you might notice the typical accent Luxembourguians have when speaking English 😄

The text we’re analysing

The text I’ll be analyzing is called Renert oder de Fuuss am Frack an a Maansgréisst, published in 1872 by Michel Rodange. My high school was named after Michel Rodange by the way! Renert is a fable featuring a sly fox as the main character, called Renert. He gets in trouble because of his shenanigans and gets sentenced to death by the Lion King. However, through further lies and deceptions, he manages to escape. After some tribulations, he proves his worth to the King by winning a duel against the wolf and becomes an aristocrat. Because it was written in the 19th century, the way some words are written may be different that how we write them in modern Luxembourgish, which might create some problems when analyzing the text.

Now starts the technical part. If you’re only interested in the results, you can skip to part 3!

Scraping the data

First of all, let’s load (or install if you don’t have them) the needed packages:

install.packages(c("tidyverse",
                   "tidytext",
                   "janitor"))

library("tidyverse")
library("tidytext")
library("janitor")

The tidyverse is a collection of packages that are very useful for a lot of different tasks. If you are not familiar with these packages, check out the tidyverse website.

tidytext is a package that uses the same principles than the tidyverse, but for text analysis. You can learn more about it here which is the book I took inspiration from for this series of blog posts.

The full text of the Renert is available here, so I’m going to use rvest, to get the text into R:

renert_link = "https://wikisource.org/wiki/Renert"

renert_raw = renert_link %>%
  xml2::read_html() %>%
  rvest::html_nodes(".mw-parser-output") %>%
  rvest::html_text() %>%
  str_split("\n", simplify = TRUE) %>%
  .[1, -c(1:24)]

I download the text using read_html() from the xml2 package (which gets loaded by the tidyverse) and then find the nodes that interest me, in this case mw-parser-output. Then I extract the text from this node, and split it on the \n character, to get a big vector where each element is a line of text. I also remove the 24 first lines, which are mostly blank. Let’s take a look at the first five lines:

renert_raw[1:5]

## [1] "Éischte Gesank.[edit]"       ""                           
## [3] "Et war esou ëm d'Päischten," "'T stung Alles an der Bléi,"
## [5] "An d'Villercher di songen"

The Renert is divided into 14 songs, so I’d like to create a list with 14 elements, where each element is the text of a song. Every song is titled “First Song”, “Second Song” etc, so I first check on which lines I find the word Gesank, which identifies the start of a song.

(indices = grepl("Gesank", renert_raw) %>% which(isTRUE(.)))

##  [1]    1  605  885 1172 1555 1906 2441 2664 2995 3686 4214 4625 5116 5963

indices contains the indices of where the songs start. So I need to create the indices of when the songs end. If you think about it, the first songs ends where the second song begins, minus 1. So I create a new vector of indices, by first removing the index for the first song, substracting 1, and then adding the index for the last line (using length(renert_raw)).

(indices2 = c(indices[-1] - 1, length(renert_raw)))

##  [1]  604  884 1171 1554 1905 2440 2663 2994 3685 4213 4624 5115 5962 6506

I can now create a list of sequences, called song_lines which contains the indices for all the songs:

song_lines = map2(indices, indices2,  ~seq(.x,.y))

And using this list of indices, I can now extract the songs into a list:

renert_songs = map(song_lines, ~`[`(renert_raw, .))

I’ll save this object for later use, using saveRDS():

saveRDS(renert_songs, "renert_songs.rds")

I will also save a version of the above list, but where each element of the list is a data frame. This will make analysis much easier later.

renert_songs_df = map(renert_songs, ~data_frame(text = .))
saveRDS(renert_songs_df, "renert_songs_df.rds")

I also need to have the full text as a single character object, so I reduce my list into a single object and also save it:

renert_full = reduce(renert_songs, c)

renert_full = data_frame(text = renert_full) %>%
  filter(!grepl("Gesank", text))

saveRDS(renert_full, "renert_full.rds")

This is the end of part 1. In part 2, we are going to prepare the data for analysis, and in part 3 we are going to analyze it.

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!