Scraping on rdata.lu Blog | Data science with R

Analysis of the Renert - Part 1: Scraping

Mon, 22 Jan 2018 00:00:00 +0000

This is part 1 of a 3 part blog post. This post presents the Luxembourgish language as well as the literary work I am going to analyze using the R programming language. Part 2 deals with preparing the data for analysis, and finally part 3 is the analysis. Hope you enjoy!

Luxembourg and the Luxembourgish language

Luxembourg is a small European country, squeezed between France, Belgium and Germany. Over the course of its history, it’s been invaded over and over by either France or Prussia (later Germany). It eventually became a state under the personal possession of William I of the Netherlands in 1815, with a… Prussian garrison to guard its capital, Luxembourg City, from further French invasions. After the Belgian revolution of 1839, the purely French-speaking part of the country was ceded to Belgium and the Luxembourgish-speaking part became what is known today as the Grand-Duchy of Luxembourg. What’s a Grand-Duchy you might wonder?

Luxembourg is the only remaining Grand-Duchy in the world. A Grand-Duchy is like a Kingdom, but instead of a King, we have a Grand Duke. The current monarch is Henri, which means that Luxembourg is a constitutional monarchy with the head of state being the prime minister, Xavier Bettel. As you can imagine, Luxembourg’s history has had a very important impact on the languages we speak today in the country; there are three official languages, French, German, and Luxembourgish. Unlike other countries with several official languages, in Luxembourg, there is not a French, or German, or Luxembourgish speaking part. In Luxembourg, you use one of the three languages based on context.

For example, the laws are all written in French, and French is mostly the language used for official or formal written correspondence.German has traditionally been the language of the press and the police. And finally Luxembourgish is the language Luxembourguians use to speak with one another. This means that on a given day, most people here might switch between these three languages; of course, add English to the pile, which is rapidly growing in the country due to all the English speaking expats that come here to work (coughbrexitcough).

There is also a sizable Portuguese community in Luxembourg, so you’ll hear a lot of Portuguese on the streets too, as well as Italian. Around 50% of the inhabitants of Luxembourg are foreign born, mostly from other EU countries. The Italians, Portuguese and a lot of others have emigrated to Luxembourg starting in the 60s to work in the metallurgic sector, and later, in the construction sector. The children of these emigrants usually speak five languages; their mother tongue, say, Portuguese, the three official languages of the country, and finally English.

You might wonder what Luxembourgish sounds like? Here is a video of our Prime Minister talking in Luxembourgish:

Here is another video of him speaking French:

Here he’s speaking German :

And here English :

On the English video, you might notice the typical accent Luxembourguians have when speaking English 😄

The text we’re analysing

The text I’ll be analyzing is called Renert oder de Fuuss am Frack an a Maansgréisst, published in 1872 by Michel Rodange. My high school was named after Michel Rodange by the way! Renert is a fable featuring a sly fox as the main character, called Renert. He gets in trouble because of his shenanigans and gets sentenced to death by the Lion King. However, through further lies and deceptions, he manages to escape. After some tribulations, he proves his worth to the King by winning a duel against the wolf and becomes an aristocrat. Because it was written in the 19th century, the way some words are written may be different that how we write them in modern Luxembourgish, which might create some problems when analyzing the text.

Now starts the technical part. If you’re only interested in the results, you can skip to part 3!

Scraping the data

First of all, let’s load (or install if you don’t have them) the needed packages:

install.packages(c("tidyverse",
                   "tidytext",
                   "janitor"))

library("tidyverse")
library("tidytext")
library("janitor")

The tidyverse is a collection of packages that are very useful for a lot of different tasks. If you are not familiar with these packages, check out the tidyverse website.

tidytext is a package that uses the same principles than the tidyverse, but for text analysis. You can learn more about it here which is the book I took inspiration from for this series of blog posts.

The full text of the Renert is available here, so I’m going to use rvest, to get the text into R:

renert_link = "https://wikisource.org/wiki/Renert"

renert_raw = renert_link %>%
  xml2::read_html() %>%
  rvest::html_nodes(".mw-parser-output") %>%
  rvest::html_text() %>%
  str_split("\n", simplify = TRUE) %>%
  .[1, -c(1:24)]

I download the text using read_html() from the xml2 package (which gets loaded by the tidyverse) and then find the nodes that interest me, in this case mw-parser-output. Then I extract the text from this node, and split it on the \n character, to get a big vector where each element is a line of text. I also remove the 24 first lines, which are mostly blank. Let’s take a look at the first five lines:

renert_raw[1:5]

## [1] "Éischte Gesank.[edit]"       ""                           
## [3] "Et war esou ëm d'Päischten," "'T stung Alles an der Bléi,"
## [5] "An d'Villercher di songen"

The Renert is divided into 14 songs, so I’d like to create a list with 14 elements, where each element is the text of a song. Every song is titled “First Song”, “Second Song” etc, so I first check on which lines I find the word Gesank, which identifies the start of a song.

(indices = grepl("Gesank", renert_raw) %>% which(isTRUE(.)))

##  [1]    1  605  885 1172 1555 1906 2441 2664 2995 3686 4214 4625 5116 5963

indices contains the indices of where the songs start. So I need to create the indices of when the songs end. If you think about it, the first songs ends where the second song begins, minus 1. So I create a new vector of indices, by first removing the index for the first song, substracting 1, and then adding the index for the last line (using length(renert_raw)).

(indices2 = c(indices[-1] - 1, length(renert_raw)))

##  [1]  604  884 1171 1554 1905 2440 2663 2994 3685 4213 4624 5115 5962 6506

I can now create a list of sequences, called song_lines which contains the indices for all the songs:

song_lines = map2(indices, indices2,  ~seq(.x,.y))

And using this list of indices, I can now extract the songs into a list:

renert_songs = map(song_lines, ~`[`(renert_raw, .))

I’ll save this object for later use, using saveRDS():

saveRDS(renert_songs, "renert_songs.rds")

I will also save a version of the above list, but where each element of the list is a data frame. This will make analysis much easier later.

renert_songs_df = map(renert_songs, ~data_frame(text = .))
saveRDS(renert_songs_df, "renert_songs_df.rds")

I also need to have the full text as a single character object, so I reduce my list into a single object and also save it:

renert_full = reduce(renert_songs, c)

renert_full = data_frame(text = renert_full) %>%
  filter(!grepl("Gesank", text))

saveRDS(renert_full, "renert_full.rds")

This is the end of part 1. In part 2, we are going to prepare the data for analysis, and in part 3 we are going to analyze it.

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Scraping data from the local elections

Fri, 27 Oct 2017 06:34:55 +0200

One of my journalist friend was looking at the result of the local election in Luxembourg and he was dissatisfied because he was unable to compare the results of all the communes. In fact, he wanted to compare the number of women that were candidates in each commune. So I asked him to hold on and I came back one hour later with this script that enables him to collect results of all communes in one table.

At the beginning, it was private code but I thought that it could be another great scraping example after the excellent post written by my colleague Bruno Rodrigues about scraping data from STATEC public tables.

So let’s get started. First, let’s load some packages:

#we have to load the packages if they are not installed on your computer,
#begin with the commented following lines:
#install.packages( "rvest" )
#install.packages( "tidyverse" )
#install.packages( "stringr" )

library(rvest) #to scrap
library(dplyr) #to manipulate data
library(stringr) #to manipulate string

We want to collect results of all communes in one data frame. We go to http://www.elections.public.lu and we collect data that is seen in the GIF bellow:

After clicking on different communes, we notice that the URLs have the same format. They have 3 parts :
1: “http://www.elections.public.lu/fr/elections-communales/2017/resultats/communes/”, the first part of the URL.
2: “communes_names” the name of the city is the second part of the URL.
3: “.html” is the last part.

For example, the complete URL for the commune of Luxembourg will be: http://www.elections.public.lu/fr/elections-communales/2017/resultats/communes/luxembourg.html

There are 103 communes, so we have to put all of them in a list. We scrape the 103 communes in one list via the script bellow :

url = "http://www.elections.public.lu/fr/elections-communales/2017/resultats/communes/bech.html"
communes = read_html(url) %>% 
        html_nodes("#communes #communes-az li") %>%
        html_text()

We verify that we have a list of 103 vectors and then we check the 5 first rows.

length(communes)

## [1] 103

head(communes,5)

## [1] "\r\n        Beaufort\r\n                a rendu l'ensemble de ses résultats\r\n    " 
## [2] "\r\n        Bech\r\n                a rendu l'ensemble de ses résultats\r\n    "     
## [3] "\r\n        Beckerich\r\n                a rendu l'ensemble de ses résultats\r\n    "
## [4] "\r\n        Berdorf\r\n                a rendu l'ensemble de ses résultats\r\n    "  
## [5] "\r\n        Bertrange\r\n                a rendu l'ensemble de ses résultats\r\n    "

It seems that the 103 communes are present but we still have to clean the data.

#Data is not cleaned and there are some useless characters that need to be removed.
#We need to clean data.
communes = gsub("a rendu l'ensemble de ses résultats"," ", communes )
communes = trimws(gsub("\r\n","",communes ))
communes = gsub("/Attert","-sur-attert", communes )
communes = gsub(" - ", "-",communes )
communes = gsub(" ", "-",communes )
communes = gsub("'", "-",communes )
communes = gsub("é", "e",communes )
communes = gsub("û", "u",communes )
communes = gsub("ä", "a",communes )

#Lower case
communes = tolower(communes)

head(communes, 5)

## [1] "beaufort"  "bech"      "beckerich" "berdorf"   "bertrange"

Now that we have the list of the 103 communes, we will write a function that will enable us to collect data that we want to display in our data frame.

#Function to have the result for one commune.
  result = function(x){
  #scrapping data
  vote = read_html(paste("http://www.elections.public.lu/fr/elections-communales/2017/resultats/communes/",x,".html", sep="")) %>% 
          html_nodes("#lux-number .lux-number ul li") %>%
          html_text()%>%.[-1]
  
  #Conditions need to be added to have clean data.
  #Here we add a trick, the vector 14 and 15 are the only ones that   haven't the string "\r\n".
  #So we add "\r\n" to these vectors.
  if(nchar(vote[14]) > 21){
          vote[14] = gsub("ble","ble\r\n", vote[14],perl = FALSE)
  }
  if(nchar(vote[15]) > 21){
          vote[15] = gsub("mé","mé\r\n", vote[15],perl = FALSE)
  }
  #We split vectors to dissociate the results (numbers) and the titles   (letters).
  vote = unlist(str_split(vote, "\r\n"))
  vote = trimws(vote)
  vote = vote[vote != ""]
  
  #Here we have similar title so we change them to not be confused.
  #Candidat Lux means Luxemburgish Candidates  & Electeur Lux means   Luxemburgish voters. 
  vote[7] = gsub("Lux", "Candidat Lux", vote[7] )
  vote[9] = gsub("Non Lu", "Candidat Non Lu", vote[9] )
  vote[13] = gsub("Lux", "Electeur Lux", vote[13] )
  vote[15] = gsub("Non Lu", "Electeur Non Lu", vote[15]  )
  
  #We create the data frame.
  #Vector with pair indice value are the results (the column val).
  pair = (1:15)*2
  
  #Vectors with odd index value are the titles (the column title).
  impair = (1:15)*2 - 1
  res = data.frame(communes = rep(x,15), title = vote[impair], val =   vote[pair])
  return(res)
}

Then we use the lapply() function to apply this function to the 103 communes and bind them in one data frame:

#We use the result function on all the communes.
res = lapply(communes, result)
#We bind the rows to have a complete data frame with all the results from all communes
#then we bind all the result.
df = do.call(rbind, res)

To see the 5 first rows of our new data frame:

#Now the data that we have look like this:
head(df,5)

##   communes                       title val
## 1 beaufort                       Total  10
## 2 beaufort                      Femmes   5
## 3 beaufort                      Hommes   5
## 4 beaufort     Candidat Luxembourgeois  10
## 5 beaufort Candidat Non Luxembourgeois   0

As you can see, it’s looking good but we are not completely satisfied because we would like to transpose data to have one result in one column. To this end, we use the tidyr package.

library(tidyr)

#Transposing data will enables us to make analysis faster.
#We transform val in a numeric variable then we transpose data.

tdf = df %>%
        mutate(val = as.numeric(gsub(" ", "", val))) %>%
        spread(title, val)

To see the 5 first rows of our new data frame:

#Now the data that we have look like this:
head(tdf,5)

##    communes Blancs Candidat Luxembourgeois Candidat Non Luxembourgeois
## 1  beaufort     78                      10                           0
## 2      bech      0                       8                           0
## 3 beckerich     67                      10                           1
## 4   berdorf     23                      20                           0
## 5 bertrange     75                      45                           7
##   Dans l'urne Electeur Luxembourgeois Electeur Non Luxembourgeois Femmes
## 1        1144                    1170                         114      5
## 2           0                     699                          95      1
## 3        1396                    1393                         137      2
## 4         857                     850                          79      7
## 5        2935                    2972                         478     22
##   Grand total exprimé Grand total possible Hommes Inscrits Nuls Total
## 1                4021                 9234      5     1284   40    10
## 2                   0                    0      7      794    0     8
## 3                5792                11637      9     1530   36    11
## 4                4929                 7371     13      929   15    20
## 5               32880                35620     30     3450  120    52
##   Valables Votes par correspondance
## 1     1026                       71
## 2        0                        0
## 3     1293                      109
## 4      819                       57
## 5     2740                      269

Now you can export the table in excel or play with your data in R!

readr::write_excel_csv(tdf,"election_lux.csv")
#To know where your file is saved, we use the following function:
#setwd()

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Scraping data from STATEC's public tables

Fri, 21 Apr 2017 06:34:55 +0200

A lot of open data is available in Luxembourg’s open data portal, but sometimes, it is not very easy to download. In the video below, I give you an example of such data and show how you can use rvest to get the data easily.

After watching the video, take a look at the code below. This code does two things; first it scrapes the data, and then it puts the data in a tidy format fur further processing.

So to summarize the idea of the video; instead of clicking the buttons to download each year’s data (which you would have to do 15 times), it is easier to simple turn off javascript and then scrape the html version of the table. It would be possible, albeit with much more effort, to scrape the tables with javascript enabled, by using a tool such as phantomjs. But since we have the possibility to view the table in html, why not take advantage of it?

To scrape the data, you will need first to install the rvest and then load it (and let’s also load the other needed packages)

library(rvest)
library(dplyr)
library(tidyr)
library(purrr)
library(janitor)

Now, using rvest::read_html(), we can download the whole html page:

page_unemp <- read_html("http://www.statistiques.public.lu/stat/TableViewer/tableViewHTML.aspx?ReportId=12950&IF_Language=eng&MainTheme=2&FldrName=3&RFPath=91")

Now, we need to extract the table from the html page, and we do this by using rvest::html_nodes() and by providing this function with the name of the class of the object we’re interested in, namely, the table.

page_unemp %>%
  html_nodes(".b2020-datatable") %>% .[[1]] %>% html_table(fill = TRUE) -> data_raw


head(data_raw)

##                          X1                         X2      X3      X4
## 1                      Year                       Year    2001    2002
## 2             Specification                       Year    2001    2002
## 3 Grand Duchy of Luxembourg  Total employed population 180,084 182,004
## 4 Grand Duchy of Luxembourg     of which: Wage-earners 162,407 164,277
## 5 Grand Duchy of Luxembourg of which: Non-wage-earners  17,677  17,727
## 6 Grand Duchy of Luxembourg                 Unemployed   5,393   6,773
##        X5      X6      X7      X8      X9     X10     X11     X12     X13
## 1    2003    2004    2005    2006    2007    2008    2009    2010    2011
## 2    2003    2004    2005    2006    2007    2008    2009    2010    2011
## 3 183,419 186,325 187,380 192,095 197,486 202,203 204,127 207,923 214,094
## 4 165,509 168,214 169,194 174,045 179,176 183,705 185,369 188,983 194,893
## 5  17,910  18,111  18,186  18,050  18,310  18,498  18,758  18,940  19,201
## 6   8,359   9,426  10,653  10,297   9,670  11,496  14,816  15,567  16,159
##       X14     X15     X16     X17      X18
## 1    2012    2013    2014    2015     2016
## 2    2012    2013    2014    2015 Measures
## 3 219,168 223,407 228,423 233,130  236,100
## 4 199,741 203,535 208,238 212,530  215,430
## 5  19,427  19,872  20,185  20,600   20,670
## 6  16,963  19,287  19,362  18,806   18,185

As you can see, we got the data in quite a nice format, but it still needs to be cleaned a bit. Let’s do this.

First, let’s use the first row as the header of the data set and then remove it:

colnames(data_raw) <- data_raw[2, ]
colnames(data_raw)[1:2] <- c("division", "variable")
data_raw <- data_raw[-c(1,2), ]
head(data_raw)

##                    division                   variable    2001    2002
## 3 Grand Duchy of Luxembourg  Total employed population 180,084 182,004
## 4 Grand Duchy of Luxembourg     of which: Wage-earners 162,407 164,277
## 5 Grand Duchy of Luxembourg of which: Non-wage-earners  17,677  17,727
## 6 Grand Duchy of Luxembourg                 Unemployed   5,393   6,773
## 7 Grand Duchy of Luxembourg          Active population 185,477 188,777
## 8 Grand Duchy of Luxembourg   Unemployment rate (in %)    2.91    3.59
##      2003    2004    2005    2006    2007    2008    2009    2010    2011
## 3 183,419 186,325 187,380 192,095 197,486 202,203 204,127 207,923 214,094
## 4 165,509 168,214 169,194 174,045 179,176 183,705 185,369 188,983 194,893
## 5  17,910  18,111  18,186  18,050  18,310  18,498  18,758  18,940  19,201
## 6   8,359   9,426  10,653  10,297   9,670  11,496  14,816  15,567  16,159
## 7 191,778 195,751 198,033 202,392 207,156 213,699 218,943 223,490 230,253
## 8    4.36    4.82    5.38    5.09    4.67    5.38    6.77    6.97    7.02
##      2012    2013    2014    2015 Measures
## 3 219,168 223,407 228,423 233,130  236,100
## 4 199,741 203,535 208,238 212,530  215,430
## 5  19,427  19,872  20,185  20,600   20,670
## 6  16,963  19,287  19,362  18,806   18,185
## 7 236,131 242,694 247,785 251,936  254,285
## 8    7.18    7.95    7.81    7.46     7.15

This is starting to look nice, but we need to replace the “,” with “.” and then convert the columns to numeric.

data_raw %>%
  map_df(function(x)(gsub(",", ".", x = x))) %>%
  mutate_at(vars(matches("\\d{4}")), as.numeric
            ) -> clean_unemp

head(clean_unemp)

## # A tibble: 6 x 18
##   division    variable    `2001` `2002` `2003` `2004` `2005` `2006` `2007`
##   <chr>       <chr>        <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
## 1 Grand Duch… Total empl… 180    182    183    186    187    192    197   
## 2 Grand Duch… of which: … 162    164    166    168    169    174    179   
## 3 Grand Duch… of which: …  17.7   17.7   17.9   18.1   18.2   18.0   18.3 
## 4 Grand Duch… Unemployed    5.39   6.77   8.36   9.43  10.7   10.3    9.67
## 5 Grand Duch… Active pop… 185    189    192    196    198    202    207   
## 6 Grand Duch… Unemployme…   2.91   3.59   4.36   4.82   5.38   5.09   4.67
## # ... with 9 more variables: `2008` <dbl>, `2009` <dbl>, `2010` <dbl>,
## #   `2011` <dbl>, `2012` <dbl>, `2013` <dbl>, `2014` <dbl>, `2015` <dbl>,
## #   Measures <chr>

This line: map_df(function(x)(gsub(",", ".", x = x))) calls purrr::map_df(), which maps a function to each column of a data frame. The function in question is function(x)(gsub(",", ".", x = x)), which is an anonymous function (meaning it does not have a name) wrapped around gsub. This function looks for the string “,” and replaces it with “.” in a single column of the data frame. But because we’re mapping this function to all the columns of the data frame with purrr::map_df(), this substitution happens in each column. We’ not done yet, because these columns are still holding characters. We need to convert each column to a numeric vector and this is what happens in the next line, mutate_at(vars(matches("\\d{4}")), as.numeric). Each column that contains exactly for digits (hence the "\\d{4}") is converted to numeric with dplyr::mutate_at().

Now, one last step to really have the data in a nice format:

clean_unemp %>% 
    gather(key=year, value, -division, -variable) %>%
    spread(variable, value) %>%
    clean_names(
           ) -> clean_unemp

head(clean_unemp)

## # A tibble: 6 x 8
##   division year  active_population of_which_non_wage_e… of_which_wage_ear…
##   <chr>    <chr> <chr>             <chr>                <chr>             
## 1 Beaufort 2001  688               85                   568               
## 2 Beaufort 2002  742               85                   631               
## 3 Beaufort 2003  773               85                   648               
## 4 Beaufort 2004  828               80                   706               
## 5 Beaufort 2005  866               96                   719               
## 6 Beaufort 2006  893               87                   746               
## # ... with 3 more variables: total_employed_population <chr>,
## #   unemployed <chr>, unemployment_rate_in_percent <chr>

By using tidyr::gather() and then tidyr::spread() we get a nice data set where each column is a variable and each row is an observation. I advise you run the above code line by line and try to understand what each function does. We finish by cleaning the names of the variables with janitor::clean_names() and that’s it.

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!