Tips on rdata.lu Blog | Data science with R

Analysis of the Renert - Part 3: Visualizations

Fri, 26 Jan 2018 00:00:00 +0000

This is part 3 of a 3 part blog post. This post uses the data that was scraped in part 1 and prepared in part 2.

Now that we have the data in a nice format, let’s make a frequency plot! First let’s load the data and the packages:

library("tidyverse")
library("ggthemes") # To use different themes and colors
renert_tokenized = readRDS("renert_tokenized.rds")

Using the ggplot2 package, I can produce a plot of the most frequent words.

renert_tokenized %>%
  count(word, sort = TRUE) %>%
  filter(n > 50) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip() +
  theme_minimal()

So, the most frequent word is kinnek, meaning King! kinnek is mentioned more times than renert, the name of the hero. Next are här and wollef meaning mister and wolf. In fifth position we have fuuss, for fox. I’ll let you use Google Translate for the other words 😄.

Now, I’m also doing sentiment analysis by using the AFINN list of words. This list of words have a score that gives its sentiment. You can download the original list from here.

Because such a list is not available in Luxembourguish, I have translated it using Google’s translate api. Here is the code to do that:

library("tidyverse")
library("translate") # google translate api
library("tidytext") # to load the AFINN dictionary

api_key = "api_key_goes_here"

set.key(api_key)

afinn = get_sentiments("afinn")

# I wrap the `translate()` function around `purrr::possibly()` so that in case of an
# error, I get the translations that worked back.

possibly_translate = purrr::possibly(translate::translate, otherwise = "error")

afinn_lux = afinn %>%
  mutate(lux = map(word, possibly_translate, source = "en", target = "lb")) %>%
  mutate(lux = unlist(lux))

write_csv(afinn_lux, "afinn_lux.csv")

For the above code to work, you need to have a Google cloud account, which you can create for free.

I did not check the quality of the translations, and I’m sure it’s far from perfect. It’s also available on the Github repository here. Again, contributions more than welcome!

Now, I need to merge the dictionary with the data from each song. First, let’s load the dictionary:

afinn_lux = read.csv("afinn_lux.csv")

# I only keep the `lux` column (and rename it to word) and the `score column`
afinn_lux = afinn_lux %>%
  select(word = lux, score)

How does this dictionary look like? Let’s see:

head(afinn_lux)

##             word score
## 1       opzeginn    -2
## 2     verloossen    -2
## 3       opzeginn    -2
## 4 entfouert ginn    -2
## 5       entlooss    -2
## 6      entfouert    -2

Let’s load the tokenized songs, and merge them with the dictionary:

renert_songs_tokenized = readRDS("renert_songs_tokenized.rds")
  
renert_songs_sentiment = map(renert_songs_tokenized, ~full_join(., afinn_lux))

I can now merge the data in a single data frame and do some further cleaning:

renert_songs_sentiment = renert_songs_sentiment %>%
  bind_rows() %>%
  filter(!is.na(score)) %>%
  filter(!is.na(gesank))

What does the final data look like? Here it is:

head(renert_songs_sentiment)

## # A tibble: 6 x 3
##   word  gesank  score
##   <chr> <chr>   <int>
## 1 rifft éischte    -2
## 2 léiw  éischte    -3
## 3 fest  éischte     2
## 4 fest  éischte     2
## 5 räich éischte     2
## 6 räich éischte     3

We see that there are words that are the same, but with different scores. That’s because the translation of the dictionary was most probably not very good. Oh well, let’s do a boxplot of the sentiment for each song:

order =  c("éischte", "zwete", "drëtte", "véierte", "fënnefte", "sechste", "siwente",
                   "aachte", "néngte", "zéngte", "elefte", "zwielefte", "dräizengte", "véierzengte")

renert_songs_sentiment %>%
  ggplot(aes(gesank, score)) + 
  scale_x_discrete(limits = order) + 
  geom_boxplot() + 
  theme_minimal() + 
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

As we can see, there is no discernible pattern. This can mean two things; either the general sentiment inside each song is fairly neutral, or the the quality of the translation was too bad for the results to make any sense.

That’s it for this series of posts! I hope you enjoyed reading it as much as I enjoyed writing it and analyzing the data!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Analysis of the Renert - Part 2: Data Processing

Wed, 24 Jan 2018 00:00:00 +0000

This is part 2 of a 3 part blog post. This post uses the data that we scraped in part 1 and prepares it for further analysis, which is quite technical. If you’re only interested in the results of the analysis, skip to part 3!

First, let’s load the data that we prepared in part 1. Let’s start with the full text:

library("tidyverse")
library("tidytext")
renert = readRDS("renert_full.rds")

I want to study the frequencies of words, so for this, I will use a function from the tidytext package called unnest_tokens() which breaks the text down into tokens. Each token is a word, which will then make it possible to compute the frequencies of words.

So, let’s unnest the tokens:

renert = renert %>%
  unnest_tokens(word, text)

We still need to do some cleaning before continuing. In Luxembourgish, the is written d’ for feminine nouns. For example d’Kaz for the the cat. There’s also a bunch of ’ts in the text, which is it. For example, the second line of the first song:

’T stung Alles an der Bléi,

Everything (it) was on flower,

We can remove these with a couple lines of code:

renert_tokenized = renert %>%
  mutate(word = str_replace_all(word, "d'", "")) %>%
  mutate(word = str_replace_all(word, "'t", ""))

But that’s not all! We still need to remove so called stop words. Stop words are words that are very frequent, such as “and”, and these words usually do not add anything to the analysis. There are no set rules for defining a list of stop words, so I took inspiration for the stop words in English and German, and created my own, which you can get on Github.

stopwords = read.csv("stopwords_lu.csv", header = TRUE)

For my Luxembourgish-speaking compatriots, I’d be glad to get help to make this list better! This list is far from perfect, certainly contains typos, or even words that have no reason to be there! Please help 😅.

Using this list of stop words, I can remove words that don’t add anything to the analysis. Creating a list of stop words for the Luxembourgish language is very challenging, because there might be stop words that come from German, such as “awer”, from the German “aber”, meaning but, but you could also use “mä”, from the French mais, meaning also but. Plus, as a kid, we never really learned how to write Luxembourgish. Actually, most Luxembourguians don’t know how to write Luxembourgish 100% correctly. This is because for a very long time, Luxembourgish was used for oral communication, and French for formal written correspondence. This is changing, and more and more people are learning how to write correctly. I definitely have a lot to learn! Thus, I have certainly missed a lot of stop words in the list, but I am hopeful that others will contribute to the list and make it better! In the meantime, that’s what I’m going to use.

Let’s take a look at some lines of the stop words data frame:

head(stopwords, 20)

##          word
## 1           a
## 2           à
## 3         äis
## 4          är
## 5         ärt
## 6        äert
## 7        ären
## 8         all
## 9       allem
## 10      alles
## 11   alleguer
## 12        als
## 13       also
## 14         am
## 15         an
## 16 anerefalls
## 17        ass
## 18        aus
## 19       awer
## 20        bei

We can remove the stop words from our tokens using an anti_join():

renert_tokenized = renert_tokenized %>%
  anti_join(stopwords)

## Joining, by = "word"

## Warning: Column `word` joining character vector and factor, coercing into
## character vector

Let’s save this for later use:

saveRDS(renert_tokenized, "renert_tokenized.rds")

I now have to do the same for the data that is stored by song. Because this is a list where each element is a data frame, I have to use purrr::map() to map each of the functions I used before to each data frame:

renert_songs = readRDS("renert_songs_df.rds")
renert_songs = map(renert_songs, ~unnest_tokens(., word, text))
renert_songs = map(renert_songs, ~anti_join(., stopwords))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, "d'", "")))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, "'t", "")))

Let’s take a look at the object we have:

head(renert_songs[[1]])

## # A tibble: 6 x 1
##   word     
##   <chr>    
## 1 éischte  
## 2 gesank   
## 3 edit     
## 4 päischten
## 5 stung    
## 6 bléi

Looks pretty nice! But I can make it nicer by adding a column containing which song the data refers to. Indeed, the first line of each data frame contains the number of the song. I can extract this information and add it to each data set:

renert_songs = map(renert_songs, ~mutate(., gesank = pull(.[1,1])))

Let’s take a look again:

head(renert_songs[[1]])

## # A tibble: 6 x 2
##   word      gesank 
##   <chr>     <chr>  
## 1 éischte   éischte
## 2 gesank    éischte
## 3 edit      éischte
## 4 päischten éischte
## 5 stung     éischte
## 6 bléi      éischte

Now I can save this object for later use:

saveRDS(renert_songs, "renert_songs_tokenized.rds")

In the final part of this series, I will use the tokenized data as well as the list of songs to create a couple of visualizations!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Launching your shiny app in 2 clicks

Tue, 26 Dec 2017 00:00:00 +0000

Hello everyone,

Why we always want to take the red pill when we can take the blue one? That’s the question!
In this post I will explain how to launch a shiny application from a shortcut. Just like that:

It’s fast and useful if you work with colleagues that don’t have a clue about R and just want to use your shiny app.

If you are on macOS:

Open a text editor and write the following lines :

 Rscript '&FullPath/&file.R' &
 open http://127.0.0.1:&PortNumber/;

& allows you to write more command lines even if the shell is still busy with the previous line. The second line opens your web browser with the chosen address. A concrete example gives this:

Rscript '/Users/user/Desktop/Clef_USB/data_science/fret_dashboard.R' &
open http://127.0.0.1:7990/;

On the second line, 7990 refers to the chosen port in your shiny app. If you didn’t choose one, add the following to your shinyApp: options = list(port= 7990), just like that:
shinyApp(ui=ui,server=server, options = list(port=7990))

If you use a .Rmd document, you should write this:

 Rscript -e  "rmarkdown::run('&FullPath/&file.Rmd', shiny_args = list(port=&PortNumber))" &
 open http://127.0.0.1:&PortNumber/&file.Rmd;

Replace &FullPath, &file and &PortNumber by the file path, the file name and the chosen port.

Then save it with the “.command” extension and try to launch your shiny application from the shortcut. If you have a message saying that the file could not be executed because you do not have appropriate access privileges, open your terminal and write the following line:

chmod u+x &FullPath/&file.command

Then it should work :)
We are almost done, now we have to add another shortcut icon to make it look more friendly. For that, go on google and download free .icns icons. I have download some icons on easyicon. To change the icon you just need to go in “get info” and drag your new icon to the previous one, just like below:

If you are on Windows:

Open a text editor and write the following lines:

start "" "&FullPath1\Rscript.exe" &FullPath2\&file.R  /k
start "Name" "http://127.0.0.1:&PortNumber/"

/k allows you to write something in the command prompt even if the Prompt is busy with the previous line. A concrete example gives that :

start "" "C:\Program Files\R\R-3.3.3\bin\Rscript.exe" C:\Users\CGP462\Documents\Shiny_app\iris.R  /k
start "iris" "http://127.0.0.1:7924/"

Then you can save your file with a .bat extension.
Be carefull, if you want to launch a .Rmd you should use this code instead :

start "&FullPath1\R.exe" -e "Sys.setenv(RSTUDIO_PANDOC='C:/Program Files/RStudio/bin/pandoc'); rmarkdown::run('&FullPath2/&file.Rmd', shiny_args = list(port =&PortNumber, launch.browser = FALSE))" /k
start "&file" "http://127.0.0.1:7924/&file.Rmd"

if it doesn’t work, check your pandoc location.

It’s almost done. Now we have to change the icon. First you have to download an icon with the .ico extension. Then you have to act in 4 steps :
1. Right click on your file and go in Properties.
2. Choose the Shortcut tab.
3. Then go in Change Icon….
4. In Browse, select the icon you want to see instead.

And now it’s done!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Skip errors in R loops by not writing loops

Thu, 21 Dec 2017 00:00:00 +0000

You probably have encountered situations similar to this one:

result = vector("list", length(some_numbers))

for(i in seq_along(some_numbers)){
  result[[i]] = some_function(some_numbers[[i]])
}

print(result)

First I initialize result, an empty list of size equal to the length of some_numbers which will contains the results of applying some_function() to each element of some_numbers. Then, using a for loop, I apply the function. This is what I get back:

NaNs producedError in sqrt(x) : non-numeric argument to mathematical function

Let’s take a look at some_numbers and some_function():

print(some_numbers)

## [[1]]
## [1] -1.9
## 
## [[2]]
## [1] 20
## 
## [[3]]
## [1] "-88"
## 
## [[4]]
## [1] -42

some_function

## function(x){
##   if(x == 0) res = 0
##   if(x < 0) res = -sqrt(-x)
##   if(x > 0) res = sqrt(x)
##   return(res)
## }

So the function simply returns the square root of x (or minus the square root of -x if x is negative), but the number in third position of the list some_numbers is actually a character. This type of mistakes can commonly happen. The result list looks like this:

print(result)

[[1]]
[1] -1.378405

[[2]]
[1] 4.472136

[[3]]
NULL

[[4]]
NULL

As you see, even though the fourth element could have been computed, the error made the whole loop stop. In such a simple example, you could correct this and then run your function. But what if the list you want to apply your function to is very long and the computation take a very, very long time? Perhaps you simply want to skip these errors and get back to them later. One way of doing that is using tryCatch():

result = vector("list", length(some_numbers))

for(i in seq_along(some_numbers)){
  result[[i]] = tryCatch(some_function(some_numbers[[i]]), 
                         error = function(e) paste("something wrong here"))
}

print(result)

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## [1] "something wrong here"
## 
## [[4]]
## [1] -6.480741

This works, but it’s verbose and easy to mess up. My advice here is that if you want to skip errors in loops you don’t write loops! This is quite easy with the purrr package:

library(purrr)

result = map(some_numbers, some_function)

There’s several advantages here already; no need to initialize an empty structure to hold your result, and no need to think about indices, which can sometimes get confusing. This however does not work either; there’s still the problem that we have a character inside some_numbers:

Error in sqrt(x) : non-numeric argument to mathematical function

However, purrr contains some very amazing functions for error handling, safely() and possibly(). Let’s try possibly() first:

possibly_some_function = possibly(some_function, otherwise = "something wrong here")

possibly() takes a function as argument as well as otherwise; this is where you define a return value in case something is wrong. possibly() then returns a new function that skips errors:

result = map(some_numbers, possibly_some_function)

print(result)

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## [1] "something wrong here"
## 
## [[4]]
## [1] -6.480741

When you use possibly() on a function, you’re politely telling R “would you kindly apply the function wherever possible, and if not, tell me where there was an issue”. What about safely()?

safely_some_function = safely(some_function)


result = map(some_numbers, safely_some_function)

str(result)

## List of 4
##  $ :List of 2
##   ..$ result: num -1.38
##   ..$ error : NULL
##  $ :List of 2
##   ..$ result: num 4.47
##   ..$ error : NULL
##  $ :List of 2
##   ..$ result: NULL
##   ..$ error :List of 2
##   .. ..$ message: chr "invalid argument to unary operator"
##   .. ..$ call   : language -x
##   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
##  $ :List of 2
##   ..$ result: num -6.48
##   ..$ error : NULL

The major difference with possibly() is that safely() returns a more complex object: it returns a list of lists. There are as many lists as there are elements in some_numbers. Let’s take a look at the first one:

print(result[[1]])

## $result
## [1] -1.378405
## 
## $error
## NULL

result[[1]] is a list with a result and an error. If there was no error, we get a value in result and NULL in error. If there was an error, this is what we see:

print(result[[3]])

## $result
## NULL
## 
## $error
## <simpleError in -x: invalid argument to unary operator>

Because lists of lists are not easy to handle, I like to use possibly(), but if you use safely() you might want to know about transpose(), which is another function from purrr:

result2 = transpose(result)

str(result2)

## List of 2
##  $ result:List of 4
##   ..$ : num -1.38
##   ..$ : num 4.47
##   ..$ : NULL
##   ..$ : num -6.48
##  $ error :List of 4
##   ..$ : NULL
##   ..$ : NULL
##   ..$ :List of 2
##   .. ..$ message: chr "invalid argument to unary operator"
##   .. ..$ call   : language -x
##   .. ..- attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
##   ..$ : NULL

result2 is now a list of two lists: a result list holding all the results, and an error list holding all the error message. You can get results with:

result2$result

## [[1]]
## [1] -1.378405
## 
## [[2]]
## [1] 4.472136
## 
## [[3]]
## NULL
## 
## [[4]]
## [1] -6.480741

I hope you enjoyed this blog post, and that these functions will make your life easier!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

Functional peace of mind

Tue, 14 Nov 2017 00:00:00 +0000

I think what I enjoy the most about functional programming is the peace of mind that comes with it. With functional programming, there’s a lot of stuff you don’t need to think about. You can write functions that are general enough so that they solve a variety of problems. For example, imagine for a second that R does not have the sum() function anymore. If you want to compute the sum of, say, the first 100 integers, you could write a loop that would do that for you:

numbers = 0

for (i in 1:100){
  numbers = numbers + i
}

print(numbers)

## [1] 5050

The problem with this approach, is that you cannot reuse any of the code there, even if you put it inside a function. For instance, what if you want to merge 4 datasets together? You would need something like this:

library(dplyr)
data(mtcars)

mtcars1 = mtcars %>%
  mutate(id = "1")

mtcars2 = mtcars %>%
  mutate(id = "2")

mtcars3 = mtcars %>%
  mutate(id = "3")

mtcars4 = mtcars %>%
  mutate(id = "4")

datasets = list(mtcars1, mtcars2, mtcars3, mtcars4)

temp = datasets[[1]]

for(i in 1:3){
  temp = full_join(temp, datasets[[i+1]])
}

## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")

glimpse(temp)

## Observations: 128
## Variables: 12
## $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19....
## $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, ...
## $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 1...
## $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, ...
## $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.9...
## $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3...
## $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 2...
## $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, ...
## $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, ...
## $ id   <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1...

Of course, the logic is very similar as before, but you need to think carefully about the structure holding your elements (which can be numbers, datasets, characters, etc…) as well as be careful about indexing correctly… and depending on the type of objects you are working on, you might need to tweak the code further.

How would a functional programming approach make this easier? Of course, you could use purrr::reduce() to solve these problems. However, since I assumed that sum() does not exist, I will also assume that purrr::reduce() does not exist either and write my own, clumsy implementation. Here’s the code:

my_reduce = function(a_list, a_func, init = NULL, ...){

  if(is.null(init)){
    init = `[[`(a_list, 1)
    a_list = tail(a_list, -1)
  }

  car = `[[`(a_list, 1)
  cdr = tail(a_list, -1)
  init = a_func(init, car, ...)

  if(length(cdr) != 0){
    my_reduce(cdr, a_func, init, ...)
  }
  else {
    init
  }
}

This can look much more complicated than before, but the idea is quite simple; if you know about recursive functions (recursive functions are functions that call themselves). I won’t explain how the function works, because it is not the main point of the article (but if you’re curious, I encourage you to play around with it). The point is that now, I can do the following:

my_reduce(list(1,2,3,4,5), `+`)

## [1] 15

my_reduce(datasets, full_join) %>% glimpse

## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")
## Joining, by = c("mpg", "cyl", "disp", "hp", "drat", "wt", "qsec", "vs", "am", "gear", "carb", "id")

## Observations: 128
## Variables: 12
## $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19....
## $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, ...
## $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 1...
## $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, ...
## $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.9...
## $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3...
## $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 2...
## $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, ...
## $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, ...
## $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, ...
## $ id   <chr> "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1", "1...

And if I need to merge another dataset, I don’t need to change anything at all. Plus, because my_reduce() is very general, I can even use it for situation I didn’t write it for in the first place:

my_reduce(list("a", "b", "c", "d", "e"), paste)

## [1] "a b c d e"

Of course, paste() is vectorized, so you could just as well do paste(1, 2, 3, 4, 5), but again, I want to insist on the fact that writing or using such functions allows you to abstract over a lot of thing. There is nothing specific to any type of object in my_reduce(), whereas the loops have to be tailored for the kind of object you’re working with. As long as the a_func argument is a binary operator that combines the elements inside a_list, it’s going to work. And I don’t need to think about indexing, about having temporary variables or thinking about the structure that will hold my results.

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!

How tidyeval could make your life easier

Sun, 27 Aug 2017 00:00:00 +0000

First thing’s first: maybe you shouldn’t care about tidyeval. Maybe you don’t need it. If you exclusively work interactively, I don’t think that learning about tidyeval is important. I can only speak for me, and explain to you why I personally find tidyeval useful.

I wanted to write this blog post after reading this twitter thread and specifically this question.

Mara Averick then wrote this blogpost linking to 6 other blog posts that give some tidyeval examples. Reading them, plus the Programming with dplyr vignette should help you get started with tidyeval.

But maybe now you know how to use it, but not why and when you should use it… Basically, whenever you want to write a function that looks something like this:

my_function(my_data, one_column_inside_data)

is when you want to use the power of tidyeval.

I work at STATEC, Luxembourg’s national institute of statistics. I work on all kinds of different projects, and when data gets updated (for example because a new round of data collection for some survey finished), I run my own scripts on the fresh data to make the data nice and tidy for analysis. Because surveys get updated, sometimes column names change a little bit, and this can cause some issues.

Very recently, a dataset I work with got updated. Data collection was finished, so I just loaded my hombrewed package written for this project, changed the path from last year’s script to this year’s fresh data path, ran the code, and watched as the folders got populated with new ggplot2 graphs and LaTeX tables with descriptive statistics and regression results. This is then used to generate this year’s report. However, by looking at the graphs, I noticed something weird; some graphs were showing some very strange patterns. It turns out that one column got its name changed, and also one of its values got changed too.

Last year, this column, let’s call it spam, had values 1 for good and 0 for bad. This year the column is called Spam and the values are 1 and 2. When I found out that this was the source of the problem, I just had to change the arguments of my functions from

generate_spam_plot(dataset = data2016, column = spam, value = 1)
generate_spam_plot(dataset = data2016, column = spam, value = 0)

generate_spam_plot(dataset = data2017, column = Spam, value = 1)
generate_spam_plot(dataset = data2017, column = Spam, value = 2)

without needing to change anything else. This is why I use tidyeval; without it, writing a function such as genereta_spam_plot would not be easy. It would be possible, but not easy.

If you want to know more about tidyeval and working programmatically with R, I shamelessly invite you to read a book I’ve been working on: http://www.brodrigues.co/fput/ It’s still a WIP, but maybe you’ll find it useful. I plan on finishing it by the end of the year, but there’s already some content to keep you busy!

Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel.
You can also contact us if you have any comments or suggestions. See you for the next post!