8  From importing to reporting

First draft

You are reading a work in progress. This page is a first draft but should be readable.

In First Steps in RStudio we typed data into R. This is not very practical when you have a lot of data! Instead, we much more commonly import data from a file. In this chapter we go through the workflow from importing, through summarising and plotting to saving a saving the figure.

8.1 Importing data from files

There are two things you need to know before you can import data from a file.

1. What format the data are in

   The format of the data determines what function you will use to import it. Often the file extension indicates format.

   • .txt a plain text file¹, where the columns are often separated by a space but might also be separated by a tab, a backslash or forward slash, or some other character
   • .csv a plain text file where the columns are separated by commas
   • .xlsx an Excel file More detail on file types was covered in Understanding file systems

   However, you should always check the file to make sure it is in the format you expect because there is little to force a match between a file’s contents and the extension in its name. You can check by opening the file in a text editor (e.g., Notepad on Windows, TextEdit on Mac) or in RStudio (see below).

2. Where the file is relative to your working directory

   R can only read in a file if you say where it is, i.e., you give its relative path. More detail on relative file paths and working directories was covered in Understanding file systems

🎬 Your turn! If you want to code along you will need to start a new RStudio project then a new script.

This chapter covers reading .txt files and .csv files using tidyverse (Wickham et al. 2019) functions and excel files using the readxl (Wickham and Bryan 2023) package. We will demonstrate what needs to be done differently if the file is not in your working directory.

library(tidyverse)
library(readxl)

8.1.1 Importing data from .txt file

The data in adipocytes.txt give the concentration of a hormone called adiponectin in some cells. There are two columns: the first gives the adiponectin concentration and the second, treatment, indicates whether the cells were treated with nicotinic acid or not. Save this file to the project folder.

A .txt extension suggests this is plain text file with columns separated by spaces. However, before we attempt to read it in, when should take a look at it. We can do this from RStudio by clicking on the file in the Files pane. Any plain text file will open in the top left pane.

Figure 8.1: The adipocytes.txt data file open. We can see the columns are separated by spaces

The files are separated by spaces as we suspected. We use the read_table() command to read in plain text files of single columns, or where the columns are separated by spaces:

adipo <- read_table("adipocytes.txt")

The data from the file has been read into a dataframe called adipo. You will and you will be able to see it in the Environment window. Clicking on it in the Environment window will open a spreadsheet-like view of the dataframe.

8.1.2 Importing a from a.csv file

The data seal.csv give the myoglobin concentration of skeletal muscle for three species of seal. There are two columns: the first gives the myoglobin concentration and the second indicates species.

The .csv extension suggests this is plain text file with columns separated by commas. We will again check this before we attempt to read it in. Click on the file in the Files pane - a pop-up will appear for files ending .csv or .xlsx. Choose View File².

Rstudio Files pane showing the data files and the View File option that appears when you click on the a particular file

CSV files will open in the top left pane (Excel files will launch Excel). We can see that the file does contain comma separated values. There is aread_csv() function which works very like read_table()[^working_with_data_rstudio-3]:

seal <- read_csv("seals.csv")

8.1.3 Importing a from a.xlsx file

The data in blood.xlsx are measurements of several blood parameters from fifty people with Crohn’s disease, a lifelong condition where parts of the digestive system become inflamed. Twenty-five of people are in the early stages of diagnosis and 25 have started treatment.

blood <- read_excel("blood.xlsx")

8.1.4 Importing data from a file not in your working directory

When using an RStudio project, your working directory is the project folder (the folder containing the .Rproj file. Suppose our file adipocytes.txt is in a folder, data-raw, in our working directory.

-- myproject
   |__myproject.Rproj
   |__import.R
   |__data-raw
      |__adipocytes.txt
      |__blood.xlsx
      |__seal.csv

We need to adjust the code to give the relative path to the datafile:

adipo <- read_table("data-raw/adipocytes.txt")

The relative path is the path from the working directory to the file. In this case, the working directory is myproject and the relative path is data-raw/adipocytes.txt.

🎬 Your turn! Create a folder called data-raw inside the project folder and move the data files to it. Now modify the data import code to import seal.csv from data-raw.

seal <- read_csv("data-raw/seal.csv")

8.2 Summarising data

We summarise data using the the dplyr (Wickham et al. 2023) package, which provides a set of functions designed for efficient data manipulation. This is a tidyverse (Wickham et al. 2019) package which you already loaded. The approach replies on the data being in a tidy format, meaning each column represents a variable, each row represents an observation, and each cell contains a single value. The pipeline is:

• Group the data: If you want to summarize your data based on certain groups, you can use the group_by()

• Summarise: Once your data grouped (if necessary), you use summarise() with functions likemean(), median(), sd(), min()and max() within it.

We will demonstrate summarising using the adipo dataframe. adiponectin is the response and is continuous and treatment is an explanatory with categorical with two levels (groups).

The most useful summary statistics for a continuous variable like adiponectin are the means, standard deviations, sample sizes and standard errors. We use the group_by() and summarise() functions along with the functions that do the calculations.

To create a data frame called adipo_summary that contains the means, standard deviations, sample sizes and standard errors for the control and nicotinic acid treated samples:

adipo_summary <- adipo  |> 
  group_by(treatment) |>
  summarise(mean = mean(adiponectin),
            std = sd(adiponectin),
            n = length(adiponectin),
            se = std/sqrt(n))

You can type:

adipo_summary
## # A tibble: 2 × 5
##   treatment  mean   std     n    se
##   <chr>     <dbl> <dbl> <int> <dbl>
## 1 control    5.55  1.48    15 0.381
## 2 nicotinic  7.51  1.79    15 0.463

or click on environment to open a spreadsheet-like view of the dataframe.

8.2.1 Visualise data

Most commonly, we put the explanatory variable on the x axis and the response variable on the y axis. A continuous response, particularly one that follows the normal distribution, is best summarised with the mean and the standard error. In my opinion, you should also show all the raw data points if possible.

We are going to create a figure like this:

8.2.1.1 ggplot2

ggplot2 (Wickham 2016) is a powerful data visualisation package in R that is part of the tidyverse. It provides a flexible and layered approach to creating high-quality and customizable graphics.

The core concept of ggplot2 is to build a plot layer by layer. The basic structure consists of three main components:

1. Data: the data frame to be used for plotting.
2. Aesthetic mappings (aes): how variables in the data map to visual elements such as x and y positions, colours, shapes, etc.
3. Geometric objects (geoms): the actual graphical elements used to visualize the data, such as points, lines, bars, etc.

To create a basic plot, you start with the ggplot() function and provide the data and aesthetic mappings.

ggplot(data = adipo, aes(x = treatment, y = adiponectin))

You can add geometric layers to the plot using specific functions such as geom_point(), geom_line(), geom_bar(), etc. These functions define the type of plot you want to create:

ggplot(data = adipo, aes(x = treatment, y = adiponectin)) +
  geom_point()

In the figure we are aiming for, we are plotting two dataframes: the adipo dataframe which contains the data points themselves; and the adipo_summary dataframe containing and the means and standard errors.

The dataframes and aesthetics for ggplot can be specified within a geom_xxxx (rather than in the ggplot()). This is very useful if the geom only applies to some of the data you want to plot.

Tip: ggplot()

You put the data argument and aes() inside ggplot() if you want all the geoms to use that dataframe and variables. If you want a different dataframe for a geom, put the data argument and aes() inside the geom_xxxx()

I will build the plot up in small steps you should edit your existing ggplot() command as we go.

We will plot the data points first. Notice that we have given the data argument and the aesthetics inside the geom_point(). The variables treatment and adiponectin are in the adipo dataframe

ggplot() +
  geom_point(data = adipo, 
             aes(x = treatment, y = adiponectin))

So the data points don’t overlap, we can add some random jitter in the x direction (edit your existing code):

ggplot() +
  geom_point(data = adipo, 
             aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0))

Note that position = position_jitter(width = 0.1, height = 0) is inside the geom_point() parentheses, after the aes() and a comma.

We’ve set the vertical jitter to 0 because, in contrast to the categorical x-axis, movement on the y-axis has meaning (the adiponectin levels).

Let’s make the points a light grey (edit your existing code):

ggplot() +
  geom_point(data = adipo, 
             aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "grey50")

Now to add the errorbars. These go from one standard error below the mean to one standard error above the mean.

Add a geom_errorbar() for errorbars (edit your existing code):

ggplot() +
  geom_point(data = adipo, aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "grey50") +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean - se, ymax = mean + se),
                width = 0.3) 

We have specified the adipo_summary dataframe and the variables treatment, mean and se are in that.

There are several ways you could add the mean. You could use geom_point() but I like to use geom_errorbar() again with the ymin and ymax both set to the mean.

Add a geom_errorbar() for the mean (edit your existing code):

ggplot() +
  geom_point(data = adipo, aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "grey50") +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean - se, ymax = mean + se),
                width = 0.3) +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean, ymax = mean),
                width = 0.2)

Alter the axis labels and limits using scale_y_continuous() and scale_x_discrete() (edit your existing code):

ggplot() +
  geom_point(data = adipo, aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "grey50") +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean - se, ymax = mean + se),
                width = 0.3) +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean, ymax = mean),
                width = 0.2) +
  scale_y_continuous(name = "Adiponectin (pg/mL)", 
                     limits = c(0, 12), 
                     expand = c(0, 0)) +
  scale_x_discrete(name = "Treatment", 
                   labels = c("Control", "Nicotinic acid"))

You only need to use scale_y_continuous() and scale_x_discrete() to use labels that are different from those in the dataset. Often this is to use proper terminology and captialisation.

Format the figure in a way that is more suitable for including in a report using theme_classic() (edit your existing code):

ggplot() +
  geom_point(data = adipo, aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "gray50") +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean - se, ymax = mean + se),
                width = 0.3) +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean, ymax = mean),
                width = 0.2) +
  scale_y_continuous(name = "Adiponectin (pg/mL)", 
                     limits = c(0, 12), 
                     expand = c(0, 0)) +
  scale_x_discrete(name = "Treatment", 
                   labels = c("Control", "Nicotinic acid")) +
  theme_classic()

The ggsave() function is used to save a ggplot object as an image file. It provides a convenient way to export your plots to various file formats, such as PNG, PDF, SVG, or JPEG.

The basic syntax of ggsave() is as follows:

ggsave(filename,
       plot,
       device,
       width,
       height,
       units,
       dpi)

You must give a file name for the output file but all the other options have defaults.

• plot: The ggplot object you want to save. Defaults to the last created plot.
• device: one of “png”, “eps”, “ps”, “tex”, “pdf”, “jpeg”, “tiff”, “png”, “bmp”, “svg” or “wmf” (windows only). Defaults to the format given by the file extension in filename
• width, height units: Plot size in units (“in”, “cm”, “mm”, or “px”). Defaults to the size of the plot in the Plots window.
• dpi: Plot resolution.

We can save the figure we just created as a 3 inch x 3 inch png file as follows:

ggsave("adipocytes.png",
       device = "png",
       width = 3,
       height = 3,
       units = "in",
       dpi = 300)

It is often a good idea to explicitly assign the ggplot object to a variable and use that in the ggsave()

fig1 <- ggplot() +
  geom_point(data = adipo, aes(x = treatment, y = adiponectin),
             position = position_jitter(width = 0.1, height = 0),
             colour = "gray50") +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean - se, ymax = mean + se),
                width = 0.3) +
  geom_errorbar(data = adipo_summary, 
                aes(x = treatment, ymin = mean, ymax = mean),
                width = 0.2) +
  scale_y_continuous(name = "Adiponectin (pg/mL)", 
                     limits = c(0, 12), 
                     expand = c(0, 0)) +
  scale_x_discrete(name = "Treatment", 
                   labels = c("Control", "Nicotinic acid")) +
  theme_classic()

ggsave("adipocytes.png",
       plot = fig1,
       device = "png",
       width = 3,
       height = 3,
       units = "in",
       dpi = 300)

8.3 Tidy data

Data that is in a format that is easy to work with is often referred to as “tidy data”. Tidy data is a concept that was introduced by Hadley Wickham in his paper Tidy Data (Wickham 2014). The paper is available online at http://vita.had.co.nz/papers/tidy-data.pdf.

Tidy data:

• Each variable should be in one column.
• Each different observation of that variable should be in a different row.
• There should be one table for each “kind” of data.
• If you have multiple tables, they should include a column in the table that allows them to be linked.

These concepts have been around for a long time and underlie formats enforced in several statistical packages such as SPSS, minitab and SAS.

Recognising the structure of your data and organising it in tidy format is one of the most important step in data analysis.

Wickham, Hadley. 2014. “Tidy Data.” Journal of Statistical Software, Articles 59 (10): 1–23. https://vita.had.co.nz/papers/tidy-data.pdf.

———. 2016. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org.

Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the Tidyverse” 4: 1686. https://doi.org/10.21105/joss.01686.

Wickham, Hadley, and Jennifer Bryan. 2023. “Readxl: Read Excel Files.” https://CRAN.R-project.org/package=readxl.

Wickham, Hadley, Romain François, Lionel Henry, Kirill Müller, and Davis Vaughan. 2023. Dplyr: A Grammar of Data Manipulation. https://CRAN.R-project.org/package=dplyr.

---

1. Plain text files can be opened in notepad or other similar editor and still be readable.

2. There is also and option to import the dataset. Do not be tempted to import data this way! Unless you are careful and know what you are doing, your data import will not be scripted or will not be scripted correctly.