2.2.1 Non-syntactic names

R has strict rules about what constitutes a valid name. A syntactic name must consist of letters¹⁰, digits, . and _ but can’t begin with _ or a digit. Additionally, you can’t use any of the reserved words like TRUE, NULL, if, and function (see the complete list in ?Reserved). A name that doesn’t follow these rules is a non-syntactic name; if you try to use them, you’ll get an error:

_abc <- 1
#> Error: unexpected input in "_"

if <- 10
#> Error: unexpected assignment in "if <-"

It’s possible to override these rules and use any name, i.e., any sequence of characters, by surrounding it with backticks:

`_abc` <- 1
`_abc`
#> [1] 1

`if` <- 10
`if`
#> [1] 10

While it’s unlikely you’d deliberately create such crazy names, you need to understand how these crazy names work because you’ll come across them, most commonly when you load data that has been created outside of R.

2.2.2 Exercises

1. Explain the relationship between a, b, c and d in the following code:

   a <- 1:10
   b <- a
   c <- b
   d <- 1:10

2. The following code accesses the mean function in multiple ways. Do they all point to the same underlying function object? Verify this with lobstr::obj_addr().

   mean
   base::mean
   get("mean")
   evalq(mean)
   match.fun("mean")

3. By default, base R data import functions, like read.csv(), will automatically convert non-syntactic names to syntactic ones. Why might this be problematic? What option allows you to suppress this behaviour?

4. What rules does make.names() use to convert non-syntactic names into syntactic ones?

5. I slightly simplified the rules that govern syntactic names. Why is .123e1 not a syntactic name? Read ?make.names for the full details.

2.3.1 tracemem()

You can see when an object gets copied with the help of base::tracemem(). Once you call that function with an object, you’ll get the object’s current address:

x <- c(1, 2, 3)
cat(tracemem(x), "\n")
#> <0x7f80c0e0ffc8> 

From then on, whenever that object is copied, tracemem() will print a message telling you which object was copied, its new address, and the sequence of calls that led to the copy:

y <- x
y[[3]] <- 4L
#> tracemem[0x7f80c0e0ffc8 -> 0x7f80c4427f40]: 

If you modify y again, it won’t get copied. That’s because the new object now only has a single name bound to it, so R applies modify-in-place optimisation. We’ll come back to this in Section 2.5.

y[[3]] <- 5L

untracemem(x)

untracemem() is the opposite of tracemem(); it turns tracing off.

2.3.2 Function calls

The same rules for copying also apply to function calls. Take this code:

f <- function(a) {
  a
}

x <- c(1, 2, 3)
cat(tracemem(x), "\n")
#> <0x7fe1121693a8>

z <- f(x)
# there's no copy here!

untracemem(x)

While f() is running, the a inside the function points to the same value as the x does outside the function:

You’ll learn more about the conventions used in this diagram in Section 7.4.4. In brief: the function f() is depicted by the yellow object on the right. It has a formal argument, a, which becomes a binding (indicated by dotted black line) in the execution environment (the gray box) when the function is run.

Once f() completes, x and z will point to the same object. 0x74b never gets copied because it never gets modified. If f() did modify x, R would create a new copy, and then z would bind that object.

2.3.3 Lists

It’s not just names (i.e. variables) that point to values; elements of lists do too. Consider this list, which is superficially very similar to the numeric vector above:

l1 <- list(1, 2, 3)

This list is more complex because instead of storing the values itself, it stores references to them:

This is particularly important when we modify a list:

l2 <- l1

l2[[3]] <- 4

Like vectors, lists use copy-on-modify behaviour; the original list is left unchanged, and R creates a modified copy. This, however, is a shallow copy: the list object and its bindings are copied, but the values pointed to by the bindings are not. The opposite of a shallow copy is a deep copy where the contents of every reference are copied. Prior to R 3.1.0, copies were always deep copies.

To see values that are shared across lists, use lobstr::ref(). ref() prints the memory address of each object, along with a local ID so that you can easily cross-reference shared components.

ref(l1, l2)
#> █ [1:0x7fe11166c6d8] <list> 
#> ├─[2:0x7fe11b6d2078] <dbl> 
#> ├─[3:0x7fe11b6d2040] <dbl> 
#> └─[4:0x7fe11b6d2008] <dbl> 
#>  
#> █ [5:0x7fe11411cc18] <list> 
#> ├─[2:0x7fe11b6d2078] 
#> ├─[3:0x7fe11b6d2040] 
#> └─[6:0x7fe114130a70] <dbl>

2.3.4 Data frames

Data frames are lists of vectors, so copy-on-modify has important consequences when you modify a data frame. Take this data frame as an example:

d1 <- data.frame(x = c(1, 5, 6), y = c(2, 4, 3))

If you modify a column, only that column needs to be modified; the others will still point to their original references:

d2 <- d1
d2[, 2] <- d2[, 2] * 2

However, if you modify a row, every column is modified, which means every column must be copied:

d3 <- d1
d3[1, ] <- d3[1, ] * 3

2.3.5 Character vectors

The final place that R uses references is with character vectors¹². I usually draw character vectors like this:

x <- c("a", "a", "abc", "d")

But this is a polite fiction. R actually uses a global string pool where each element of a character vector is a pointer to a unique string in the pool:

You can request that ref() show these references by setting the character argument to TRUE:

ref(x, character = TRUE)
#> █ [1:0x7fe114251578] <chr> 
#> ├─[2:0x7fe10ead1648] <string: "a"> 
#> ├─[2:0x7fe10ead1648] 
#> ├─[3:0x7fe11b27d670] <string: "abc"> 
#> └─[4:0x7fe10eda4170] <string: "d">

This has a profound impact on the amount of memory a character vector uses but is otherwise generally unimportant, so elsewhere in the book I’ll draw character vectors as if the strings lived inside a vector.

2.3.6 Exercises

1. Why is tracemem(1:10) not useful?

2. Explain why tracemem() shows two copies when you run this code. Hint: carefully look at the difference between this code and the code shown earlier in the section.

   x <- c(1L, 2L, 3L)
   tracemem(x)
   
   x[[3]] <- 4

3. Sketch out the relationship between the following objects:

   a <- 1:10
   b <- list(a, a)
   c <- list(b, a, 1:10)

4. What happens when you run this code?

   x <- list(1:10)
   x[[2]] <- x

   Draw a picture.

2.4.1 Exercises

1. In the following example, why are object.size(y) and obj_size(y) so radically different? Consult the documentation of object.size().

   y <- rep(list(runif(1e4)), 100)
   
   object.size(y)
   #> 8005648 bytes
   obj_size(y)
   #> 80,896 B

2. Take the following list. Why is its size somewhat misleading?

   funs <- list(mean, sd, var)
   obj_size(funs)
   #> 17,608 B

3. Predict the output of the following code:

   a <- runif(1e6)
   obj_size(a)
   
   b <- list(a, a)
   obj_size(b)
   obj_size(a, b)
   
   b[[1]][[1]] <- 10
   obj_size(b)
   obj_size(a, b)
   
   b[[2]][[1]] <- 10
   obj_size(b)
   obj_size(a, b)

2.5.1 Objects with a single binding

If an object has a single name bound to it, R will modify it in place:

v <- c(1, 2, 3)

v[[3]] <- 4

(Note the object IDs here: v continues to bind to the same object, 0x207.)

Two complications make predicting exactly when R applies this optimisation challenging:

• When it comes to bindings, R can currently¹⁵ only count 0, 1, or many. That means that if an object has two bindings, and one goes away, the reference count does not go back to 1: one less than many is still many. In turn, this means that R will make copies when it sometimes doesn’t need to.

• Whenever you call the vast majority of functions, it makes a reference to the object. The only exception are specially written “primitive” C functions. These can only be written by R-core and occur mostly in the base package.

Together, these two complications make it hard to predict whether or not a copy will occur. Instead, it’s better to determine it empirically with tracemem().

Let’s explore the subtleties with a case study using for loops. For loops have a reputation for being slow in R, but often that slowness is caused by every iteration of the loop creating a copy. Consider the following code. It subtracts the median from each column of a large data frame:

x <- data.frame(matrix(runif(5 * 1e4), ncol = 5))
medians <- vapply(x, median, numeric(1))

for (i in seq_along(medians)) {
  x[[i]] <- x[[i]] - medians[[i]]
}

This loop is surprisingly slow because each iteration of the loop copies the data frame. You can see this by using tracemem():

cat(tracemem(x), "\n")
#> <0x7f80c429e020> 

for (i in 1:5) {
  x[[i]] <- x[[i]] - medians[[i]]
}
#> tracemem[0x7f80c429e020 -> 0x7f80c0c144d8]: 
#> tracemem[0x7f80c0c144d8 -> 0x7f80c0c14540]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c14540 -> 0x7f80c0c145a8]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c145a8 -> 0x7f80c0c14610]: 
#> tracemem[0x7f80c0c14610 -> 0x7f80c0c14678]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c14678 -> 0x7f80c0c146e0]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c146e0 -> 0x7f80c0c14748]: 
#> tracemem[0x7f80c0c14748 -> 0x7f80c0c147b0]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c147b0 -> 0x7f80c0c14818]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c14818 -> 0x7f80c0c14880]: 
#> tracemem[0x7f80c0c14880 -> 0x7f80c0c148e8]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c148e8 -> 0x7f80c0c14950]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c14950 -> 0x7f80c0c149b8]: 
#> tracemem[0x7f80c0c149b8 -> 0x7f80c0c14a20]: [[<-.data.frame [[<- 
#> tracemem[0x7f80c0c14a20 -> 0x7f80c0c14a88]: [[<-.data.frame [[<- 

untracemem(x)

In fact, each iteration copies the data frame not once, not twice, but three times! Two copies are made by [[.data.frame, and a further copy¹⁶ is made because [[.data.frame is a regular function that increments the reference count of x.

We can reduce the number of copies by using a list instead of a data frame. Modifying a list uses internal C code, so the references are not incremented and only a single copy is made:

y <- as.list(x)
cat(tracemem(y), "\n")
#> <0x7f80c5c3de20>
  
for (i in 1:5) {
  y[[i]] <- y[[i]] - medians[[i]]
}
#> tracemem[0x7f80c5c3de20 -> 0x7f80c48de210]: 

While it’s not hard to determine when a copy is made, it is hard to prevent it. If you find yourself resorting to exotic tricks to avoid copies, it may be time to rewrite your function in C++, as described in Chapter 25.

2.5.2 Environments

You’ll learn more about environments in Chapter 7, but it’s important to mention them here because their behaviour is different from that of other objects: environments are always modified in place. This property is sometimes described as reference semantics because when you modify an environment all existing bindings to that environment continue to have the same reference.

Take this environment, which we bind to e1 and e2:

e1 <- rlang::env(a = 1, b = 2, c = 3)
e2 <- e1

If we change a binding, the environment is modified in place:

e1$c <- 4
e2$c
#> [1] 4

This basic idea can be used to create functions that “remember” their previous state. See Section 10.2.4 for more details. This property is also used to implement the R6 object-oriented programming system, the topic of Chapter 14.

One consequence of this is that environments can contain themselves:

e <- rlang::env()
e$self <- e

ref(e)
#> █ [1:0x7fe114214cd8] <env> 
#> └─self = [1:0x7fe114214cd8]

This is a unique property of environments!

2.5.3 Exercises

1. Explain why the following code doesn’t create a circular list.

   x <- list()
   x[[1]] <- x

2. Wrap the two methods for subtracting medians into two functions, then use the ‘bench’ package¹⁷ to carefully compare their speeds. How does performance change as the number of columns increase?

3. What happens if you attempt to use tracemem() on an environment?