Notes on webscraping

Iteration

I write another function, which will create a tibble¹⁵ for a job post, with the information I want. I can then use a for loop to apply the function to each job post and to put the result into a list of tibbles.

After I have a list of tibbles, I can combine them into a single dataset using dplyr’s bind_rows().

## function for cleaning all elements I want on a page ----

clean_ad <- function(link) {
  page_html <- read_html(link) ## get HTML
  ## the list of items I want in each post
  title <- clean_element(page_html, "h1") ## parse HTML
  description <- clean_element(page_html, ".description-website-container")
  jobtype <- clean_element(page_html, ".attribute:nth-child(4) .value")
  employer <- clean_element(page_html, ".attribute:nth-child(3) .value")
  location <- clean_element(page_html, ".attribute:nth-child(1) .value")
  time <- Sys.time() ## current time

  ## I put the selected post info in a tibble
  dat <-
    tibble(
      title,
      description,
      jobtype,
      location,
      time
    )
  return(dat)
}

## set up ----

## fixing" link addresses
links <- paste0("https://www.gumtree.co.za", links)
links <- sample(links, 10) ## sampling 10 random links for the example

## I create a list (for storing tibbles), and a for loop to iterate through links
output <- list()

## iterating through links with a for loop ----

for (i in 1:length(links)) {
  deets <- clean_ad(links[i]) ## my function for extracting selected content from a post
  output[[i]] <- deets ## add to list
  Sys.sleep(2) ## rest
}

## combining all tibbles in the list into a single big tibble
all <- bind_rows(output) ## fab!

glimpse(all)

> Rows: 10
> Columns: 5
> $ title       <chr> "Handyman /driver /farmhand", "Permanent Admin Assistant …
> $ description <chr> "DescriptionHandyman /driver /farmhand. An entry level jo…
> $ jobtype     <chr> NA, NA, NA, NA, NA, NA, "Non EE/AA", NA, "EE/AA", NA
> $ location    <chr> "Howick, Midlands", "Somerset West, Helderberg", "Inner C…
> $ time        <dttm> 2020-12-16 14:22:19, 2020-12-16 14:22:22, 2020-12-16 14:…

all$description <- str_trunc(all$description, 100, "right") ## truncate
all$description <- sub("^Description", "", all$description)

## I'm using kable to make an html table for the site
all %>%
  kable() %>%
  kable_paper(bootstrap_options = c("striped", "responsive"))

Nice!

How would one approach multiple pages?

One (hacky) approach is to use the URL and another for loop.¹⁶

In the example below, I extract the job links for the first 3 pages:

## set-up ----

pages <- 3 ## let's just do 3
job_list <- list()
link <- "https://www.gumtree.co.za/s-jobs/page-" ## url fragment

## iterate over pages ----

for (i in 1:pages) {
  jobs <-
    read_html(paste0(link, i, "/v1c8p", i, "?q=entry+level")) ## using paste0
  links <-
    jobs %>%
    html_nodes(".related-ad-title") %>%
    html_attr("href") ## get links
  job_list[[i]] <- links ## add to list
  Sys.sleep(2) ## rest
}

links <- unlist(job_list) ## make a single list
links <- paste0("https://www.gumtree.co.za", links)
head(links, 10) ## looks good!

>  [1] "https://www.gumtree.co.za/a-logistics-jobs/eastern-pretoria/junior-warehouse-controller-+-pta-non+ee/1008328733720911316948209"                   
>  [2] "https://www.gumtree.co.za/a-internship-jobs/durbanville/are-you-young-and-unemployed-are-you-looking-for-an-opportunity/1008463343470912436560209"
>  [3] "https://www.gumtree.co.za/a-clerical-data-capturing-jobs/somerset-west/permanent-admin-assistant-financial-services/1008560434300912467960009"    
>  [4] "https://www.gumtree.co.za/a-hr-jobs/city-centre/recruitment-officer/1008558338510910255520809"                                                    
>  [5] "https://www.gumtree.co.za/a-hr-jobs/city-centre/human-resources-assistant/1008558343790910255520809"                                              
>  [6] "https://www.gumtree.co.za/a-office-jobs/brackenfell/office-assistant-brackenfell-r8-000-+-r9-000-pm/1008557939220911146640309"                    
>  [7] "https://www.gumtree.co.za/a-general-worker-jobs/somerset-west/permanent-administration-assistant/1008459525670911102256409"                       
>  [8] "https://www.gumtree.co.za/a-logistics-jobs/paarl/permanent-jnr-logistics-coordinator/1008548513350911102256409"                                   
>  [9] "https://www.gumtree.co.za/a-fmcg-jobs/durbanville/storeman-pharmaceutical-fmcg/1008551592230911341410109"                                         
> [10] "https://www.gumtree.co.za/a-accounting-finance-jobs/paarl/permanent-jnr-financial-accountant/1008548473800911102256409"

Things to think about

Make sure your script can handle errors¹⁷:

• CSS-selectors change over time (if you’re planning to run your script again in future, you should check)
• links stop working (page names change, or they stop existing)
• not all pages are necessarily the same (some elements may be missing from some pages) — see the “job type” variable above, for instance
• sites time out (that can break your script if you don’t account for it)
• the script might take a long time to run; how can you track progress?
• what other things might break your script?

Putting it all together

Approach: One function to clean an individual element; One function to clean an individual post (clean multiple elements and store the result as a tibble); One loop to extract links to individual posts over multiple pages and to store these in a list; One loop to iterate over a list of links, to apply the post cleaning function, and to store the result in a list (a list of tibbles); finally, to collapse the list of tibbles into a single data set, and to export to your chosen location.

Bells and whistles: I use the purrr package’s possibly() function to deal with errors (so my script doesn’t break when a link is broken); I use the progress package to show me, in my console, how far along the scraping is.

## Libraries ----

library(tidyverse) ## for data manipulation (various packages)
library(rvest) ## for scraping
library(textclean) ## some cleaning functions
library(progress) ## for tracking progress in console

## Function 1: Clean an element ----

clean_element <- function(page_html, selector) {
  output <- page_html %>%
    html_nodes(selector) %>%
    html_text() %>%
    replace_html() %>% ## strip html markup (if any)
    str_squish() ## remove leading and trailing whitespace
  output <- ifelse(length(output) == 0, NA, output) ## in case an element is missing
  return(output)
}

## Function 2: Clean an ad ----

clean_ad <- function(link) {
  page_html <- read_html(link) ## get HTML
  ## the list of items I want in each post
  title <- clean_element(page_html, "h1") ## parse HTML
  description <- clean_element(page_html, ".description-website-container")
  jobtype <- clean_element(page_html, ".attribute:nth-child(4) .value")
  employer <- clean_element(page_html, ".attribute:nth-child(3) .value")
  location <- clean_element(page_html, ".attribute:nth-child(1) .value")
  time <- Sys.time() ## current time

  ## I put the selected post info in a tibble
  dat <-
    tibble(
      title,
      description,
      jobtype,
      location,
      time
    )
  return(dat)
}

## Loop 1: Get my links ----

pages <- 5 ## let's just do 5
job_list <- list()
link <- "https://www.gumtree.co.za/s-jobs/page-" ## url fragment

for (i in 1:pages) {
  jobs <-
    read_html(paste0(link, i, "/v1c8p", i)) ## using paste0
  links <-
    jobs %>%
    html_nodes(".related-ad-title") %>%
    html_attr("href") ## get links
  job_list[[i]] <- links ## add to list
  Sys.sleep(2) ## rest
}

links <- unlist(job_list) ## make a single list
links <- paste0("https://www.gumtree.co.za", links)

## Trouble shooting ----

links <- sample(links, 20) ## sampling 20 (comment this out for more)
head(links) ## looks good!
links <- c(links, "ww.brokenlink.co.bb") ## append faulty link to test error handling

## Let's go! ----

## track progress
total <- length(links)
pb <- progress_bar$new(format = "[:bar] :current/:total (:percent)", total = total)

## error handling
safe_clean_ad <- possibly(clean_ad, NA) ## see purrr package

## list
output <- list()

## the loop

for (i in 1:total) {
  pb$tick()
  deets <- safe_clean_ad(links[i])
  output[[i]] <- deets # add to list
  Sys.sleep(2) ## resting
}

## combining all tibbles in the list into a single big tibble

all <- output[!is.na(output)] ## remove empty tibbles, if any
all <- bind_rows(all) ## Fab!
glimpse(all)

write_csv(all, "myfile.csv") ## export