Running background R jobs in Shiny

Shiny is a great tool for creating interactive dashboards and websites in R. It uses a reactive framework so that as buttons, dropdowns, and other inputs are adjusted the outputs and intermediate values are thoughtfully updated. Thanks to the reactive approach the R code in an app only needs to run when it’s relevant for the front-end. The downside of this approach is that occasionally you have activities that don’t fit into the reactive style and it’s hard to figure out how to set them up in Shiny.

One such task which has plagued me multiple times is trying to have Shiny trigger a background task. Typically this is a long-running set of code that I want to make sure both doesn’t cause the front-end to freeze while it’s running and also will continue to run after the user’s browser is closed.

This week, as part of a larger project I am working on I have found a solution that I think works really well for Shiny background tasks. It’s based on an solution I saw created by Barret Schloerke in a GitHub issue about plumber. I modified Barret’s idea slightly, but at it’s core it’s the same (and also works in Plumber too!).

The idea is to use the {callr} package, which lets you from within R start a separate R process that runs concurrently. {callr} is a really useful package when you have R tasks that you want to run without interrupting your existing R process, but it’s a bit tricky to use in Shiny since Shiny so aggressively deletes variables that aren’t reactive.

To get {callr} to run a background job with Shiny, you need to do two things:

• Use the function callr::r_bg() which will start the new R process in the background and not cause the current process to wait for it to complete.
• Store the resulting r_process object created by callr::r_bg() in a list outside of the Shiny server function. So far as I can tell this is the only way to avoid Shiny ending the process if the user’s session ends. You need to use the <<- operator to edit the the value of the list outside of the server function.

Here is a simple example showing the background process in action. This UI contains only a button that when pressed, starts a job to save a csv file which takes ten seconds. The job will finish even if you close the browser, so long as Shiny is running.

Warning: if you use the “Run App” button in RStudio to launch the app you will have a medicore experience. In that situation closing the RStudio browser that automatically launches will cause Shiny to stop running, and thus cancel the background task. Ideally when a session ends Shiny will keep running, which is what happens in all browsers except the RStudio one. To avoid this issue, run the Shiny app with the command shiny::runApp(launch.browser = FALSE), and then go to the URL that the message says Shiny is listening on (ex: Listening on http://127.0.0.1:3263).

library(shiny)
library(uuid)

ui <- fluidPage(
    titlePanel("Test background job"),
    actionButton("startjob","Start Job"),
)

jobs <- list()

# the toy example job
run_token <- function(token){
    Sys.sleep(10)
    write.csv(data.frame(),paste0(token,".csv"))
}

server <- function(input, output, session) {
    observe({
        if(input$startjob == 1){
            token <- UUIDgenerate()
            message(paste0("running task for token: ", token))
            # the if statement is to avoid rerunning a job again
            if(is.null(jobs[[token]])){
                # call the job in the background session
                jobs[[token]] <<- callr::r_bg(run_token, args = list(token = token))
            }
        }
    })
}

shinyApp(ui = ui, server = server)

While this is a pretty bland example, there are lots of practical applications to this, such as starting long-running data manipulations or having Shiny do HTTP calls to other services. The only thing that should interrupt the background task is the Shiny app shutting down, so take care when using services like shinyapps.io which automatically shuts down an idle server after 15 minutes.