2 Creating Plugins

2.1 Plugin Architecture

As you may have seen from configuring the Job Launcher, plugins are simply executables that are started by the main Job Launcher executable, rstudio-launcher. Once your plugin is configured as an available plugin and the rstudio-launcher process starts, it will execute your plugin process as a child of itself, and communicate with it via stdin/stdout.

As requests come in to the Job Launcher from RStudio products, the Job Launcher creates simple JSON-based requests and sends them to your plugin through stdin. Your plugin should respond to each request as appropriate, and send responses back to the main rstudio-launcher process via stdout.

2.2 Plugin Configuration

Each plugin is passed a set of configuration values as arguments to the plugin when it is started. These configuration options contain default values determined by the rstudio-launcher binary, and include configuration options set in the /etc/rstudio/launcher.conf config file.

The following configuration options are passed to the plugin when started:

Argument name Description
plugin-name The plugin’s name, which is the cluster name configured for the plugin within the main launcher configuration file.
server-user The name of the user that the process should be run as. Plugins start with root privilege and SHOULD lower their privilege to run as the user specified by this argument. Plugins start with root privilege to allow for user impersonation when communicating with cluster software that requires it.
enable-debug-logging A 0 or 1 indicating whether or not debug logging should be enabled. If so, a higher level of verbosity should be used when logging information.
scratch-path The path assigned by the Job Launcher that the plugin is allowed to write to. This is where the plugin may store temporary data, such as cached information about jobs.
heartbeat-interval-seconds The number of seconds in between heartbeats. If this is 0, then heartbeats should be disabled. If the plugin misses 3 heartbeats in a row, the Job Launcher considers it to be in an invalid state and will kill and restart the plugin. For more information, see the subsequent section on heartbeat messages.
config-file Path to the configuration file for this plugin (user-specified). This should override any default configuration file provided by the plugin. If the default file should be used, this argument will not be present.
launcher-config-file Path to the Job Launcher’s configuration file. This is not needed by the vast majority of plugins.

Each plugin should also allow for configuration overrides of these values and any additional necessary configuration via file, located at /etc/rstudio/launcher.<plugin name>.conf. Note that this should be overrideable, as specified in the config-file argument mentioned above.

2.3 Plugin Messages

The following sections outline the various requests and responses that can be sent and received to and from the rstudio-launcher process.

2.3.1 General message structure

Each message has a 4-byte big-endian message length indicating the length of the message to be read, followed by the message payload. While not strictly necessary for communication over stdin/stdout, the message length makes it easy to delineate message boundaries. Big-endian is chosen in order to make any potential future transition to over-network communication with plugins fairly pain free.

Each message payload is one of three types: PluginRequest, PluginResponse, or ErrorResponse, which is simply a JSON object containing, at a minimum, the following fields:

PluginRequest (sent by the rstudio-launcher binary to the plugin)

Field name Description
messageType The type ID of the message. The ID for a given message type can be found in the section for the given message.
requestId The monotonically increasing request ID for this request. This should be saved by the plugin for use in responses.
username The username of the user that initiated the request. Used for auditing/security purposes. Some requests have the ability to specify * for this value, indicating all users’ information should be returned. This can occur if a super user (admin user) makes a request, or if authorization is disabled.
requestUsername The actual username used when the request was submitted. This should be ignored in most plugins, and the username value above should be used. This field is provided for additional information if needed.

PluginResponse (sent by the plugin to the rstudio-launcher binary)

Field name Description Value
messageType The type ID of the message. The ID for a given message type can be found in the section for the given message. Int
requestId The request ID for the request that initiated this response. This is used by the rstudio-launcher binary to properly determine which response belongs to which request. Int
responseId The monotonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. Must start from 0. Int

ErrorResponse (sent by the plugin to indicate an error)

Field name Description Value
messageType The type ID of the message. -1
requestId The request ID for the request that initiated this response. This is used by the rstudio-launcher binary to properly determine which response belongs to which request. Int
responseId The monotonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. Must start from 0. Int
errorCode The error code. More details can be found in the Errors section. Int
errorMessage The error message. More details can be found in the Errors section. String

ErrorResponses can be sent in response to all requests. For more information, see Errors.

2.3.2 Heartbeats

As mentioned in the configuration option heartbeat-interval-seconds, heartbeats should be sent to the plugin once every heartbeat-interval-seconds seconds, and the plugin should respond back with a heartbeat message.

HeartbeatRequest

Field name Field value
messageType 0
requestId 0

Sample payload:

{
   "messageType": 0,
   "requestId": 0
}

HeartbeatResponse

Field name Field value
messageType 0
requestId 0
responseId 0

Sample payload:

{
   "messageType": 0,
   "requestId": 0,
   "responseId": 0
}

Note that for heartbeats, the request and response IDs are always 0.

2.3.3 Bootstrap

BootstrapRequests are sent as the first message to the plugin, and the plugin should respond with a BootstrapResponse message when it is ready to start servicing requests.

BootstrapRequest

Field name Description Field value
messageType 1
requestId 0
version Object which describes a version. Version. See below for details.

Sample payload

{
   "messageType": 1,
   "requestId": 0,
   "version" : { "major" : 1, "minor": 0, "patch": 2 }
}

BootstrapResponse

Field name Description Field value
messageType 1
version Object which describes a version. Version. See below for details.

Sample payload

{
   "messageType": 1,
   "requestId": 0,
   "responseId": 0
   "version" : { "major" : 1, "minor": 0, "patch": 0 }
}

Version

Field name Description Field value
major The major version number. Integer Number
minor The minor version number. Integer Number
patch The patch version number. Integer Number

2.3.4 Cluster Info

Cluster info requests allow for the retrieval of information about a particular cluster targeted by a plugin.

ClusterInfoRequest

Field name Field value
messageType 9

Sample payload:

{
   "messageType": 9,
   "requestId": 10
}

ClusterInfoResponse

Field name Description Field value
messageType 8
supportsContainers Whether or not the cluster allows containerization. Boolean
queues Array of available queue names. Only used for batch processing systems. String array
config Cluster-specific configuration options available for a job. Job Config Array. See below for details.
resourceLimits Available resource limits for submitted jobs, for example max CPU usage or max memory usage. ResourceLimit array. See below for details.
placementConstraints Available placement constraints for defining which node(s) a job may be placed on. PlacementConstraint array. See below for details.

For example:

ClusterInfoResponse

{
   "messageType": 8,
   "supportsContainers": false,
   "queues": ["Prod", "Dev"],
   "config": [
      {"name": "SpecialVal", "valueType": "int"}
   ],
   "resourceLimits": [
      {"limitType": "memory", "defaultValue": "1024"},
      {"limitType": "cpuCount", "maxValue": "4"}
   ],
   "placementConstraints": [
      {"name": "region", "value": "us"},
      {"name": "region", "value": "eu"},
      {"name": "disk", "value": "nvme"}
      {"name": "disk", "value": "ssd"}
   ]
}

JobConfig

Field name Description Value
name Name of the job config param String
valueType A string representing the value type. Can be one of “string”, “int”, “float”, or “enum”.

ResourceLimit

Field name Description Value
limitType A string representing the limit type. Can be one of “cpuCount”, “cpuTime”, “memory” or “memorySwap”.
defaultValue A string representing the default value that will be used for jobs that do not specify a limit. String
maxValue A string representing the maximum value that a job may use. Jobs submitted with a higher value will be denied. String

** PlacementConstraint**

Field name Description Value
name A string representing the name/key of the placement constraint. String
value A string representing a possible value for the given constraint. String

PlacementConstraints specify a grouping of name/value pairs that represent the available constraint name and its possible values. The value can be left empty if it is a free-entry field and not an enumeration of specifically allowed values. In the example above, we are advertising that placement can be constrained by both region and disk type, with us and eu being valid regions and nvme and ssd being valid disk types.

2.3.4.1 Submit Job

Submit job requests allow new jobs to be submitted to the cluster, which handles the scheduling of the job.

SubmitJobRequest

Field name Description Value
messageType 2
username The name of the user submitting the job. String
job The job object. Object (see Swagger documentation for definition).

A JobStateResponse is returned in response to a SubmitJobRequest. This will be described in the next section.

2.3.5 Job State

Job state requests allow users to get the current state of a specific job, or all of their jobs that match specific criteria.

JobStateRequest

Field name Description Value
messageType 3
username The name of the user requesting job information. If all users’ job information should be returned, this field will be "*". You must ensure the security of job objects and only allow authorized users access to their own jobs, as appropriate. String
jobId The job ID to get information for. If all of the user’s jobs information is requested, this will be “". If username is "”, all jobs (for all users) in the entire system should be returned. String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String
tags If populated, return only jobs that match all specified tags. String Array
startTime A time in the format YYYY-MM-DDThh:mm:ss indicating the earliest UTC submission time. Only jobs submitted at or after this point in time will be included in the result set. String, optional.
endTime A time in the format YYYY-MM-DDThh:mm:ss indicating the latest UTC submission time. Only jobs submitted at or before this point in time will be included in the result set. String, optional.
status Desired job status. Only jobs that match this status will be included in the result set. String, optional.
fields The list of fields to return for the job(s). Return all fields if this is empty or unspecified. The id field must ALWAYS be returned. String Array, optional.

JobStateResponse

Field name Description Value
messageType 2
jobs Array of jobs that meet the criteria. Job Array

Even if only one Job should be returned, jobs should be an array containing only one job.

2.3.6 Job Status Stream

A job status stream allows users to stream the status of a particular job or all jobs.

JobStatusRequest

Field name Description Value
messageType 4
username The name of the user requesting job information. If all users’ job statuses should be streamed, this field will be "*". You must ensure the security of job objects and only allow authorized users access to their own jobs, as appropriate. String
jobId The job ID to stream status for. If all of a particular users’ job statuses should be streamed, this will be "*". String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String
cancel Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. Boolean

Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobStatusRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.

Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobStatusResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.

JobStatusResponse

Field name Description Value
messageType 3
sequences An array of stream sequences indicating which requests this update satisfies. StreamSequence Array
jobId The ID of the job which status has changed. String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String
jobName The name of the job. String
status The status of the job. String

StreamSequence

Field name Description Value
requestId The requestId of a stream request that matches this update. Int
seqId The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. Int

2.3.6.1 Stream example

As an example, say we receive two job status stream requests, one for all of user bob’s jobs, and one for a specific job of user bob.

Request 1

{
   "requestId": 14,
   "messageType": 4,
   "username": "bob",
   "jobId": "*"
}

Request 2

{
   "requestId": 45,
   "messageType": 4,
   "username": "bob",
   "jobId": "AFA2532XSAZ"
}

Now, let’s say that the status of job AFA2532XSAZ has changed. The following response would be sent out.

Response 1

{
   "messageType": 3,
   "jobId": "AFA2532XSAZ",
   "jobName": "My job",
   "status": "Running",
   "sequences": [
      { "requestId": 14, "seqId": 1 },
      { "requestId": 45, "seqId": 1 }
   ]
}

Now, let’s say later on, some other job B23532AD has its status change. The following response would be sent out.

Response 2

{
   "messageType": 3,
   "jobId": "B23532AD",
   "jobName": "Another job",
   "status": "Killed",
   "sequences": [
      { "requestId": 14, "seqId": 2 }
   ]
}

Note that only the matching sequences are sent.

Now let’s say that one of the streams is canceled.

Request 3

{
   "requestId": 45,
   "messageType": 4,
   "username": "bob",
   "jobId": "AFA2532XSAZ",
   "cancel": true
}

Note that the requestId matches that of the original request so that the plugin knows which request to cancel.

Now, let’s say once more the job AFA2532XSAZ changes its status. Now that one of the streams was canceled, we will not send sequence information for that particular request anymore. The response would be:

Response 3

{
   "messageType": 3,
   "jobId": "AFA2532XSAZ",
   "jobName": "My job",
   "status": "Finished",
   "sequences": [
      { "requestId": 14, "seqId": 3 }
   ]
}

2.3.7 Job Control

The job control request allows users to modify their jobs to suspend, resume, stop (SIGTERM), or kill (SIGKILL) them.

JobControlRequest

Field name Description Value
messageType 5
username Username of the initiator of this request. "*" if requested by a super user or auth is disabled. You must ensure that users can only control their own jobs, as appropriate. String
jobId The job ID of the job to control. String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String
operation The operation to perform. JobOperation (described below).

JobOperation (enum)

Suspend = 0,
Resume = 1,
Stop = 2,
Kill = 3

JobControlResponse

Field name Description Value
messageType 4
statusMessage Human-friendly status message indicating the result of the operation. String
operationComplete Whether or not the operation has already fully completed successfully. Should be false if the request was merely submitted to the cluster and it must be processed asynchronously. Boolean

2.3.8 Job Output Stream

Job output streams allow users to stream the stdout/stderr of their jobs in real-time.

JobOutputRequest

Field name Description Value
messageType 6
username The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the output of their own jobs, as appropriate. String
jobId The ID of the job for which to get output. String
outputType The type of output to fetch. JobOutputType (see below).
cancel Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. Boolean

JobOutputType (enum)

Stdout = 0,
Stderr = 1,
Both = 2

Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobOutputRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.

JobOutputResponse

Field name Description Value
messageType 5
seqId Monotonically increasing sequence ID for this stream, starting at 1. Int
output Next chunk of output received from the job. String
outputType The type of the output (see JobOutputType) JobOutputType
complete Flag indicating whether or not the output stream is complete. Set to true if the job has and will never have more output (such as if the job is finished and the output has been fully read). Boolean

Streaming should commence similarly to job status streaming (described above), except output streams are only ever associated with one request and as such do not have multiple sequences per response.

2.3.9 Job Resource Utilization Stream

Job resource utilization streams allow users to stream the most up-to-date resource utilization of their jobs in real-time. This is similar in concept to commands like top.

JobResourceUtilizationRequest

Field name Description Value
messageType 7
username The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the resource utilization of their own jobs, as appropriate. String
jobId The ID of the job for which to get resource utilization. String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String
cancel Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. Boolean

Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobResourceUtilizationRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.

Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobResourceUtilizationResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.

JobResourceUtilizationResponse

Field name Description Value
messageType 6
cpuPercent The CPU percent used since the last streaming update was sent. May be null. Decimal Number
cpuSeconds The number of total elapsed CPU seconds. May be null. Decimal Number
virtualMemory The total number of MB of virtual memory in use. May be null. Decimal Number
residentMemory The total number of MB of resident (actual/physical) memory in use. May be null. Decimal Number
sequences An array of stream sequences indicating which requests this update satisfies. StreamSequence Array
complete Flag indicating whether or not the resource utilization stream is complete. Set to true if the job has and will never have more utilization (such as if the job is finished or killed). Boolean

StreamSequence

Field name Description Value
requestId The requestId of a stream request that matches this update. Int
seqId The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. Int

For an example stream flow, see the section Stream Example.

2.3.10 Job Network Request

The job network request allows users to fetch the network information about the host that is running a job.

JobNetworkRequest

Field name Description Value
messageType 8
username The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the network info of their own jobs, as appropriate. String
jobId The ID of the job. String
encodedJobId The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. String

JobNetworkResponse

Field name Description Value
messageType 7
host The hostname or IP address of the host that is running the job. String
ipAddresses A list of IP addresses of the host. String Array

2.3.11 Errors

An error may be sent in response to a request if an error has occurred. For example, if a request comes in for a job that does not exist, a JobNotFound error can be returned. The various ErrorResponse error codes that can be sent in case of an error are detailed below.

UnknownError = 0,
RequestNotSupported = 1,
InvalidRequest = 2,
JobNotFound = 3,
JobNotRunning = 6,
JobOutputNotFound = 7,
InvalidJobState = 8,
JobControlFailure = 9,
UnsupportedVersionError = 10

For example, to indicate a JobNotFound error:

ErrorResponse

{
   "messageType": -1,
   "errorCode": 3,
   "errorMessage": "Job does not exist"
}