2 Creating Plugins
2.1 Plugin Architecture
As you may have seen from configuring the Job Launcher, plugins are simply executables that are started by the main Job Launcher executable, rstudio-launcher. Once your plugin is configured as an available plugin and the rstudio-launcher process starts, it will execute your plugin process as a child of itself, and communicate with it via stdin/stdout.
As requests come in to the Job Launcher from RStudio products, the Job Launcher creates simple JSON-based requests and sends them to your plugin through stdin. Your plugin should respond to each request as appropriate, and send responses back to the main rstudio-launcher process via stdout.
2.2 Plugin Configuration
Each plugin is passed a set of configuration values as arguments to the plugin when it is started. These configuration options contain default values determined by the rstudio-launcher binary, and include configuration options set in the /etc/rstudio/launcher.conf config file.
The following configuration options are passed to the plugin when started:
| Argument name | Description |
|---|---|
| plugin-name | The plugin’s name, which is the cluster name configured for the plugin within the main launcher configuration file. |
| server-user | The name of the user that the process should be run as. Plugins start with root privilege and SHOULD lower their privilege to run as the user specified by this argument. Plugins start with root privilege to allow for user impersonation when communicating with cluster software that requires it. |
| enable-debug-logging | A 0 or 1 indicating whether or not debug logging should be enabled. If so, a higher level of verbosity should be used when logging information. |
| scratch-path | The path assigned by the Job Launcher that the plugin is allowed to write to. This is where the plugin may store temporary data, such as cached information about jobs. |
| heartbeat-interval-seconds | The number of seconds in between heartbeats. If this is 0, then heartbeats should be disabled. If the plugin misses 3 heartbeats in a row, the Job Launcher considers it to be in an invalid state and will kill and restart the plugin. For more information, see the subsequent section on heartbeat messages. |
| config-file | Path to the configuration file for this plugin (user-specified). This should override any default configuration file provided by the plugin. If the default file should be used, this argument will not be present. |
| launcher-config-file | Path to the Job Launcher’s configuration file. This is not needed by the vast majority of plugins. |
Each plugin should also allow for configuration overrides of these values and any additional necessary configuration via file, located at /etc/rstudio/launcher.<plugin name>.conf. Note that this should be overrideable, as specified in the config-file argument mentioned above.
2.3 Plugin Messages
The following sections outline the various requests and responses that can be sent and received to and from the rstudio-launcher process.
2.3.1 General message structure
Each message has a 4-byte big-endian message length indicating the length of the message to be read, followed by the message payload. While not strictly necessary for communication over stdin/stdout, the message length makes it easy to delineate message boundaries. Big-endian is chosen in order to make any potential future transition to over-network communication with plugins fairly pain free.
Each message payload is one of three types: PluginRequest, PluginResponse, or ErrorResponse, which is simply a JSON object containing, at a minimum, the following fields:
PluginRequest (sent by the rstudio-launcher binary to the plugin)
| Field name | Description |
|---|---|
| messageType | The type ID of the message. The ID for a given message type can be found in the section for the given message. |
| requestId | The monotonically increasing request ID for this request. This should be saved by the plugin for use in responses. |
| username | The username of the user that initiated the request. Used for auditing/security purposes. Some requests have the ability to specify * for this value, indicating all users’ information should be returned. This can occur if a super user (admin user) makes a request, or if authorization is disabled. |
| requestUsername | The actual username used when the request was submitted. This should be ignored in most plugins, and the username value above should be used. This field is provided for additional information if needed. |
PluginResponse (sent by the plugin to the rstudio-launcher binary)
| Field name | Description | Value |
|---|---|---|
| messageType | The type ID of the message. The ID for a given message type can be found in the section for the given message. | Int |
| requestId | The request ID for the request that initiated this response. This is used by the rstudio-launcher binary to properly determine which response belongs to which request. |
Int |
| responseId | The monotonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. Must start from 0. | Int |
ErrorResponse (sent by the plugin to indicate an error)
| Field name | Description | Value |
|---|---|---|
| messageType | The type ID of the message. | -1 |
| requestId | The request ID for the request that initiated this response. This is used by the rstudio-launcher binary to properly determine which response belongs to which request. |
Int |
| responseId | The monotonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. Must start from 0. | Int |
| errorCode | The error code. More details can be found in the Errors section. | Int |
| errorMessage | The error message. More details can be found in the Errors section. | String |
ErrorResponses can be sent in response to all requests. For more information, see Errors.
2.3.2 Heartbeats
As mentioned in the configuration option heartbeat-interval-seconds, heartbeats should be sent to the plugin once every heartbeat-interval-seconds seconds, and the plugin should respond back with a heartbeat message.
HeartbeatRequest
| Field name | Field value |
|---|---|
| messageType | 0 |
| requestId | 0 |
Sample payload:
{
"messageType": 0,
"requestId": 0
}HeartbeatResponse
| Field name | Field value |
|---|---|
| messageType | 0 |
| requestId | 0 |
| responseId | 0 |
Sample payload:
{
"messageType": 0,
"requestId": 0,
"responseId": 0
}Note that for heartbeats, the request and response IDs are always 0.
2.3.3 Bootstrap
BootstrapRequests are sent as the first message to the plugin, and the plugin should respond with a BootstrapResponse message when it is ready to start servicing requests.
BootstrapRequest
| Field name | Description | Field value |
|---|---|---|
| messageType | 1 | |
| requestId | 0 | |
| version | Object which describes a version. | Version. See below for details. |
Sample payload
{
"messageType": 1,
"requestId": 0,
"version" : { "major" : 1, "minor": 0, "patch": 2 }
}BootstrapResponse
| Field name | Description | Field value |
|---|---|---|
| messageType | 1 | |
| version | Object which describes a version. | Version. See below for details. |
Sample payload
{
"messageType": 1,
"requestId": 0,
"responseId": 0
"version" : { "major" : 1, "minor": 0, "patch": 0 }
}Version
| Field name | Description | Field value |
|---|---|---|
| major | The major version number. | Integer Number |
| minor | The minor version number. | Integer Number |
| patch | The patch version number. | Integer Number |
2.3.4 Cluster Info
Cluster info requests allow for the retrieval of information about a particular cluster targeted by a plugin.
ClusterInfoRequest
| Field name | Field value |
|---|---|
| messageType | 9 |
Sample payload:
{
"messageType": 9,
"requestId": 10
}ClusterInfoResponse
| Field name | Description | Field value |
|---|---|---|
| messageType | 8 | |
| supportsContainers | Whether or not the cluster allows containerization. | Boolean |
| queues | Array of available queue names. Only used for batch processing systems. | String array |
| config | Cluster-specific configuration options available for a job. | Job Config Array. See below for details. |
| resourceLimits | Available resource limits for submitted jobs, for example max CPU usage or max memory usage. | ResourceLimit array. See below for details. |
| placementConstraints | Available placement constraints for defining which node(s) a job may be placed on. | PlacementConstraint array. See below for details. |
For example:
ClusterInfoResponse
{
"messageType": 8,
"supportsContainers": false,
"queues": ["Prod", "Dev"],
"config": [
{"name": "SpecialVal", "valueType": "int"}
],
"resourceLimits": [
{"limitType": "memory", "defaultValue": "1024"},
{"limitType": "cpuCount", "maxValue": "4"}
],
"placementConstraints": [
{"name": "region", "value": "us"},
{"name": "region", "value": "eu"},
{"name": "disk", "value": "nvme"}
{"name": "disk", "value": "ssd"}
]
}JobConfig
| Field name | Description | Value |
|---|---|---|
| name | Name of the job config param | String |
| valueType | A string representing the value type. | Can be one of “string”, “int”, “float”, or “enum”. |
ResourceLimit
| Field name | Description | Value |
|---|---|---|
| limitType | A string representing the limit type. | Can be one of “cpuCount”, “cpuTime”, “memory” or “memorySwap”. |
| defaultValue | A string representing the default value that will be used for jobs that do not specify a limit. | String |
| maxValue | A string representing the maximum value that a job may use. Jobs submitted with a higher value will be denied. | String |
** PlacementConstraint**
| Field name | Description | Value |
|---|---|---|
| name | A string representing the name/key of the placement constraint. | String |
| value | A string representing a possible value for the given constraint. | String |
PlacementConstraints specify a grouping of name/value pairs that represent the available constraint name and its possible values. The value can be left empty if it is a free-entry field and not an enumeration of specifically allowed values. In the example above, we are advertising that placement can be constrained by both region and disk type, with us and eu being valid regions and nvme and ssd being valid disk types.
2.3.4.1 Submit Job
Submit job requests allow new jobs to be submitted to the cluster, which handles the scheduling of the job.
SubmitJobRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 2 | |
| username | The name of the user submitting the job. | String |
| job | The job object. | Object (see Swagger documentation for definition). |
A JobStateResponse is returned in response to a SubmitJobRequest. This will be described in the next section.
2.3.5 Job State
Job state requests allow users to get the current state of a specific job, or all of their jobs that match specific criteria.
JobStateRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 3 | |
| username | The name of the user requesting job information. If all users’ job information should be returned, this field will be "*". You must ensure the security of job objects and only allow authorized users access to their own jobs, as appropriate. | String |
| jobId | The job ID to get information for. If all of the user’s jobs information is requested, this will be “". If username is "”, all jobs (for all users) in the entire system should be returned. | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
| tags | If populated, return only jobs that match all specified tags. | String Array |
| startTime | A time in the format YYYY-MM-DDThh:mm:ss indicating the earliest UTC submission time. Only jobs submitted at or after this point in time will be included in the result set. |
String, optional. |
| endTime | A time in the format YYYY-MM-DDThh:mm:ss indicating the latest UTC submission time. Only jobs submitted at or before this point in time will be included in the result set. |
String, optional. |
| status | Desired job status. Only jobs that match this status will be included in the result set. | String, optional. |
| fields | The list of fields to return for the job(s). Return all fields if this is empty or unspecified. The id field must ALWAYS be returned. |
String Array, optional. |
JobStateResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 2 | |
| jobs | Array of jobs that meet the criteria. | Job Array |
Even if only one Job should be returned, jobs should be an array containing only one job.
2.3.6 Job Status Stream
A job status stream allows users to stream the status of a particular job or all jobs.
JobStatusRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 4 | |
| username | The name of the user requesting job information. If all users’ job statuses should be streamed, this field will be "*". You must ensure the security of job objects and only allow authorized users access to their own jobs, as appropriate. | String |
| jobId | The job ID to stream status for. If all of a particular users’ job statuses should be streamed, this will be "*". | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobStatusRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobStatusResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.
JobStatusResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 3 | |
| sequences | An array of stream sequences indicating which requests this update satisfies. | StreamSequence Array |
| jobId | The ID of the job which status has changed. | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
| jobName | The name of the job. | String |
| status | The status of the job. | String |
StreamSequence
| Field name | Description | Value |
|---|---|---|
| requestId | The requestId of a stream request that matches this update. |
Int |
| seqId | The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. |
Int |
2.3.6.1 Stream example
As an example, say we receive two job status stream requests, one for all of user bob’s jobs, and one for a specific job of user bob.
Request 1
{
"requestId": 14,
"messageType": 4,
"username": "bob",
"jobId": "*"
}Request 2
{
"requestId": 45,
"messageType": 4,
"username": "bob",
"jobId": "AFA2532XSAZ"
}Now, let’s say that the status of job AFA2532XSAZ has changed. The following response would be sent out.
Response 1
{
"messageType": 3,
"jobId": "AFA2532XSAZ",
"jobName": "My job",
"status": "Running",
"sequences": [
{ "requestId": 14, "seqId": 1 },
{ "requestId": 45, "seqId": 1 }
]
}Now, let’s say later on, some other job B23532AD has its status change. The following response would be sent out.
Response 2
{
"messageType": 3,
"jobId": "B23532AD",
"jobName": "Another job",
"status": "Killed",
"sequences": [
{ "requestId": 14, "seqId": 2 }
]
}Note that only the matching sequences are sent.
Now let’s say that one of the streams is canceled.
Request 3
{
"requestId": 45,
"messageType": 4,
"username": "bob",
"jobId": "AFA2532XSAZ",
"cancel": true
}Note that the requestId matches that of the original request so that the plugin knows which request to cancel.
Now, let’s say once more the job AFA2532XSAZ changes its status. Now that one of the streams was canceled, we will not send sequence information for that particular request anymore. The response would be:
Response 3
{
"messageType": 3,
"jobId": "AFA2532XSAZ",
"jobName": "My job",
"status": "Finished",
"sequences": [
{ "requestId": 14, "seqId": 3 }
]
}2.3.7 Job Control
The job control request allows users to modify their jobs to suspend, resume, stop (SIGTERM), or kill (SIGKILL) them.
JobControlRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 5 | |
| username | Username of the initiator of this request. "*" if requested by a super user or auth is disabled. You must ensure that users can only control their own jobs, as appropriate. | String |
| jobId | The job ID of the job to control. | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
| operation | The operation to perform. | JobOperation (described below). |
JobOperation (enum)
Suspend = 0,
Resume = 1,
Stop = 2,
Kill = 3JobControlResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 4 | |
| statusMessage | Human-friendly status message indicating the result of the operation. | String |
| operationComplete | Whether or not the operation has already fully completed successfully. Should be false if the request was merely submitted to the cluster and it must be processed asynchronously. | Boolean |
2.3.8 Job Output Stream
Job output streams allow users to stream the stdout/stderr of their jobs in real-time.
JobOutputRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 6 | |
| username | The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the output of their own jobs, as appropriate. | String |
| jobId | The ID of the job for which to get output. | String |
| outputType | The type of output to fetch. | JobOutputType (see below). |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
JobOutputType (enum)
Stdout = 0,
Stderr = 1,
Both = 2Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobOutputRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
JobOutputResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 5 | |
| seqId | Monotonically increasing sequence ID for this stream, starting at 1. | Int |
| output | Next chunk of output received from the job. | String |
| outputType | The type of the output (see JobOutputType) |
JobOutputType |
| complete | Flag indicating whether or not the output stream is complete. Set to true if the job has and will never have more output (such as if the job is finished and the output has been fully read). | Boolean |
Streaming should commence similarly to job status streaming (described above), except output streams are only ever associated with one request and as such do not have multiple sequences per response.
2.3.9 Job Resource Utilization Stream
Job resource utilization streams allow users to stream the most up-to-date resource utilization of their jobs in real-time. This is similar in concept to commands like top.
JobResourceUtilizationRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 7 | |
| username | The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the resource utilization of their own jobs, as appropriate. | String |
| jobId | The ID of the job for which to get resource utilization. | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobResourceUtilizationRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobResourceUtilizationResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.
JobResourceUtilizationResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 6 | |
| cpuPercent | The CPU percent used since the last streaming update was sent. May be null. | Decimal Number |
| cpuSeconds | The number of total elapsed CPU seconds. May be null. | Decimal Number |
| virtualMemory | The total number of MB of virtual memory in use. May be null. | Decimal Number |
| residentMemory | The total number of MB of resident (actual/physical) memory in use. May be null. | Decimal Number |
| sequences | An array of stream sequences indicating which requests this update satisfies. | StreamSequence Array |
| complete | Flag indicating whether or not the resource utilization stream is complete. Set to true if the job has and will never have more utilization (such as if the job is finished or killed). | Boolean |
StreamSequence
| Field name | Description | Value |
|---|---|---|
| requestId | The requestId of a stream request that matches this update. |
Int |
| seqId | The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. |
Int |
For an example stream flow, see the section Stream Example.
2.3.10 Job Network Request
The job network request allows users to fetch the network information about the host that is running a job.
JobNetworkRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 8 | |
| username | The username of the requestor. "*" if requested by a super user or auth is disabled. You must ensure that users can only view the network info of their own jobs, as appropriate. | String |
| jobId | The ID of the job. | String |
| encodedJobId | The job ID that the user requested, which is an encoded value which includes cluster information. This is different from jobId, as it is the identifier that users see when interacting with the Launcher API, which includes additional metadata needed by the launcher to route the request to the plugin. This should be ignored by most plugins, but it is provided for additional information. |
String |
JobNetworkResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 7 | |
| host | The hostname or IP address of the host that is running the job. | String |
| ipAddresses | A list of IP addresses of the host. | String Array |
2.3.11 Errors
An error may be sent in response to a request if an error has occurred. For example, if a request comes in for a job that does not exist, a JobNotFound error can be returned. The various ErrorResponse error codes that can be sent in case of an error are detailed below.
UnknownError = 0,
RequestNotSupported = 1,
InvalidRequest = 2,
JobNotFound = 3,
JobNotRunning = 6,
JobOutputNotFound = 7,
InvalidJobState = 8,
JobControlFailure = 9,
UnsupportedVersionError = 10For example, to indicate a JobNotFound error:
ErrorResponse
{
"messageType": -1,
"errorCode": 3,
"errorMessage": "Job does not exist"
}