R Service Bus – Documentation

Documentation: Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

{#rsb-architecture}

The architecture of RSB is organized around a messaging core that handles job requests and results.

The goal of this messaging core is twofold:

it throttles the workload towards pool(s) of remotely managed R nodes, which are the ones in charge of actually processing the jobs,
it stores the results until they get propagated to their relevant destination (application specific result files for the REST API, immediate response for the SOAP API, emails or response directories).

The messaging core consists in an embedded JMS provider (ActiveMQ) which hosts queues that are specific to the applications submitting R jobs. This allows a segregation of the submitted jobs and their related results. It also opens the door for creating application specific job workers in order to boost the priority of job execution (discussed further on).

The job workers can optionally report statistics on a pluggable store, like Redis.

Job Acceptors and Result Handlers

Specific job acceptors and result handlers are deployed as frontal elements to the messaging core described previously. They handle the specifics of the particular protocol, format or transport they deal and take care of communicating with the job queues and reading responses from the response queues.

Here is a summary of the different job acceptors and their mechanics:

REST: handles raw (API style) and multi-part (web form style) HTTP requests. It works in an asynchronous manner: jobs are submitted and acknowledged with a unique ID, which is used to later on retrieve the corresponding result. Worker responses are dequeued from the response queues and immediately written to a result store, from where they're served over HTTP. The REST API offers functions for browsing and deleting these result files.
SOAP: MTOM-enabled web service that offers synchronous semantics over the shared messaging core, this by consuming the result queue in a blocking manner on behalf of the client performing the SOAP call.
POP3/IMAP & SMTP: works asynchronously, regularly polling an inbox for jobs and routing responses over SMTP
Directory Deposit: works asynchronously too, scanning a deposit directory and writing results back into another directory. Processed jobs are archived in a third directory.

Though they present very different semantics, all the above acceptors use the same messaging core, which allows a complete decoupling of the job accepting phase from the job processing one.

Jobs and results are transported as serialized objects over JMS and respectively extend AbstractJob and AbstractResult.

Job Workers

Job Types

RSB supports two main types of jobs:

Multi-files jobs: usually composed of several data files and an R script.
Function call jobs: a single String payload (XML or JSON) passed to a predefined R function.

Default configuration: No Application Bias

{#img-job_workers_default}

By default, a single pool of JMS transacted message receivers consume all the R jobs in all the r.jobs.* queues. This JMS pool is sized to match the number of nodes in the targeted RPooli pool. That way, each concurrent worker thread that will fetch a job and request an R node to process it will never starve.

Optional configuration: Application Dedicated Pools

{#img-job_workers_app_bias}

If an application needs to have its jobs processed with a higher precedence than the other applications, the idea is to create a dedicated pool of transacted JMS message receivers sized accordingly with the other dedicated pools, the common pool and the actual number of available nodes on RPooli.

A JMS filter is used on the common pool of receivers so they do not consume jobs of the privileged application. Conversely, the consumers of the privileged application work only with the specific jobs queue of the privileged application.

Optional configuration: Application Dedicated RPooli Instance

By default, RSB connects to a single RPooli instance and uses the nodes it finds there for its R processing needs.

It is also possible to configure RSB to use application specific RPooli instances. This allows to guarantee that the execution of certain jobs only occur on a server where R is configured with the desired extensions or where the desired operating system is in use.

Job Error Handling

RPooli Node Starvation

If the processing of a particular R job fails because no RPooli node was available, processing will be retried again. This happens by letting the exception received from RPooli client (because of the unavailability of node) roll-back the transaction into which the JMS message has been consumed. The JMS provider will then redeliver the message, after a configured amount of time and for a configured number of attempts.

If the maximum number of retry attempts is reached, the failing job will be sent to a dead letter queue consumed by an RSB service which takes care of reporting the error to the original application (via the relevant results queue) and to the RSB admin.

Faulty Job

If a job is not processable because it contains error (wrong XML, bad R syntax…) it will not be retried. It is immediately rejected and reported faulty to the calling application through the relevant results queue.

Documentation: Installation

Mon, 01 Jan 0001 00:00:00 +0000

Snapshot Debian Packages

Should you want to get started with the latest RSB snapshot and have all the required packages and dependencies installed in a convenient manner, simply run the following command:

sudo apt-get install rsb-snapshot r-properties r-rsbxml r-rsbjson

These packages are found in this repository: deb.openanalytics.eu. Follow this guide to add this repository to your system.

Prerequisites

RSB needs R, several R packages and also relies on RPooli for interacting with R.

If you haven’t installed R already, consider doing so using the r-base and r-rj packages from deb.openanalytics.eu

The R packages can be installed in two ways:

    install.packages(c("properties", "RSBXml", "RSBJson"), repos = "http://repos.openanalytics.eu", type = "source")

Install the Debian packages: r-properties, r-rsbxml, r-rjsonio and r-rsbjson from deb.openanalytics.eu

RPooli is bundled with RSB in its full distribution (which also contains a Tomcat web container).

If you chose not to use the full distribution of RSB, you’ll need to install RPooli yourself, which is easily done with the RPooli package from deb.openanalytics.eu

WAR-only Deployment

RSB should deploy fine on any Java web-container supporting Servlet 3.1 running with Java 11 or newer.

To ensure an RSB instance is running and can access its default RPooli pool, browse to or curl:

http://${RSB host}/rsb/api/rest/system/health/check

If all is good, RSB should respond: OK

Full Distribution Installation

Simply unzip the distribution Zip archive in your preferred location and ensure the files in the bin directory are executable.

Documentation: Administration

Mon, 01 Jan 0001 00:00:00 +0000

JMX

RMI

RSB exposes MBeans accessible with JMX locally and over RMI.

By default, the remote process URI is: service:jmx:rmi://127.0.0.1:9098/jndi/rmi://localhost:9099/myconnector.

HTTP

All the MBeans are also accessible over HTTP.

By default, the (browsable) HTTP URL is: http://localhost:8889.

Documentation: Scalability

Mon, 01 Jan 0001 00:00:00 +0000

Scaling Up

The workload of single RSB instance is expected to be minimal, as the bulk of the computation is performed by RPooli while RSB takes care of moving job requests and results between JMS, file system and e-mail destinations.

Increasing the processing throughput of a single RSB node can be achieved by increasing the size of the RPooli node pool and the number of RSB workers. Alternatively, if a single RPooli instance is reaching a limit that prevents adding more nodes, RSB can be configured to connect multiple RPooli instances in order to spread the workload across them. The selection of a particular pool could be achieved by assigning each independent RSB worker to a specific RPooli node or by using the configurable association between applications and RPooli pools (for example, to dispatch process intensive applications to a specific RPooli pool).

Scaling Out

Currently, RSB is architectured to run as a single stand-alone node. Should it become necessary to run several nodes in parallel, the following must be considered:

RSB polls email resources: running several instances concurrently consuming the same inboxes would create issues, as some jobs could potentially be retrieved several times (an email resource is not transactional). A possible mitigation is to configure each node differently so they don’t compete for the same inboxes.
RSB uses the local file system for handling multi-file jobs: if several RSB nodes get connected to a single JMS provider (instead of each of them using an embedded one), it is possible that a JMS message carries a pointer that references a file present in the file system of another node. A possible mitigation consists in carrying the full multi-file job payloads in JMS messages instead of File references.
Results for the REST API are stored in the local file system: in order to have several RSB nodes serve the same results either the file system where the results are stored should be shared across machines (like with an NFS mount) or an alternative implementation of [ResultStore] (https://rsb-doc.openanalytics.eu/current/apidocs/index.html?eu/openanalytics/rsb/data/ResultStore.html) that allows sharing over the network (for example a DB or Redis backed implementation).

Documentation: Build

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Sun/Oracle JDK 1.6
Maven 3

To produce rsb.war, run:

mvn clean install

Note that this war embeds javax dependencies and is designed to be deployed on web containers (like Tomcat or Jetty).

To produce a rsb.war that can be deployed on application servers, run:

mvn -P-javax-dependencies clean install

Packaged Distribution

To build a complete distribution consisting of Tomcat with both the RPooli and RSB web application deployed in it, run:

mvn -Pjavax-dependencies,tomcat-distribution clean package

Test Run

On Linux/OS X, you can easily start RSB for a quick ride with:

mvn jetty:run

to start RSB configured to use a locally running RPooli.

Similarly, if you have a locally running RPooli, it’s also possible to quickly try RSB with Tomcat by running:

mvn -Pjavax-dependencies,tomcat cargo:run

Press Control+C when you’ll be done with testing.

In both cases, RSB is accessible at this URL: http://localhost:8888/rsb.

Integration Tests

RPooli must be running locally before starting the integration tests.

If that is the case, then run:

mvn -Pjavax-dependencies,it -Dit.test=SuiteITCase clean verify

Documentation: HA

Mon, 01 Jan 0001 00:00:00 +0000

Reliability / HA

Because it uses JMS queues internally and consumes messages transactionally, RSB exhibits a reliable behavior as far as guaranteed job processing and result delivery (up to the HTTP, email or file system interaction) are concerned.

In order to roll-out a highly available deployment, several instances of RSB running in parallel must be deployed. The following must be considered:

RSB polls email resources: refer to this discussion for more information.
The REST API supports network level load balancing natively: nothing particular must be done to run RSB behind an HTTP load balancer. The job URIs generated by the REST API are built by using the host name of the incoming request, which guarantees that these URIs are respectful of any network-level indirection mechanism that could exist in front of RSB.
Each RSB node embeds a JMS provider: if an RSB node is down, all the jobs pending processing or the results pending delivery will not be accessible until the node is restarted. To be truly highly-available, the embedded JMS providers should be replaced with an external highly-available JMS provider to which all RSB nodes would connect. If this is pursued, keep in mind the following drawback (described here): RSB uses the local file system for handling multi-file jobs.

Documentation: Security

Mon, 01 Jan 0001 00:00:00 +0000

Secure Mode

It is possible to run RSB in secure mode where only authorized users can use certain applications. This is done by doing the following:

Activate security by uncommenting the blocks titled “Uncomment to enable security” in web.xml and security-beans.xml.
Configure a Spring Security authentication provider in security-beans.xml to retrieve users from LDAP, Database…
Configure RSB to associate applications with users and/or roles authorized to act as application users and admins, as shown in the next section.
Optionally configure users and/or roles authorized to act as RSB admins, as shown in the following section.

Application Level Security

The following demonstrate how to configure RSB application level security:


   "applicationSecurityConfiguration": {
      "secure_app_1": {
        "userPrincipals":["bob","dan"]
      },
      "secure_app_2": {
        "userRoles":["ROLE_SCIENTIST","ROLE_ANALYST"]
      },
      "secure_app_3": {
        "userPrincipals":["bob","dan"],
        "userRoles":["ROLE_SCIENTIST","ROLE_ANALYST"]
      },
      "secure_app_4": {
        "userPrincipals":["bob","dan"],
        "userRoles":["ROLE_SCIENTIST","ROLE_ANALYST"],
        "functionCallAllowed": true,
        "scriptSubmissionAllowed": true
      },
      "secure_app_5": {
        "userPrincipals":["bob","dan"],
        "userRoles":["ROLE_SCIENTIST","ROLE_ANALYST"],
        "adminPrincipals":["jack"],
        "adminRoles":["ROLE_ADMIN"]
      }
    }

Application admins, declared with adminPrincipals and adminRoles are de-facto users of the concerned application.

Notice how the functionCallAllowed and scriptSubmissionAllowed attributes are used to explicitly allow the users of secure_app_4 to execute jobs that can potentially impact the environment where R executes. This is disabled by default. Application admins are not affected by these flags.

RSB Admin Security

The following demonstrate how to configure the optional RSB admin roles/roles:

   "rsbSecurityConfiguration": {
        "adminPrincipals":["joe"],
        "adminRoles":["ROLE_ADMIN"]
    }

RSB admins are de-facto users and admins of all the applications.

Application Aware Catalog

In a secure environment, it can be desirable to partition the catalog per application name. This prevents one application or one application admin to read or write files in another application.

This option, which is off by default, is activated with the following configuration setting:

    "applicationAwareCatalog" : true

JMX RMI

Once RSB is running in “Secure Mode” (see above), it’s possible to secure the JMX RMI connector. It’s done by editing jmx-beans.xml and uncommenting the blocks marked with “Uncomment to enable security”.

Only the RSB admins (defined by users/roles as explained in “RSB Admin Security”) will be allowed to connect to the JMX RMI interface.

If the default JMX ports have be left unchanged, the remote process URI to use to securely connect is:

    service:jmx:rmi://localhost:9098/jndi/rmi://localhost:9099/jmxRMIConnector

JMX Web UI (MX4J)

It’s possible to secure the JMX Web UI with HTTP Basic Auth by configuring a dedicated username / password pair in the RSB configuration:

    "jmxConfiguration": {
       "stubPort": 1098,
       "registryPort": 1099,
       "httpPort": 8889,
       "httpAuthenticationUsername": "jmxui_username",
       "httpAuthenticationPassword": "U2FsdGVkX1+nOXQF+qvSVHeDdDG+jUyJxPzm6NW4NkI="
    }

The password is encrypted with DES3 using the username as the encryption password. For example, with jmxui_username / jmxui_pass as the username / password pair, one can encrypt the password with:

    $ echo -n "jmxui_pass" | openssl enc -a -e -salt -des3 -pass pass:jmxui_username
    U2FsdGVkX1+nOXQF+qvSVHeDdDG+jUyJxPzm6NW4NkI=

It is recommended to enable SSL encryption for the JMX Web UI by using a frontal web-server, like Nginx.

Note that this security option is available even if RSB is not running in “Secure Mode” (as described above).

Documentation: Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

From version 5.x to 6.x

The JobStatisticsHandler has been modified in a non backwards compatible fashion. The change is trivial: the storeJobStatistics method now takes a Job interface as a parameter from which the application name and job ID, previously passed as distinct arguments, can be retrieved.

Documentation: Building / Testing

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

JDK 1.8
Maven 3

To produce the war file, run:

mvn clean install

Note that this war embeds javax dependencies and is designed to be deployed on web containers (like Tomcat or Jetty).

To produce a rsb.war that can be deployed on application servers, run:

mvn -P-javax-dependencies clean install

Packaged Distribution

To build a complete distribution consisting of Tomcat with both the RPooli and RSB web application deployed in it, run:

mvn -Pjavax-dependencies,tomcat-distribution clean package

Test Run

On Linux/OS X, you can easily start RSB for a quick ride with:

mvn jetty:run

to start RSB configured to use a locally running RPooli.

Similarly, if you have a locally running RPooli, it’s also possible to quickly try RSB with Tomcat by running:

mvn -Pjavax-dependencies,tomcat cargo:run

Press Control+C when you’ll be done with testing.

In both cases, RSB is accessible at this URL: http://localhost:8888/rsb.

Integration Tests

RPooli must be running locally before starting the integration tests.

If that is the case, then run:

mvn -Pjavax-dependencies,it -Dit.test=SuiteITCase clean verify

Documentation: /admin/catalog

Mon, 01 Jan 0001 00:00:00 +0000

/admin/catalog

Mount Point: /rest/admin/catalog

The following operations are supported in this resource:

GET

Parameters

name	description	type	default
X-RSB-Application-Name	(no documentation provided)	header

Output Payload

element	produceable content type(s)	description
catalog (XML)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

Documentation: /admin/catalog/{catalogName}/{fileName}

Mon, 01 Jan 0001 00:00:00 +0000

/admin/catalog/{catalogName}/{fileName}

Mount Point: /rest/admin/catalog/{catalogName}/{fileName}

The following operations are supported in this resource:

GET

Parameters

name	description	type
catalogName	(no documentation provided)	path
fileName	(no documentation provided)	path
X-RSB-Application-Name	(no documentation provided)	header

Output Payload

element	produceable content type(s)	description
(custom)	`application/xml` `/`	(no documentation provided)

PUT

Parameters

name	description	type
catalogName	(no documentation provided)	path
fileName	(no documentation provided)	path
X-RSB-Application-Name	(no documentation provided)	header

Input Payload

element	consumeable content type(s)	description
(custom)	`text/plain`	(no documentation provided)

Output Payload

element	produceable content type(s)	description
(custom)	`application/xml` `/`	(no documentation provided)

Documentation: /admin/system/configuration

Mon, 01 Jan 0001 00:00:00 +0000

/admin/system/configuration

Mount Point: /rest/admin/system/configuration

The following operations are supported on this resource:

GET

Output Payload

element	produceable content type(s)	description
(custom)	`application/json`	(no documentation provided)

PUT

Input Payload

element	consumeable content type(s)	description
(custom)	`application/json`	(no documentation provided)

Output Payload

element	produceable content type(s)	description
(custom)	`/` `application/xml`	(no documentation provided)

Documentation: /admin/system/r_packages

Mon, 01 Jan 0001 00:00:00 +0000

/admin/system/r_packages

Mount Point: /rest/admin/system/r_packages

The following operations are supported on this resource:

POST

POST

Parameters

name	description	type
rServiPoolUri	(no documentation provided)	query
sha1hexsum	(no documentation provided)	query
packageName	(no documentation provided)	query

Input Payload

element	consumeable content type(s)	description
(custom)	`application/gzip`	(no documentation provided)

Documentation: /admin/system/restart

Mon, 01 Jan 0001 00:00:00 +0000

/admin/system/restart

Mount Point: /rest/admin/system/restart

The following operations are supported on this resource:

POST

POST

Output Payload

element	produceable content type(s)	description
(custom)	`/` `application/xml`	(no documentation provided)

Documentation: /admin/system/rservi_pools

Mon, 01 Jan 0001 00:00:00 +0000

/admin/system/rservi_pools

Mount Point: /rest/admin/system/rservi_pools

The following operations are supported on this resource:

GET

Output Payload

element	produceable content type(s)	description
rServiPools (XML)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

Documentation: /data

Mon, 01 Jan 0001 00:00:00 +0000

/data

Mount Point: /rest/data

The following operations are supported on this resource:

GET

Output Payload

element	produceable content type(s)	description
directory (XML)	`application/xml` `application/vnd.rsb+json` `application/vnd.rsb+xml` `application/json`	(no documentation provided)

Documentation: /data/{rootId}{b64extension}

Mon, 01 Jan 0001 00:00:00 +0000

/data/{rootId}{b64extension}

Mount Point: /rest/data/{rootId}{b64extension }

The following operations are supported on this resource:

GET

Parameters

name	description	type	default
rootId	(no documentation provided)	path
b64extension	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
directory (XML)	`application/xml` `application/vnd.rsb+json` `application/vnd.rsb+xml` `application/json`	(no documentation provided)

Documentation: /jobs

Mon, 01 Jan 0001 00:00:00 +0000

/jobs

Mount Point: /rest/jobs

The following operations are supported on this resource:

POST (JSON payload)
POST (XML payload)
POST (ZIP payload)
POST (multi-part form)

POST (JSON payload)

Handles a function call job with a XML payload.

Input Payload

element	consumeable content type(s)	description
(custom)	`application/json`	Argument passed to the function called on RServi.

Output Payload

element	produceable content type(s)	description
(custom)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

POST (XML payload)

Handles a function call job with a XML payload.

Input Payload

element	consumeable content type(s)	description
(custom)	`application/xml`	Argument passed to the function called on RServi.

Output Payload

element	produceable content type(s)	description
(custom)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

POST (ZIP payload)

Handles a function call job with a ZIP payload.

Input Payload

element	consumeable content type(s)	description
(custom)	`application/x-zip-compressed` `application/x-zip` `application/zip`	Input stream of the ZIP data.

Output Payload

element	produceable content type(s)	description
(custom)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

POST (multi-part form)

Handles a multi-part form job upload.

Input Payload

element	consumeable content type(s)	description
(custom)	`multipart/form-data`	(no documentation provided)

Output Payload

element	produceable content type(s)	description
(custom)	`application/xml` `application/json`	(no documentation provided)

Documentation: /process

Mon, 01 Jan 0001 00:00:00 +0000

/process

Mount Point: /rest/process

The following operations are supported on this resource:

POST (JSON payload)
POST (XML payload)

POST (JSON payload)

Handles a function call job with a JSON payload.

Input Payload

element	consumeable content type(s)	description
(custom)	`application/json`	Argument passed to the function called on RServi.

Output Payload

element	produceable content type(s)	description
(custom)	`application/json`	(no documentation provided)

POST (XML payload)

Handles a function call job with a XML payload.

Input Payload

element	consumeable content type(s)	description
(custom)	`application/xml`	Argument passed to the function called on RServi.

Output Payload

element	produceable content type(s)	description
(custom)	`application/xml`	(no documentation provided)

Documentation: /result/{applicationName}/{resourceName}

Mon, 01 Jan 0001 00:00:00 +0000

/result/{applicationName}/{resourceName}

Mount Point: /rest/result/{applicationName}/{resourceName}

The following operations are supported on this resource:

GET
HEAD

GET

Serves a single result file.

Parameters

name	description	type	default
applicationName	(no documentation provided)	path
resourceName	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
(custom)	`application/octet-stream`	(no documentation provided)

HEAD

Provides HTTP meta-information only for a single result file.

Parameters

name	description	type	default
applicationName	(no documentation provided)	path
resourceName	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
(custom)	`application/octet-stream`	(no documentation provided)

Documentation: /results/{applicationName}

Mon, 01 Jan 0001 00:00:00 +0000

/results/{applicationName}

Mount Point: /rest/results/{applicationName}

The following operations are supported on this resource:

GET

Parameters

name	description	type	default
applicationName	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
results (XML)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

Documentation: /results/{applicationName}/{jobId}

Mon, 01 Jan 0001 00:00:00 +0000

/results/{applicationName}/{jobId}

Mount Point: /rest/results/{applicationName}/{jobId}

The following operations are supported on this resource:

GET
DELETE

GET

Parameters

name	description	type	default
applicationName	(no documentation provided)	path
jobId	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
result (XML)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

DELETE

Parameters

name	description	type	default
applicationName	(no documentation provided)	path
jobId	(no documentation provided)	path

Output Payload

element	produceable content type(s)	description
(custom)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)

Documentation: /system/health/check

Mon, 01 Jan 0001 00:00:00 +0000

/system/health/check

Mount Point: /rest/system/health/check

The following operations are supported on this resource:

GET

Output Payload

element	produceable content type(s)	description
(custom)	`text/plain`	(no documentation provided)

Documentation: /system/info

Mon, 01 Jan 0001 00:00:00 +0000

/system/info

Mount Point: /rest/system/info

The following operations are supported on this resource:

GET

element	produceable content type(s)	description
nodeInformation (XML)	`application/vnd.rsb+json` `application/vnd.rsb+xml`	(no documentation provided)