Wingman Infrastructure

The wingman system is composed of a virtual machine hosting 3 main services, with the addition of GitHub.

A Jenkins service for scheduling
An OSGI application, pentaho-wingman, which handles the orchestration and analysis of the builds.
A docker container, which hosts the builds to run in a sandbox.

Jenkins

There is a basic Jenkins instance doing the orchestration and management. It has 3 jobs.

wingman-trigger: This is the scheduled process.
wingman: This is the main job which calls the Orchestrator service.
wingman-update: This is a management process. It fetches, compiles and updates the pentaho-wingman service (see below).

wingman-trigger

The trigger is a scheduled job. It runs each X minutes and asks GitHub for a list of the current pull requests for all projects in a list of organizations known to GitHub.

It was designed as part of task ENGOPS-1586. Physically, it consists of a python script. The script can be seen here. It runs periodically on the Jenkins server and takes the following parameters.

Parameter	Description
-o	A comma separated list of the names of the organizations to scan for a list of projects. Usually something like: "pentaho,webdetails".
-t	The API token from GitHub.
-b	A comma separated list of the names of the branches to scan. All pull requests for these branches will be considered.
-ao	A comma separated list of organization names to which a user must belong to in order for his pull request to be built.

A typical invocation of this script should look something like this:

python PullRequestRecon.py -o webdetails,pentaho -t 0123456789abcdef -b master,6.1,6.0 -ao webdetails,pentaho

wingman

This is the main job on Jenkins which communicates with the Wingman service. It is a pretty simple job which gets invoked by the wingman-trigger job. It can also be launched manually.

This job takes a few parameters.

Parameter	Description
WINGMAN_ORG	The name of the organization which owns the pull request we want built.
WINGMAN_REPO	The name of the repository which owns the pull request we want built.
WINGMAN_PR_NUMBER	The number of the pull request to build.

With these parameters, Jenkins simply invokes the REST endpoint of the wingman service by adding the parameters supplied. A typical invocation should look something like this:

#!/bin/bash

JSON=`echo "{\"SourceRetriever\": { \"SourceControlType\": \"github\", \"Organization\": \"${WINGMAN_ORG}\", \"Repository\": \"${WINGMAN_REPO}\", \"PullRequest\": \"${WINGMAN_PR_NUMBER}\"}, \"StatusUpdater\": {\"WingmanUrl\": \"$BUILD_URL\"}}"`
curl -N -s -S -H "Content-Type: application/json"\
     -X POST -d "$JSON"\
     http://localhost:8181/cxf/orchestrator/orchestrate

wingman-updater

This job updates the Wingman service. The update job performs a few tasks.

Build wingman from the latest code
Stop the wingman service
Replace the current build with the new one
Start the wingman service

This is done through the deploy.sh script present within the project.

pentaho-wingman

This system is composed of an OSGI application. The code lives here. It contains numerous modules which fall in one of those categories.

Source control modules
Builders
Analyzers
A single Orchestrator module

To get the code built and tested, please refer to the project's README file.

Once deployed, the service is invoked through a URL.

http://localhost:8181/cxf/orchestrator/orchestrate

This service takes a few parameters. To invoke it, do a POST operation on its endpoint and pass the following JSON message. Notice the presence of a few variables. Replace these tokens by their proper values.

{  
   "SourceRetriever":{  
      "SourceControlType":"github",
      "Organization":"${WINGMAN_ORG}",
      "Repository":"${WINGMAN_REPO}",
      "PullRequest":"${WINGMAN_PR_NUMBER}"
   },
   "StatusUpdater":{  
      "WingmanUrl":"http://wingman.pentaho.com:8080/job/wingman/106/"
   }
}