Wingman Infrastructure

Wingman Infrastructure

The wingman system is composed of a virtual machine hosting 3 main services, with the addition of GitHub.

• A Jenkins service for scheduling
• An OSGI application, pentaho-wingman, which handles the orchestration and analysis of the builds.
• A docker container, which hosts the builds to run in a sandbox.

Jenkins

There is a basic Jenkins instance doing the orchestration and management. It has 3 jobs. 

• wingman-trigger: This is the scheduled process. 
• wingman: This is the main job which call the Orchestrator service.
• wingman-update: This is a management process. It fetches, compiles and updates the pentaho-wingman service (see below).

wingman-trigger

The trigger is a scheduled job. It runs each X minutes and asks GitHub for a list of the current pull requests for all projects in a list of organizations known to GitHub. 

It was designed as part of task ENGOPS-1586. Physically, it consists of a python script. The script can be seen here. It runs periodically on the Jenkins server and takes the following parameters.

A typical invocation of this script should look something like this:

python PullRequestRecon.py -o webdetails,pentaho -t ./credentials.cred -b master,6.1,6.0 -ao webdetails,pentaho

wingman

This is the main job on Jenkins which communicates with the Wingman  service. It is a pretty simple job which gets invoked by the wingman-trigger job. It can also be launched manually. 

This job takes a few parameters.

With these parameters, Jenkins simply invokes the REST endpoint of the wingman service by adding the parameters supplied. A typical invocation should look something like this:

#!/bin/bash

JSON=`echo "{\"SourceRetriever\": { \"SourceControlType\": \"github\", \"Organization\": \"${WINGMAN_ORG}\", \"Repository\": \"${WINGMAN_REPO}\", \"PullRequest\": \"${WINGMAN_PR_NUMBER}\"}, \"StatusUpdater\": {\"WingmanUrl\": \"$BUILD_URL\"}}"`
curl -N -s -S -H "Content-Type: application/json"\
     -X POST -d "$JSON"\
     http://localhost:8181/cxf/orchestrator/orchestrate

wingman-updater

This job updates the Wingman service. There are a few prerequisites before running this job.

• Disable the Jenkins jobs names wingman and wingman-trigger.
• Make sure that no wingman jobs are currently running. If they are, wait until they complete.
• You can now run the wingman-update job

The update job performs a few tasks.

• Stop the wingman service.
• Wait for the service to stop.
•  Pull the latest code from master
• Run the build
• Start the wingman service.

Below is the code which we currently use to accomplish these tasks. (Line breaks added for readability)

~/pentaho-wingman/assembly/build/assembly-0.0.1/bin/stop;
sleep 5;
cd ~/pentaho-wingman/
&& git pull origin master
&& cd ~/pentaho-wingman/assembly/
&& gradle clean build
&& cd ~/pentaho-wingman/assembly/build/
&& unzip ~/pentaho-wingman/assembly/build/distributions/assembly-0.0.1.zip
&& cd ~/pentaho-wingman/assembly/build/assembly-0.0.1/ && bin/start"

pentaho-wingman

This system is composed of an OSGI application. The code lives here. It contains numerous modules which fall in one of those categories.

• Source control modules
• Builders
• Analyzers
• A single Orchestrator module

To get the code built and tested, please refer to the project's README file.

Once deployed, the service is invoked through a URL.

http://localhost:8181/cxf/orchestrator/orchestrate

This service takes a few parameters. To invoke it, do a POST operation on its endpoint and pass the following JSON message. Notice the presence of a few variables. Replace these tokens by their proper values.

{  
   "SourceRetriever":{  
      "SourceControlType":"github",
      "Organization":"${WINGMAN_ORG}",
      "Repository":"${WINGMAN_REPO}",
      "PullRequest":"${WINGMAN_PR_NUMBER}"
   },
   "StatusUpdater":{  
      "WingmanUrl":"http://wingman.pentaho.com:8080/job/wingman/106/"
   }
}