Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Community Build Framework

Author and Maintainer: Pedro Alves
Contributors: Daniel Einspanjer
Latest Version: 2.0
Latest Update: July 4th, 2008

Table of contents

Concepts

Overview

Community Build Framework (CBF) is an ant build.xml file script and alternate way to setup and deploy Pentaho based applications. Focused on a multi-project/ multi-environment scenario, here are the main characteristics:

  • Ability to switch from totally different projects on the fly
  • Supports multiple environments (eg: development, production, worker1, worker2...)
  • No changes to the original files that could get overwritten in a upgrade
  • Supports multiple platform versions
  • Support for jboss and tomcat (the latest only for platform >= 1.7)
  • Simplifies version upgrades
  • Debug-friendly
  • VCS-friendly
  • Support all kinds of different customization in different projects, from using a different databases to different security types.
  • Supports patches to the source code for fine-grain customization
  • Supports deploy to local/remote machines
  • OS independent

Description

Project pentaho already has a good dev_build.xml ant script, but it wasn't enough for my needs, since I needed to change the svn sources in order to configure a solution. So I built my own structure, and its like this:

Directory layout and code checkout

MyProjectDir
|-- build.xml
|-- pentaho
|-- pentaho-data
|-- pentaho-preconfiguredinstall
|-- pentaho-solutions
|-- project-client
|   |-- patches
|   |   |-- pentaho
|   |   `-- target-preconfiguredinstall
|   `-- solution
|-- target-build
|-- target-preconfiguredinstall
`-- target-solutions

pentaho/* directories are cheched out from svn, and for me are read-only. Feel free to use any svn client you want; I usually work in linux, or on a windows machine with cygwin, so I use the original client. I checked out the sources with:

$ svn co svn://source.pentaho.org/svnroot/legacy/pentaho/branches/1.7

$ svn co http://source.pentaho.org/svnroot/legacy/pentaho-solutions/branches/1.7 pentaho-solutions

$ svn co svn://source.pentaho.org/svnroot/legacy/pentaho-data/branches/1.7 pentaho-data

# next line for jboss PCI
$ svn co svn://source.pentaho.org/svnroot/legacy/pentaho-preconfiguredinstall/branches/1.7 pentaho-preconfiguredinstall

# next line for tomcat
$ svn co  svn://source.pentaho.org/svnroot/legacy/pentaho-tomcat/branches/1.7 pentaho-tomcat

If you are under a proxy you can use http:// instead of svn:// . I'm currently using version 1.7 (this one, or later, is required for tomcat support)

target-/* directories are the ones generated by the build script. I don't use target-solutions at all (honestly, I'm not sure what this are for, cause they seem a copy of pentaho-solutions. Guess the point is to pick target* and copy it, w/o any other dependencies, to the desired machine). target-build is the temporary build directory. Since we may need to patch the pentaho source, CBF will make an entire copy of pentaho/ to target-build and patch the later, leaving the original source intact.

This leaves us with the "project-client" directory. As mentioned before, the idea is not write anything under pentaho* directories, so all my changes go to this directory. Here is an example structure in more detail:

project-client/
|-- config
|   |-- build.properties
|   `-- build-pedro.properties
|-- patches
|   |-- target-build
|   |   `--
|   `-- target-preconfiguredinstall
|       `-- server
|           `-- default
|               |-- conf
|               |   |-- jboss-minimal.xml
|               |   `-- jboss-service.xml
|               |-- deploy
|               |   |-- PentahoHibernate-ds.xml
|               |   |-- client-ds.xml
|               |   |-- pentaho.war
|               |   |   `-- WEB-INF
|               |   |       |-- applicationContext-acegi-security-jdbc.xml
|               |   |       |-- applicationContext-pentaho-security-jdbc.xml
|               |   |       |-- classes
|               |   |       |   `-- hibernate.cfg.xml
|               |   |       |-- jboss-web.xml
|               |   |       `-- web.xml
|               |   |-- quartz-ds.xml
|               |   |-- sampledata-ds.xml
|               `-- lib
|                   `-- postgresql-8.2-505.jdbc3.jar
`-- solution
    |-- Portal.properties
    |-- Portal.url
    |-- Portal_pt.properties
    |-- admin
    .... etc

All this is created manually according to the specific project needs.

Note to CBF version 1 users

Note the absence of a patched override.properties file; All the needed information is in the properties files under the config directory

The idea is very simple: All changes that would normally go to pentaho/* are placed under "patches" directory. CBF ant script just picks them and copies the files to the top level directory. It's not recommended to patch anything under pentaho/*; sources changes are patched in to target-build/* and all other changes are made by patching the final directory, target-preconfiguredinstal (for jboss) or target-preconfiguredinstall-tomcat (for tomcat)

Tip: Making the solution ready to both jboss and tomcat

One very simple way to make the project running in both jboss and tomcat with no relevant maintenance is to make a symbolic from target-preconfiguredinstall/server/default/deploy/pentaho.war to target-preconfiguredinstall-tomcat/webapps/pentaho and the same to pentaho-style.war

For the example scenario, those were the files I had to change. I had to change some jboss's config files just because I had to change a port that was unavailable; added my datasource (client-ds.xml) and changed the jboss-web.xml and web.xml to register it; changed all other ds's cause I'm using postgresql instead of hsql; added the jdbc lib; changed the security system from memory to jdbc.

This is scalable. Just change files there and they will be copied to the destination target. CBF does not allow (yet) to remove files from target to destination, but I also don't think that's a relevant issue.

Tip: Upgrading Jars

Sometimes there's the need to upgrade some Jar version. A very simple way to do so is to patch 2 files: myJar-oldversion.jar and myJar-newVersion.jar where the first one is an empty zip file that will overwrite the original one

The solution is under project-client/solution, and our build.properties points to this dir. For now, I just copied pentaho-solutions to here, and will start from there by adding new solutions and playing a bit.

Tip: Maintaining system files

Tip: Since project-client/solution/system is a generally untouched directory, it's easier to maintain the svn info there. Just copying the directory from pentaho-solutions/system or making a

svn co svn://source.pentaho.org/svnroot/pentaho-solutions/branches/1.7/system

will allow svn updates later. The same approach can be used for admin, samples and test directories.

Configuration files

The main config file is in config/build.properties; specific environment configurations can then be set in config/build-env.properties . Here's an example file:

build.properties
#####################################
## GENERIC PROPERTIES
######################################

javac.path = NOT SET

# Solution path. Required for compile time
solution.path = NOT SET

# Solution path required for runtime. Defaults to the above value but can be different if we're
# deploying to another place. Should be used in WEB-INF/web.xml in the solution-path parameter
solution.deploy.path = /home/pedro/tex/pentaho/project-client/solution/

#accepted values: tomcat or jboss
server.name = tomcat

# Java options for the run command
java.opts = -Xmx512m -XX:MaxPermSize=512m -Xrunjdwp:transport=dt_socket,address=8765,server=y,suspend=n

#####################################
## PROJECT DIRECTORIES - The defaults usually work fine
######################################
pentaho.dir = pentaho/
pentaho.build.dir = target-build/

#####################################
## DEPLOY OPTIONS - USE AT YOUR OWN RISK
######################################

deploy.mode = NONE
deploy.args = -Cavz
deploy.dest.server = UNSET
deploy.dest.solution = UNSET


#####################################
## DYNAMIC TOKENS TO BE USED AS FILTERS
#####################################

DB_LOCATION = 127.0.0.1
DB_USER = pedro
DB_PASSWORD = bar
BASE_URL = http://127.0.0.1:8080/pentaho/

I use this to set the default options and put my specific information in my environment configuration

build-pedro.properties
javac.path = /usr/lib/java/bin/javac

solution.path = /home/pedro/tex/pentaho/project-client/solution/

solution.deploy.path = /home/pedro/tex/pentaho/project-client/solution/

pentaho.dir = pentaho-1.7/

deploy.mode = rsync
deploy.args = -avze 'ssh -p 8022' --delete
deploy.dest.server = machine1@10.10.10.2:/opt/pentaho/pentaho-server
deploy.dest.solution = machine1@127.0.0.1:/pentaho/pentaho/pentaho-solution

Tokens

To extend customization to a greater extent, I've implemented dynamic filter tokens for changeable attributes, such as database locations and passwords. Whenever CDF finds a text file with a string of the form @@DB_LOCATION@@ that token will be replaced at compile time.

CBF execution and available targets

Building and executing

After all this environment is correctly set up, CBF is ready to be executed. ant executable needs to be on the path and the properties project and optionally env need to be passed. If no env property is passed only build.properties will be used.

Here are the available targets. The description speaks for itself:

CDF targets
$ ant -Dproject=client -Denv=pedro -p
Buildfile: build.xml
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo] ------        CBF - Community Build Framework          -------
     [echo] ------                 Version: 2.0.1                  -------
     [echo] ------ Author: Pedro Alves (pedro.alves@webdetails.pt) -------
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo]
     [echo]                          SETTINGS:
     [echo]
     [echo]    Project:         metrics
     [echo]    Environment:     build-pedro.properties
     [echo]    Config File:     project-client/config/build-pedro.properties
     [echo]    Server:          tomcat
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
Pentaho BI Platform build helper..
Main targets:

 all              Compiles and builds the entire project
 clean            Cleans the solution
 copy-finish      Copy target files
 copy-init        Copy project files
 deploy-all       Deploys both the solution and the server to the specified destinations
 deploy-server    Deploys the entire server to the specified destination
 deploy-solution  Deploys the solution to the specified destination
 deploy-war       Deploys the pentaho.war and pentaho-style.war to the specified destination
 dev-setup        Compiles and builds the entire project
 dist-clean       Deletes the target preconfigured install dir
 run              Runs the solution
Default target: all

To clean, compile and run pentaho all that's needed is, for example:

Simple execution
$ ant -Dproject=client -Denv=pedro dist-clean all run

Just by passing different properties we can immediately build and run a totally different project, in a different platform version or different application server. With the right properties and tokens, we can instantly deploy a solution in a remote server.

Deploy

Despite the property deploy.mode, only the rsync method is currently supported. It's by far the most appropriate way to handle my kind of requirements, but a lot more modes can be implemented, like copy, scp or even ftp transfer.

Debug

With the appropriate JAVA_OPTS parameters, all that's needed to debug the platform is do a socket attach using your favorite IDE.

Download

Old versions

FAQ

I don't have any questions. Do you? (smile)

Changelog

Version

Changes

2.0

Major changes; Introduced different configurations and deployments; dynamic tokens; deploys

1.0

Initial CBF

  • No labels