Pentaho Architect's Bootcamp Training Now Available

Last week, I had the pleasure of offering our very first 5-day session of the Pentaho Architect's Bootcamp training, and overall, it was a big success!

There comes a point in many advanced deployments of the Pentaho BI Suite where some feature or requirement pushes the boundaries of the out-of-the-box product capabilities. Since the beginning of Pentaho time, we've marched to the beat of "make it possible first, then make it pretty/easy", and it's this scenario where our approach pays big dividends to our customers/community/users. Because the platform/server/tools were built for extensibility, there are numerous places where you can roll up your sleeves and leverage a simple API to implement a customization that suits your specific requirements.

The Pentaho Architect's Bootcamp is geared for developers, partners, customers, consultants that are ready to roll up their sleeves and understand the complex problems that are surfacing in large scale BI implementations, and how to extend the Pentaho suite of products to answer far more questions and customizations than the boundaries of the out-of-the-box product.

Here's a sampling of some of the questions that are answered during Architect Bootcamp training:

• How do I integrate my own custom visualizations (maps, charts, gauges, etc) into Pentaho?
• How do I accommodate multiple companies/groups/organizations' data in my solutions, while maintaining each companies/groups/organizations' personal point of view of the data?
  
• How do I dynamically drive row level security in Pentaho Analysis and Pentaho Metadata?
  
• How do I integrate Pentaho solution content into my own application?
  
• How do I customize security across the Pentaho platform and pillars?
  
• How do I create integrated solutions using ETL, reporting, analysis and metadata to deliver my customer's specific solutions?
• How do I plug my custom content into the Pentaho BI Server? How do I then integrate my custom data/functionality with the Pentaho pillars (ETL, reporting, analysis, metadata, etc)?
  

I really believe this course is a game-changer for Pentaho users and solution developers. I teach the course, so don't take just my word for it. Here's some feedback we received from the first course offering:

"We traveled from India to Florida for this course, and we are extremely glad that we did. The information is very valuable."

"The presentation was excellent, [I e]specially liked the interactive nature of the class."

"All of the lessons uncovered new boundaries; all topics had more priority!"

"YEEEEAAAA.. just built my first Pentaho BI Server plugin... Pentaho Architect's Bootcamp RULES!!! #Pentaho @Pentaho"

"Day 4 Pentaho Architect's Bootcamp brought so many excitements, I loved it. even though my brain feels like it's going to explode!! #Pentaho"

Barcelona Pentaho Community Meetup 2009 Pics

We fly home from Barcelona tomorrow, a lovely vacation and another fantastic Pentaho community gathering.

I'll be chatting with many of you online in the near future, and hopefully will see everyone again next year - Vienna is it? Sweet!

As promised, here's my pics :)

kindest regards,
G

Hola from Barcelona, Community Gathering 2009

We're close to restarting the sessions for the afternoon, just dropping in to update my fellow Pentaho colleagues on the gathering:)

We got a bit of a late start this morning, mostly because the community started the meetup last night, and the socializing lasted into the wee hours for the group. A great time was had by all :) The morning's speakers had great content, covering a variety of topics, from Mozilla statistics presentation with CDF to the latest revision of PAT, the community analysis tool. Roland and Jos are here, our celebrated Pentaho Solutions authors, signing books and presenting the basics of developing custom CDF components.

And in traditional European holiday style, we're late in getting the afternoon sessions started, most of our group (around 40 attendees with community and Pentaho included)is still in earnest roadmap discussions at the cervesseria :)

Next post, some pics for posterity.

buenos tarde!
Gretchen

Pentaho Community, Together in Barcelona

Doug and I arrived in Barcelona this morning, early enough to see some of this beautiful city before the Pentaho Community Meetup this weekend.

This is the second annual community meetup, an event that is organized and planned completely by Pentaho community for Pentaho community. No fluffy corporate speak, just a full weekend of Pentaho community developers and users showing off their stuff, talking through their current projects and solutions, and having a few beers and some fun. Many thanks to Tom Barber for planning and sponsoring much of this year's event.

We look forward to seeing familiar faces, and new community as well:0) See you all very soon!

Thanks Roland and Jos: Pentaho Solutions IN PRINT!

I received my pre-ordered copy yesterday of Pentaho Solutions: Business Intelligence and Data Warehousing with Pentaho and MySQL. A huge congratulations and thank you to Roland Bouman and Jos van Dongen, two long time Pentaho community members who wrote the book.

I can' t tell you how excited I am to see this book! For many years, developers and project managers that I've worked with have felt that a book like this one is the missing link to helping customers achieve success with their warehouse and business intelligence strategies. Most books on business intelligence are either too abstract or offer guidance only on select pillars (for example, only reporting solutions), which leave the reader with unfulfilled requirements and no direction for filling in the gaps.

With Pentaho Solutions, the reader gets a concrete explanation and best-of-breed Pentaho implementation of ETL, reporting, analysis, dashboarding and data mining solutions; 5 core pillars and their concepts that contribute to a healthy, whole, successful BI strategy and implementation.

You can pre-order your copy at Amazon.com :)

Roland and Jos, the team has already sunk their teeth in, and they love what they're reading. Well, the picture says it all :)

Development and Debugging with GWT and Javascript

Java code is the bread and butter of what I do, but as most Java developers know, there is a plethora of good frameworks and technologies that surround Java and provide a means to build very powerful and extendable software.

Lately, I've popped back into GWT land and have been dealing with lots of Javascript, both original and generated. I originally introduced myself to GWT building a small volunteer information submittal form for Brevard Rescue Mission. This tiny application only scratched the surface of what GWT could do in its earliest stages. The magic that the Pentaho development team have performed with Pentaho Dashboarding is a new level of web-goodness, fully capitalizing on the power of GWT. I've been dabbling in the chart rendering layers of dashboards, and have learned some simple, effective means of making life a bit easier when dealing with debugging and developing Javascript and GWT generated Javascript. I hope you find these tips useful, and it's certainly nice to have them aggregated in one place! I have to give credit for this info to Nick Baker and Mike D'Amour, two of my colleagues at Pentaho. Use this blog post as a starting point for googling the original sources for more details on each tip:)

Helpful tips for Developing and Debugging with GWT & Javascript in General

Limit the user.agent Property

GWT compiling is resource intensive due to the number of compilations that happen for the browsers supported. At times, you will run out heap space or other resources before the compile can finish (this usually manifests itself as a StackOverflowError).

The following entries in your *.gwt.xml file can help by only compiling for the single browser you may be testing on:

<inherits name="com.google.gwt.user.UserAgent"/> 
  <set-property name="user.agent" value="ie6" />

Valid values for the user.agent property are: ie6,gecko,gecko1_8,safari,opera

Limit the gwt.compile.localWorkers

You can also scale back the number of threads to use for running parallel compilation. While this may hurt performance, you will be able to finish the compilation without running out of resources. This property, gwt.compile.localWorkers, can be added to the compile option in your ant script.

Bump the GWT version from 1.6.4 to 1.7.0

GWT 1.7.0 seems to have resolved many of the compilation resource issues with GWT.

GWT Pretty Print Compile

By default, we obfuscate our GWT compiled Javascript. To debug readable GWT compiled Javascript, compile with pretty print turned on. This property, gwt-style, can be added to the compile option in your ant script. Valid values include OBF, PRETTY, and DETAILED.

IE Javascript Debugging Help

If you need to debug Javascript in IE, it is highly recommended that you get IE8. You can install IE8 for the duration of your testing, then uninstall it when you no longer need it, as it has conflicts with GWT. IE8 has a new set of features called Developer Tools that make debugging Javascript very easy.

Helpful In Line Javascript Alerts

You can use the following line of code to send alert windows whereever you like in your Javascript code:

$wnd.alert("Hello World")

A good example from Nick:

This is literally saving me hours. By adding a line to the the end of the
printStackTrace() function you can alert out the stacktraces that normally do nothing when compiled.

Open up the gwt script file (xxxxxxxxxxxxxxxxxxxxxxxxx.cache.html) for your particular browser. I find it by seeing what's loaded in firebug.

Search for "function $printStackTrace"

Add a new line right before the function returns:

$wnd.alert(msg.impl.string);

It should now look like this.

function $printStackTrace(this$static){
var causeMessage, currentCause, msg;
msg = $StringBuffer(new StringBuffer());
currentCause = this$static;
while (currentCause) {
causeMessage = currentCause.getMessage();
if (currentCause != this$static) {
msg.impl.string += 'Caused by: ';
}
$append_4(msg, currentCause.getClass$().typeName);
msg.impl.string += ': ';
msg.impl.string += causeMessage == null?'(No exception detail)':causeMessage;
msg.impl.string += '\n';
currentCause = currentCause.cause;
}
$wnd.alert(msg.impl.string);
}

In Code Breakpoints

Rather than sifting through the script debugger window trying ot figure out where to put a breakpoint, you can use the following line of code to embed a breakpoint:

debugger;

Feel free to comment and send your favorite tricks for working with GWT and Javascript.

Pentaho Analysis Tool Integrated as a Pentaho Plugin

I had the chance this week to play around with the still-under-construction Pentaho plugin architecture in the Citrus code line. The new architecture is just what BI developers have been waiting for: totally flexible with several new ways to integrate with the server, simple to use and allows for building nicely decoupled extensions.

With Aaron Phillip's help, I got my head around the new features in less than a day, and had my first plugin written shortly after: The Pentaho Analysis Tool (PAT) plugin. Before I get into the details of the PAT plugin, let's first talk about the new tools and capabilities in the Pentaho BI server's plugin layer.

The plugin architecture consists of several different fun ways you can hook into the Pentaho BI Server, without having to modify server code or disturb the platform deployment. All avenues for leveraging the plugin architecture expect that the necessary files and code will be found in the solutions folders. The layer currently has the following capabilities:

1. Customization of the menu system of the "classic" and more recent PUC (Pentaho User Console) user interfaces
2. Customization of various page contents (overlays)
3. New types of content to be added to the solution repository and operated upon in the user console
4. New Java classes that generate UI pages to be dynamically added to the server
5. (new in 3.0) Add your own BI Component to the platform without having to modify system files and paths

You can get more details about these features and how they work by reading the documentation here. Aaron also created a sample plugin that demonstrates each of these features in a simple plugin mockup, that also is a great template to use for new plugin creation.

So that's exactly what I did. Here's a screenshot of the results of my plugin:



Using the EchoPlugin sample as a guide, I created a new content type (.xpav, for Pentaho Analysis View) which is the first notion of a view definition file for PAT. When you "open" this new content type in PUC, it initializes and launches PAT, which is a separately deployed web application. This is accomplished by creating a new content generator in the plugin that delegates the generation to the PAT webapp. It takes a bit to put it all together: you need a bleeding edge Citrus BI Server download, the latest PAT code and the plugin project. If you are interested in seeing it in action, read the integration instructions here.

I only took advantage of a couple of the new plugin layer's capabilities in my first plugin. I'm looking forward to playing with the new web services as well as the component that allows my plain old Javabean to look like a BI component automagically. I can foresee great extensions coming fast for the Pentaho BI Server with this new architecture!

I've listed some good references for those who are ready to take a look at plugins:

Here is the documentation:
http://wiki.pentaho.com/display/ServerDoc2x/BI+Platform+Plugins+in+V2

and here is the Plugin Depot, where you can show others the cool new extensions you've built:
http://wiki.pentaho.com/display/ServerDoc2x/Plugin+Depot

and if you have questions, comments or problems, or think you may have spotted a bug, chat with some of the developers about it here:
http://forums.pentaho.org/forumdisplay.php?f=73

kindest regards,
Gretchen

Maven: The Definitive Guide

My one true nerdy tendency: I like writing technical documentation. It's ironic then (or a bit of a hypocrisy) that I loathe reading it. Chalk it up to my lack of passion for technology. I am a passionate problem solver; technology is a sometimes rewarding, sometimes frustrating means to help effectively get the problem solving job done.

Recently, I began reading Maven: The Definitive Guide while getting a pedicure at my local salon (laugh it up guys, I can guess where most of you do your leisure reading). I strongly recommend that any developer approaching Maven for the first (or tenth) time give chapters 3 through 8 a read. This guide is what you hope most technical guides or books would be, but then usually quite early on, they disappoint.

The guide starts with a quick, understandable introduction to Maven terminology and concepts, via a short step by step example. As I was reading this from a "make this worth my while" perspective, I had specific use case questions that immediately popped into my head ... and then, I was pleasantly surprised to find the answers in the next few paragraphs.

For example, the guide mentions early on that "support for transitive dependencies is one of Maven's most powerful features". To that my questions were "What about conflicts in dependency hierarchies?" and "What about compile time dependencies that I don't want to package?". The rest of the chapter addresses exactly those questions with explanations on dependency exclusions and scoping. Finally. A book that thinks like I do :)

A quick summary of the rest of the meat of the guide: chapters 4 through 8 build on the core concepts introduced in 3, with bite size chunks of additional functional explanation in each chapter. The material is presented as a hands on example, building in feature complexity little by little. Chapter 4 shows you how to add new dependencies to your project; 5 introduces simple web application features; chapters 6 and 7 cover multi-module project and enterprise project features. This presentation worked for me on a few different levels:

1. The graduated approach to the introduction of new materials made a large amount of Maven terminology, concepts and finally usage documentation digestible.
2. The authors take great care in describing WHY they arrange and refactor the projects as they do, in a very modular fashion. This approach, in practice, lends itself not only to Maven's default conventions, but also to best practices for software project layout. Note that this introduced complexity to the examples that wasn't necessary to explain the features at hand. But the authors bit that bullet in order to present a good and useful way of developing a project.
   
3. The example projects described in the guide are immediately relevant for me. I write Java code. I use Spring. I use Hibernate. This, of course, will not be the case for every reader, but it was a nice bonus for me.
   

So after all of my monologue thoughts, I leave you with a few tids:

• Read the guide (at least Part 1), then decide how problematic you perceive Maven to be. I know my perception changed dramatically.
• Note that it is a bit outdated, deprecated goals and such (the guide is updated for Maven 2.0.10, I downloaded Maven 2.1). This really didn't distract me at all.
• The chapters I reference are only the tip of the iceberg. Part 2 of the guide includes another 200 reference pages that I have yet to use. I'll let you know how that goes for me:)
  

Every Developer Needs a Roadshow

It's been a few days since returning from the Pentaho Partner Summit. When I get the chance to attend company events, conferences or seminars (the opportunities are rare), I try to sum for myself the benefits of having traveled, gathered and given my attention to the occasion at hand.

In the case of the Partner Summit, I thought of several key revelations that came about as a result of the trip. The one that stands at the forefront of my mind: every developer needs a roadshow.

Not as a roadie in a product tour, or as booth Bob at a trade show, but as an interested attendee at an event that showcases whatever you have been working on as a developer. Mind you, this is not a NEW revelation for me; I've had the privilege of representing Hyperion Analyzer at Java One as a developer on that project, and talked to many talented Oracle folks about Pentaho at ODTUG, as well as many other roadshows of my own. I always have come back saying the same thing to my peers - "You guys have to hear what they are saying! You have to feel the excitement!". ( Yes, their was a maddening amount of energy and excitement around the Pentaho Partner Summit!)

The benefits to sending developers out to events that have nothing to do with development and everything to do with the project or product are many. The first benefit that I got excited about in Menlo Park was that I was able to hear how our partners and customers were using Pentaho. I'm committed to focusing on what questions BI users are asking as I re-enter the BI space as a developer, and this was a prime audience. During networking opportunities, partners told stories about customers with big data on Vertica, MySQL, and InfoBright; in intranets, in DMZs, and of course, now in the Cloud. Pentaho partners OpenBI had an attentive and boisterous audience as they discussed their Cloud implementation with client Nutricia.

I also really enjoyed having face time with the consumers of the fruits of my previous efforts. I have been away for some time, but I think some parts of the Pentaho projects are still riddled with my signature:) It's OK that many, but not all comments were glowing; that's the point, right? I feel like I understand just a little bit better some of our users' pain points. And that puts me in a better place to alleviate some of that pain. (No worries, Brian and Nick and Domingo ... Will and Thomas will get right on native crosstabs!!!)

The Partner Summit event gave me the opportunity to lift my head up from the details of our projects and see the field from our partners' perspective. Can I get a lot of the same information surfing the web or hitting the forums? Sure. The perspective is unique though, to spending time with the people who are providing business intelligence solutions in the market. That, I believe, only comes on the road.

Pentaho Partner Summit: Menlo Park, CA

It's a fortunate coincidence that I'm in California at the same time the Pentaho Partner Summit is going on. The event is packed, with partners and interests attending from more than 15 countries. The speakers yesterday were really quality, talking about everything from business intelligence in the Cloud to commercial open source business strategy.

So I've been able to spend lots of quality time connecting with old friends and colleagues, and have met some new, really talented folks. More details on the event later, but for now, check out some pics of the event here.

Cheers, -G

Because I've Got Issues ...

I have to hand it to the guys over at Atlassian, JIRA is a pretty killer app (although I know now that Love is the REAL Killer App:) ).

I've worked with JIRA for, well, a really long time. I've always worked in companies where you needed to wear many hats, and I'm one of those developers that doesn't get snobby when I'm asked to step outside of my comfy Java home and help out the IT folks. So it's usually me that gets those prize winning projects like migrating forums, internationalizing wikis, or looking for new software to streamline our internal processes. I've spun JIRA around the dance floor several times, XSLT'ing crazy aggregate reports from XML backup formats, writing plugins to support externalizing JIRA data, customizing schemes, changing workflows. Every time, the same epiphany gets me - JIRA just works, exactly how you would think it should.

While some who are not so in the know might think, "Gretchen, you simpleton, it's a series of instructions to a processor, of course that's how it works". But those of us who bend software over and around daily know that few apps are actually written with quality, exceptional exception handling and in an intuitive manner that doesn't require years of higher learning and great tolerance for pain to adopt. (This is a very familiar concept particularly for those who use a certain unreasonable operating system).

This time, we need to move Mondrian's tracker issues from their original home on Sourceforge over to JIRA, which is our tool of choice for managing work and issues at Pentaho. With an assist from my other favorite killer app, Kettle, it has been a dreamy couple of days putting together the pieces to get Mondrian's issues to their new home. OK, maybe not dreamy, but certainly pain free.

Kudos, my Atlassian friends. You Aussies got it going on.

Where have you been??

When I say "you", of course I mean me! I silently fell off the radar about 7-8 months ago, and am now finally re-emerging. Well, let me tell you what I've been doing in my "off" time:

• First, let me introduce you to Jack David, my new baby boy. Doug and I were blessed with this little guy August 15th, 2008. He is the primary cause for my hiatus. I have been loving every minute of being home with my peanuts (Anthony, 13, Bella, 3 and baby Jack)! Alas, the lure of olap cubes, ETL and bug squashing safaris was just too compelling to resist, and it's time to return to the business world.

• 
• I also joined the board of directors as secretary of the Brevard Rescue Mission, a faith based ministry that provides whole-life transition resources for near-homeless moms and their kids. My dear friend Stacia Glavas is the founder, and I have been privileged to be able to handle her communications, marketing and administrative needs in between diaper changes.
  
• I have been dabbling in a bit of graphic design and found that while I have no natural talent, the Adobe suite allows me to appear semi-talented in creating fun and compelling designs. I have since designed the web site for rescue mission mentioned above, as well as the logo for my daughter's new preschool, several business cards and stationery for friends, and my latest, most daring adventure: skinning a mySpace page for my photographer and friend, Yvette Gioia! Where WILL my curiosities take me?????
• I also have managed to talk a friend of mine into letting me "borrow" his home renovation crew to renovate the entire exterior of our 25 year old home. That's right, I've decided that I also have some sort of qualifications as a contractor. Or possibly just a penchant for frustration and pain, we'll soon see!

As you can see, when Doug and I decided that it would be a good idea for me "take some time off" to adjust and organize our growing family, well, I may have misread the "time off" instructions :) I have thoroughly enjoyed my very full, engaged foray into being a stay-at-home mom. I actually just this month officially earned my soccer mom title, as Bella joined the local soccer team. I never knew there was a position for grass-pickers in soccer, but sure enough, my daughter has the exalted title!

While I will miss much of the freedom that comes from being my own manager, I miss my career more. So expect to hear from me soon, as I mull over my next career adventure! And of course, immerse myself back into the Pentaho Nation!

Kindest regards!
Gretchie

Pentaho Meetup in Mainz, Germany

Pentaho is hosting its first community meetup in Mainz, Germany in a few short weeks. I like the format we've chosen, as it combines informal sessions with lots of food, drink and some touring. And everyone is encouraged to bring their ideas and latest projects for a show and tell.

Find out details and register here. It's sure to be informative and provide an opportunity to make some great Pentaho contacts.

Creating a Minimum Width Constraint on an SWT SashForm

My friend Matt encouraged me to blog a quick snippet of code that many SWT coders might find useful.

If you've used the SWT SashForm, you know it's a nice widget in that it saves you a bit of work that you would do using a traditional Sash. One of the things I expected to get for "free" was the ability to limit how far the sash could expand or collapse a single control, or eseentially constrain the sash from completely hiding any number of it's child controls. To my surprise, I found that this is not a feature of the SashForm.

I have figured out how to create that behavior, by adding a selectionListener to the sash on the sashform. I have to give most of the credit to Duong Nguyen, who posted a partial solution to my problem here.

Without further adue, here's the code:

// Set a minimum width on the sash so that the
// controls on the left are always visible.

// First, find the sash child on the sashform...
Control[] comps = sashform.getChildren();
for (Control comp : comps){

if (comp instanceof Sash){

// My limit is derived from the size of a
// toolbar; yours can be any size you wish ...

int limit = 10;
for (ToolItem item : view.getParent().getItems()){
limit += item.getWidth();
}

final int SASH_LIMIT = limit;
final Sash sash = (Sash)comp;

sash.addSelectionListener (new SelectionAdapter () {
public void widgetSelected (SelectionEvent event) {
Rectangle rect = sash.getParent().getClientArea();
event.x = Math.min (Math.max (event.x, SASH_LIMIT), rect.width - SASH_LIMIT);
if (event.detail != SWT.DRAG) {
sash.setBounds (event.x, event.y, event.width, event.height);
sashform.layout();
}
}
});
}
}

Hope this is useful!
kindest regards,
G

Kettle, XUL, SWT and a Bit of Theory...

I've been working on a very fun and challenging proof of concept / usability project that involves Pentaho Data Integration (Kettle), XUL, SWT, Swing and a very cool, slightly controversial theory: with a lot of work frontloaded, we should be able to port user interfaces to any UI technology, and expect consistent behavior AND look and feel without much, if any, additional code .

The project I'm in the middle of encompasses three different but equally compelling goals:

1. To move Pentaho forward in providing common layers across our core pillars (reporting , analysis, data mining, ETL and dashboards).
2. To provide a proof of concept for the Pentaho XUL Framework, an architecture built to help us support common UIs across all of our applications and tools .
3. To provide a common way of describing and managing the information needed to connect to a database.

So, we began by taking Pentaho Data Integration's connection dialog ( a powerhouse of detail and metadata concerning over 30 different database connection types), applied some usability design to it, and are in the middle of rewriting the dialog using XUL and some new design concepts.

This required a lot of up front work on the part of our lead Pentaho XUL developer, Nick Baker. Nick has plumbed an architecture for Pentaho that makes it possible for us to define a user interface in XML (specifically, using the Mozilla XUL specification), and use a common set of wrapper libraries to render the UI in SWT or Swing, without rewriting any of the UI, and reusing a great amount of the business logic code. There are also plans to begin a XUL-to-Web 2.0 library as well, which will cover most of the UI technologies that we use for the Pentaho projects.

Interested? Download the source and demo, take it for a ride, inspect the code and let me know what you think. Here are the home pages for both the Pentaho XUL Framework project, and the Common Database Connection Dialog proof of concept.

Common Database Connection Dialog
Pentaho XUL Framework Project

Go ahead and post here, on the wiki, or shoot me an email. I look forward to your reactions and suggestions:)

kind regards,
Gretch

PS. All you Kettle users & developers, here's a chance to weigh in on some needed UI changes for Kettle's database UI... don't miss the opportunity!

Pentaho on Oracle's App Server (OC4J)

Wow! It has really been 4 months since my last post?? Moving over to development has cut into the time I had for blogging, documenting, communicating, you name it! We are coding like crazy.

Well, I'm back because I am heading out to ODTUG Kaleidoscope next week, and in preparation for the show, I decided to setup Pentaho on Oracle's Java Edition App Server, which is OC4J, which is based on the Orion app server. I was pleased that I managed the migration in less than a day, and I wanted to share the steps with all those folks who are too impatient to wait for this to get into our J2EE deployment distribution :)

Mind you, it takes a bit of tweaking, but it is certainly very do-able, and all server features are stable (minus the portal stuff, I didn't get a chance to address moving the portal over). Here is the repro of where I started, what I tweaked and what I came out with. Enjoy!

Where I Started

I started out by downloading a Pentaho J2EE deployment distribution from the Pentaho downloads site. The version I used for this exercise was pentaho_j2ee_deployments-1.5.4.716-0.zip. This distribution is found on our downloads page under Pentaho Open BI Suite Web Archive (.war). I know, the name implies a .war distribution, but trust me, its the deployments zip file.

Unpack this distribution to a work directory of your choosing. This distribution has an Ant build script that lets you build several different .war files and .ear files configured for different app servers and RDBMSs. Next, I'll detail the tweaks I had to make to get the Orion build target working, which is sufficient to buid an .ear file appropriate for Oracle's OC4J.

I also downloaded the sample solutions distribution and the sample data (Hypersonic) distribution, so I would have stuff to test against. You can get both of these distributions from the Pentaho downloads site as well.

What I Tweaked

About two years ago, we had started to incorporate build scripts for the Orion application server in anticipation of great community demand for this build. However, with the multitude of projects we have taken on, and a surprising lack of banging from the community, we never found time nor priority to finish that build, until now.

I really want to be able to demo our stuff on Oracle at the user conference, so I took on the task of cleaning up and repairing the Orion build. NOTE that you will be able to get these fixes in a near future build, as soon as we get our M4 release out the door and I can check this stuff in.

The Build

For the brave and impatient, here's what it needs out of the gate:

1. We'll call the root of your work directory [pentaho_j2ee_deployments]. In this directory, you will find a build.xml file. Open that file in your favorite text editor.
   
2. Find the ant target named "build-orion2.0.5-ear". Delete that target entirely. Replace the target with the following XML:
   
   <!-- ===================================================================
   target: build-orion2.0.5-ear
   =================================================================== -->
   <target name="build-orion2.0.5-ear" depends="zip-pentaho-style-war">
   
   <antcall target="war-pentaho-tomcat">
   <param name="rdbms" value="hsqldb" />
   </antcall>
   <mkdir dir="${build.ears.dir}/orion" />
   
   <ear destfile="${build.ears.dir}/orion/pentaho.ear"
   appxml="${ear.dir}/application.xml">
   <manifest>
   <attribute name="Implementation-Title"
   value="${impl.title}" />
   <attribute name="Implementation-Version"
   value="${impl.version}" />
   <attribute name="Implementation-Vendor"
   value="${impl.vendor}" />
   </manifest>
   <fileset dir="${build.wars.dir}">
   <include name="pentaho-style.war" />
   <include name="sw-style.war" />
   </fileset>
   <fileset dir="${build.wars.dir}/tomcat/hsqldb"
   includes="pentaho.war" />
   <metainf dir="pentaho-res/orion" includes="*.xml" />
   </ear>
   
   </target>
   
3. Save and close the build.xml file.
4. In the same directory, open the override.properties file. Add the following line to this file:
   ear.dir=pentaho-res/ear
5. Save and close the override.properties file.

Those few steps fix up the build files so that you can build a pentaho.ear for OC4J. You of course will have to have Ant in your system path, and a JDK available of at least 1.4.2. I will assume if you are climbing this mountain, those are easy steps you already know how to set up:)

Don't run the build yet! There are several configuration file tweaks that have to be added in order for this app to be configured properly.

The web.xml File

We need to make a few minor changes in the Pentaho web.xml.

First, the OC4J container's default application is running on port 8888. Pentaho's default port is set to 8080. So I changed Pentaho's default port to be 8888, since that seemed to be the easiest road. NOTE that you want to add your changes to two web.xml files, since it is duplicated in the deployer distribution.

You need to modify BOTH

- [pentaho_j2ee_deployments]/pentaho-precompiled-jsps/pentaho.war/WEB-INF/web.xml

AND

- [pentaho_j2ee_deployments]pentaho-webapp/WEB-INF/web.xml !!

1. Open both web.xml files.
2. Find the base-url param-name, with the value http://localhost:8080/pentaho/. Change the value to http://localhost:8888/pentaho/.

We also need to make sure the server can find the Pentaho solutions directory. If you haven't yet, unpack the solutions distribution that you downloaded to a work directory of your choice.

1. Find the param-name pentaho-solutions, and replace the value with the absolute path to the pentaho-solutions directory that you just unpacked. The value should look something like d:\work\pentaho-solutions.
2. Save and close both web.xml files.

The application.xml File

The application.xml file lives in the [pentaho_j2ee_deployments]/pentaho-res/ear directory.

1. Open the application.xml file.
2. Delete all modules under the comment <!-- additional web apps --> , as well as the web-uri module for the sw-style.war above the comment. The only modules your application.xml should have left is the pentaho.war and the pentaho-style.war.
3. Save and close the application.xml file.

Note that the sw-style.war provides the styles for the Steel Wheels samples in the solutions, but I did this quick and dirty, so I left out as much extras as possible. You may want to include the sw-styles.war, and see if it works. I left it out.

The orion-web.xml File

The orion-web.xml file is non-existent, you need to create one.

1. Add an orion-web.xml file to [pentaho_j2ee_deployments]/pentaho-webapp/WEB-INF.
2. Add the following XML to the file:
   <orion-web-app>
   <resource-ref-mapping location="jdbc/HibernateDS" name="jdbc/Hibernate"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/SampleData"/>
   <resource-ref-mapping location="jdbc/QuartzDS" name="jdbc/Quartz"/>
   <resource-ref-mapping location="jdbc/SharkDS" name="jdbc/Shark"/>
   <resource-ref-mapping location="jdbc/SampleDataAdminDS" name="jdbc/SampleDataAdmin"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/datasource1"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/datasource2"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/datasource3"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/datasource4"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/solution1"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/solution2"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/solution3"/>
   <resource-ref-mapping location="jdbc/SampleDataDS" name="jdbc/solution4"/>
   </orion-web-app>
3. Save and close the orion-web.xml.

Build the pentaho.ear File

You've got everything you need now to build the pentaho.ear file. Go to a command prompt, navigate to your [pentaho_j2ee_deployments] directory, and execute:

ant build-orion2.0.5-ear

OC4J Server Configuration Changes

Now that you have your .ear file, you will obviously want to deploy it through OC4J Server Console. First, shut down your OC4J instance, because we have a few mods that we need to make to the server configuration.

TopLink Conflicts

Now, I know this is not nice, but since we have no use for Toplink with Pentaho, and the antlr.jar conflicts with our hibernate3 library, you will need to delete the Toplink directory under [OC4J_Home]. This causes no harm as long as you're not using Toplink for some other reason. If so, then maybe you can investigate further with Oracle how to avoid the library conflicts.

This will cause the java_sso webapp in the default module deployed in the server to fail. I simply undeployed the app.

Out of Memory Errors

Pentaho is a big engine, and as such uses a bit of memory. I found that I was running out of memry early and often. So I modified the JVM_ARGS parameter in the [OC4J_Home]/bin/oc4j.cmd file to include:

set JVMARGS=%OC4J_JVM_ARGS% -Xms128m -Xmx512m -XX:MaxPermSize=128m

Add the hsqldb.jar to the [OC4J_Home/lib] Directory

Since we will be testing against the Hypersonic sample data, we need to add the JDBC driver for Hypersonic to the [OC4J_Home/lib] directory. You can find the hsqldb.jar in [pentaho_j2ee_deployments]/pentaho-third-party.

Note that when you deploy the .ear file, you MUST set an additional entry on the classloader's classpath for this jar!

Mod the data-sources.xml File

And finally, we need to add our datasources to the [OC4J_Home]/j2ee/home/config/data-sources.xml file. Add the following xml to the file, before the <datasources/> end tag:

<data-source
class="com.evermind.sql.DriverManagerDataSource"
name="Hibernate"
location="jdbc/HibernateDS"
xa-location="jdbc/xa/HibernateDS"
ejb-location="jdbc/Hibernate"
connection-driver="org.hsqldb.jdbcDriver"
username="hibuser"
password="password"
url="jdbc:hsqldb:hsql://localhost/hibernate"
inactivity-timeout="30"
/>
<data-source
class="com.evermind.sql.DriverManagerDataSource"
name="SampleData"
location="jdbc/SampleDataDS"
xa-location="jdbc/xa/SampleDataDS"
ejb-location="jdbc/SampleData"
connection-driver="org.hsqldb.jdbcDriver"
username="pentaho_user"
password="password"
url="jdbc:hsqldb:hsql://localhost/sampledata"
inactivity-timeout="30"
/>
<data-source
class="com.evermind.sql.DriverManagerDataSource"
name="SampleDataAdmin"
location="jdbc/SampleDataAdminDS"
xa-location="jdbc/xa/SampleDataAdminDS"
ejb-location="jdbc/SampleDataAdmin"
connection-driver="org.hsqldb.jdbcDriver"
username="pentaho_admin"
password="password"
url="jdbc:hsqldb:hsql://localhost/sampledata"
inactivity-timeout="30"
/>
<data-source
class="com.evermind.sql.DriverManagerDataSource"
name="Quartz"
location="jdbc/QuartzDS"
xa-location="jdbc/xa/QuartzDS"
ejb-location="jdbc/Quartz"
connection-driver="org.hsqldb.jdbcDriver"
username="pentaho_user"
password="password"
url="jdbc:hsqldb:hsql://localhost/quartz"
inactivity-timeout="30"
/>
<data-source
class="com.evermind.sql.DriverManagerDataSource"
name="Shark"
location="jdbc/SharkDS"
xa-location="jdbc/xa/SharkDS"
ejb-location="jdbc/Shark"
connection-driver="org.hsqldb.jdbcDriver"
username="sa"
password=""
url="jdbc:hsqldb:hsql://localhost/shark"
inactivity-timeout="30"
/>

Problem in the Solutions' XSLTs

I don't get this at all, but Oracle has some sort of built-in XSLT processor that complains when your XSLT tries to reference a Java class as the value for a namespace. So what you need to do is go through the [pentaho-solutions]/system/custom/xsl directory, and check every xsl for this occurrence (I would say at least 20 of them have it) and make the following mod to those XSLTs:

1. Open the xslt file.
2. Find the java class reference in the namespace declaration. They are always at the top of the file, and look simialar to xmlns:msg="org.pentaho.messages.Messages" . Note that xmlns:msg isn't the only possibility, just an example for you to follow.
3. Prepend the java classname value with http://www.oracle.com/XSL/Transform/java/. So the fixed example would be xmlns:msg="http://www.oracle.com/XSL/Transform/java/org.pentaho.messages.Messages" .
4. Repeat the steps above for every xslt file in that directory that has the namespace occurence.

There is one syntax problem in one of the xslt's as well. JBoss and Xalan don't seem to care about it, but OC4J will error out, so we need to fix that.

1. Open the file [pentaho-solutions]/system/custom/xsl/files-list.xml.
2. Search for the following lines of XML, and delete them. Note they are not right next to each other.
   <xsl:param name="target" />
   <xsl:with-param name="target" select="$target"/>
   

Deploying the Pentaho .ear File

Finally!! You can deploy your .ear file now. Start your OC4J server back up, and use the Server Console to deploy. Here are a few notes on the deployment options:

1. You do not need to set up a security provider in the deployer wizard. Pentaho uses the standard J2EE security via ACEGI, and no custmo extensions to the configuration are necessary.
2. Make sure that you map a path on the classloader to the hsqldb.jar that you moved into OC4J's lib directory!
3. You may see some log4j exceptions on start-up, but as long as you see a statement toward the end of the console log that says "Pentaho BI Platform ready", then you should be ready to go!

Navigate in your browser to http://localhost:8888/pentaho/Home, and test it out!

What I've Got - Pentaho on OC4J!

So, now I'm stoked because I can talk intelligently to Oracle tools users about Pentaho on Oracle. I think I might even dive in and use Kettle to move our sample data over to Oracle Express.

Very soon all this should be doc'ed and available through our SVN repository... hopefully any early comers who use this tip will let meknow how it goes for them:)

And please, if you will be at the conference in Daytona next week, stop by the Pentaho booth and say hey! Can't wait to meet you all!

Gretchie

Wrap up on the Pentaho Implementation Workshop

All in all, I had great expectations for the 3 day implementation workshop, and I wasn't disappointed!

The last session on Thursday, Dashboards and AJAX, was crammed full of great technical information. The most important info that I can give you all is that the current JSP based dashboards are going away, in exchange for dashboards built upon Pentaho AJAX components, currently under development. Of course, we all were very excited to hear about the new AJAX component architecture, and happy to hear that we can get to the code. The ETA for GA delivery is somewhere in the 3rd or 4th quarter of this year (with milestone builds available earlier most certainly), but always stomping on that hairy edge, I can't wait to dive into the code and contribute to implementation (in my spare time, ha ha! ).

James Dixon, Chief Geek for Pentaho, also went into a bit of the history of Pentaho Dashboards, which was really helpful in gaining perspective on why Dashboards require so much coding today. The philosophy and design goals for Dashboards (really for the platform, in general) is to remain delivery agnostic - meaning we want the platform output to be delivered via the client's choice of technology, not our own. So if you are a JSF shop, .NET shop, or Java applet guru, it won't matter to us, since we deliver the content from the platform in XML. You can take it from there, and transform that XML any way you wish. Well, that design goal is tough to stick to when you are implementing a Dashboard architecture, since Dashboards are heavy on UI, usually containing reports, charts, dials, gauges and numerous other widgets, in some portal type fashion specific to the user's point of view. So we went about component-izing all of the above mentioned widgets, and used a simple JSP (or not so simple JBoss Portal) to demonstrate what COULD be done. The response from our community has been to make Dashboards easier to build, and fortunately, AJAX has conme along (or been there, depending on how you look at AJAX) so that we can deliver that ease of implementation.

I haven't had a chance to say much about the group that attended the class with me. I was tickled to finally meet some folks that I have been chatting with via email, some for more than a year now! Nice, intelligent, talented and truly passionate about BI - I could spend alot more time with these folks, we share so many traits and interests (I know, I think very highly of me ;))! The attendees came from Canada, the Netherlands, Germany, France and the US, which speaks for the demand that our training generates, as well as the global presence that Pentaho has earned in a short 2.5 years. I can't believe sometimes that I am a part of something this big, and this bold! On a day to day basis, it feels like we are just a bunch of guys doing what we have done best for a long time - building BI. But when you gather your community, partners and teammates in a room like we did last week, it sure feels a whole lot bigger, a lot more significant. And so, I can someday explain to my daughter why my job makes me proud :)

Workshop, Day 3

I know, I know, what happened to Day 2? Well, between busy sessions, picking up the kids, and teaching my night class at Brevard Community College, blogging fell to the way side. For my three avid fans, I sincerely apologize, but by now you are used to it :)

So Day 2 of the workshop, in all honesty was a bit of a fire fighting exercise. Since we are working with the very latest code for the platform (and I mean VERY latest), we ran into a couple of problems during the Subscriptions session that prevented us from seeing the results of the subscribed bursting examples that we set up. But the content was good and subscriptions in the stable builds is very very powerful. We have the ability with the platform to relieve the administration and information overload that occurs with the typical scheduled reporting process. Subscriptions also prove to be flexible and easy to use, which makes it a nice tool for consumers of Pentaho reports, analysis, ETL and processes.

The Day 2 afternoon session was all about advanced deployments of the platform, as well as customizing deployments for each user's environment. Brian Hagan walked through the complex details of manually deploying web application through JBoss and Tomcat, focusing on the touch points that are required when you have your own app server installation already in place. Overall a good session that could be helpful to anyone that struggles with J2EE deployments today.

Day 3 started out with Bill Seyler, a stellar Pentaho engineer, presenting the life cycle management features within Pentaho. For anyone who is not familiar with the term and what it means in Pentaho, life cycle management is versioning solution content for the platform. Bill covered the architecture of how Pentaho interacts with version control systems, which seems to be a very clean and simple implementation. The beauty of life cycle management in Pentaho is that due its simple interface, any version control system can be used, as well as using multiple systems for one Pentaho deployment.

Next up was Anthony DeShazor, our engineering wrangler, talking about scalability and clustering. Much of this session covered the general topology and infrastructure issues that prevent almost any application from scaling. The point I took away is we can control what the Pentaho application does, but how you get at your data and how you deliver it out to consumers can bottleneck any good app. It was great to participate in tis discussion, since many in the room are experienced in the field and had much to contribute. Anthony then took us through the JBoss Clustering presentation that James Dixon presented at JBossWorld late last year. It was a simple architectural discussion covering JBoss Clustering, ending with some pretty impressive benchmarks that proved Pentaho's ability to scale. The most interesting news that came out at the end of this session is that Pentaho has started to build a BI benchmarking bundle, based on the Transaction Processing Performance Counsel processes, and plans to release it to the open source community for benchmark responses. Feel free to email James (communityconnection@pentaho.org and it will get forwarded) if you are interested in participating in that effort!

Our last session after lunch is Dashboarding and AJAX, a session that all the trainees, including myself are looking forward to. I'll fill you in tonight on how that session goes and how this all wraps up.

Implementation Workshop: Metadata

"My users think I'm a God." , Matt Harbert, DivX

I had to start with that quote, I was so pleased to hear it during discussions over lunch this afternoon during the workshop. I was talking with a couple of classmates about Kettle, the Pentaho Data Integration tool, reveling in stories of leaping tall buildings with Kettle as my booster pack. From across the room, Matt tossed the aforementioned words of glory, and I thought "well, if that doesn't just sum it all up". Kettle really is that good, and you'll only know if you dive into it, because, of course, I am a Pentaho-an, making me slightly biased ;)

These were the topics of discussion that opened the Metadata module of the workshop. Metadata is another project being architected and driven by Matt Casters, founder of the Kettle project. Jake Cornelius led the module, and did a nice job of showing us the Pentaho metadata Editor, a handy tool that assists in building Pentaho metadata models. The Metadata Editor is not quite 2 months old, and from what we were shown, is proving to hold lots of potential. The core functionality is there, and the user interface is intuitive, once you learn the new jargon. Amidst a short array of funky behavior and a few bugs, the Metadata Editor's power shined through with it's ability to model not only mappings to physical tables, but also extended formulas, formatting and style properties in a hierarchical fashion (termed "concepts"), and internationalization functions.

This is the end of day one, time to pop over and visit the dev guys on my way home. I have a renewed sense of excitement today, partly because this training is turning out to be even better than I had expected (and my expectations were pretty high), and partly because I'm training up for a brand new seat on the Pentaho ride :)

Implementation Workshop: Security Simplified

Well, we just finished the security module of the workshop, and I have to say, sans the network issues, I am really impressed. Mat Lowry, a Pentaho engineer who focuses on security during his day job, put together the content for the module today. Mat took a pretty complex set of topics (LDAP, Acegi, CAS and J2EE Container Security) and delivered just enough content to understand easily what Pentaho Security is made up of, and what Pentaho adds to the standard technologies available to you in a J2EE environment.

It seems we have done a very nice job of separating the wrangling of authentication and authorization from the functionality of the BI platform. I plan to follow up this workshop with a deeper dive into Acegi, as Mat has gotten me really excited about what it and the Spring framework can do. I'm taking away the relief that Acegi can handle a good 80% to 90% of my web resource security problems, without me having to write more code. I like it, I like it.

My thoughts on the hands on lab is it really made me think and I was pleasantly surprised to find that I understood the concepts Mat covered, and could apply them in the 60 minute lab exercise that was given to us. This was not your typical training class exercise that with loads of screenshots and step-by-step instructions, you could achieve one simple implementation of security. This was more like "A train leaves Tampa at 400 miles an hour at the same time a train leaves Daytona at 200 miles an hour, when will they meet" type of exercise. Now, I think I've mentioned before that I'm a very bright, but pretty simple person, and frankly this type of exercise reminds me how little focus I have. Once we stopped chatting, and I could read the lab carefully, I had no trouble implementing my own switch over from memory based security to LDAP based security within the Pentaho platform.

I have to admit, I feel a tad bit smarter than I used to :)