HAPI FHIR Example Server Using Docker on AWS

Confession: I am not an abstract thinker. In order to learn something new, I have to have it in my hands. So when I was asked to evaluate and understand the FHIR standard, my first course of action was to lift the hood on one of the more mature reference FHIR servers available, the HAPI FHIR JPA Server.

You must realize that following this tutorial will not get you any closer to understanding the FHIR standard … but it WILL give you your own environment where you can inspect every aspect of FHIR, from issuing REST API calls to examining the data model for storage that HAPI chose to use. This setup provides me total transparency from input to process to output, all with technology I am comfortable with. Your goals may be different, I just hope the guide provides value whatever they may be.

By default, the HAPI JPA example server starts up seamlessly using an embedded Apache Derby database and Jetty server. I’m more comfortable with Tomcat and PostgreSQL, and from the documentation, HAPI seemed … well … happy, to let me switch. 

While it was a pleasantly straight forward exercise to set this server up the way I wanted it, there were a few gotchas. This is the guide I wish I had found when I googled “HAPI FHIR JPA server Tomcat PostgreSQL”. 🙂 

I am big Docker and Docker-Compose fan (and with good reason). So I chose to create a docker-compose file to start a Tomcat server and a PostgreSQL database and deploy the HAPI FHIR JPA Server example application to the application server. 

The last bit (that became oh-so-simple thanks to my upfront efforts with Docker) is launching my server on an AWS EC2 instance. This is a normally just a nice environment to work in, but for the HAPI FHIR JPA Server to be of any use to me, it was also a necessity.  You see the sample data that I would have liked to have loaded from the HL7 website does not load properly to the HAPI server for dSTU3. Fortunately the folks at Crucible have some synthetic patient records they will load for you, all you need is a public server URL (hence the AWS move). 

Read on below for the step-by-step details. You can download the Docker artifacts from this article. And all you FHIR gurus out there,  please feel free to let me know where I could have made my life easier!

Create Dockerfiles for Tomcat and PostgrSQL. 

Lukas Pradel has already done a pretty fantastic job of writing a nice tutorial for dockerizing a Tomcat & PostgreSQL setup. 

There is very little we need to change in the Dockerfiles from that posting.  One important change will be to reference the HAPI FHIR JPA Server example .war. We haven't built this yet, but we will soon. I also changed the username, password and database name to match what I wanted for my application. You will see in the next steps where you will tell the HAPI application what these values are, so keep them handy. 

The Application Server (Tomcat) Dockerfile

---

FROM tomcat:8-jre8 MAINTAINER gmoran
RUN echo "export JAVA_OPTS=\"-Dapp.env=staging\"" > /usr/local/tomcat/bin/setenv.sh

COPY ./hapi-fhir-jpaserver-example.war /usr/local/tomcat/webapps/fhir.war
CMD ["catalina.sh", "run"]


The Database (PostgreSQL) Dockerfile

---

FROM postgres:9.4  
MAINTAINER  gmoran

ENV POSTGRES_USER gmoran  
ENV POSTGRES_PASSWORD XXXXXXXXX  
ENV POSTGRES_DB fhirdata  

Create a Docker-Compose file to orchestrate and launch the system.

The docker-compose.yml file provided in the post above also works quite well for this application. I chose to stick with port 8080 for simplicity. Also I map port 5432:5432 so that I can use the psql PostgreSQL utility from any machine to interrogate the database tables.

The docker-compose.yml File

---

app-web: build: ./web ports: - "8080:8080" links: - app-db app-db: build: ./db expose: - "5432" ports: - "5432:5432" volumes_from: - app-db-data app-db-data: image: cogniteev/echo command: echo 'Data Container for PostgreSQL' volumes: - /var/lib/postgresql/data

Our next chore is to build the HAPI FHIR JPA example server. If you are familiar with Git and Maven, it is easy.

Download HAPI source from GIT. 

You can download the HAPI source following these instructions, or use the git clone command-line as follows:

$ git clone https://github.com/jamesagnew/hapi-fhir.git

Modify the FhirServerConfig.java to wire the database configuration. 

Since we want to use PostgreSQL, we will need to add the PostgreSQL JDBC driver jar to our Maven pom.xml file. The attribute values for the 9.4 version of the jar are as follows: 

                 <dependency>
   <groupId>org.postgresql</groupId>
   <artifactId>postgresql</artifactId>
   <version>9.4-1201-SNAPSHOT</version>
  </dependency>

In the source code navigate to the FhirServerConfig.java class. Navigate first to the hapi-fhir-jpaserver-example folder, then the class is nested down here: 

src/main/java/ca/uhn/fhir/jpa/demo/FhirServerConfig.java

Using your favorite code editor, make the changes shown in red as follows:

public DataSource dataSource() { BasicDataSource retVal = new BasicDataSource(); retVal.setDriver(new org.postgresql.Driver()); retVal.setUrl("jdbc:postgresql://app-db:5432/fhirdata"); retVal.setUsername("gmoran"); retVal.setPassword("XXXXXXXX"); return retVal; } @Override @Bean() public LocalContainerEntityManagerFactoryBean entityManagerFactory() { LocalContainerEntityManagerFactoryBean retVal = super.entityManagerFactory(); retVal.setPersistenceUnitName("HAPI_PU"); retVal.setDataSource(dataSource()); retVal.setJpaProperties(jpaProperties()); return retVal; } private Properties jpaProperties() { Properties extraProperties = new Properties(); extraProperties.put("hibernate.dialect", org.hibernate.dialect.PostgreSQL94Dialect.class.getName());

Note what we changed.

The driver class must be the PostgreSQL driver class (org.postgresql.Driver()).

The URL is the standard JDBC URL for connecting to a database, comprising the protocol, the host, the port number and the name of the database. The host is an interesting value: app-db. If you look back at our docker-compose.yml file, you will see that we named our container app-db, and therefore, we can reference that name as the hostname for the database. The rest of the values in the URL MUST match the values we set in the docker-compose and Dockerfile configurations.

host: app-db
port: 5432
database: fhirdata
username: gmoran
password: XXXXXXXX


Also note that we changed the hibernate dialect (org.hibernate.dialect.PostgreSQL94Dialect.class.getName()). If for some reason you change the version of PostgreSQL used in the Dockerfile, you will want to make sure this dialect class matches the version you chose.

Once you are satisfied your changes match  the configuration, save this file. We will now return to the command line to build the server using Maven.

Build the HAPI FHIR JPA Server example .war file.

One item to note: With the changes that we made to use PostgreSQL instead of Apache Derby, you will see errors in at the end of the build. These are test failures and did not affect the stability of the server, so I ignored them (don't judge me).

Return to a command line, and navigate in the HAPI source code to the hapi-fhir-jpaserver-example folder. Run the following command from that location:

$ mvn install

Locate the target folder in the hapi-fhir-jpaserver-example folder. There should be a hapi-fhir-jpaserver-example.war file created.

Copy the hapi-fhir-jpaserver-example.war file into the /web subfolder you created or downloaded with the Dockerfiles. 

Spin up an AWS EC2 instance (I used Ubuntu free tier). 

I'm not going to go into great detail on HOW to work with EC2. Amazon does a pretty good job with documentation, and you should be able to find what you need to start a free AWS account and launch an EC2 instance.

Be certain to allow access to SSH and port 8080 (or whatever port you may have used for the web application server) in your AWS Security Group for your server. Allow access to port 5432 as well, if you want to use psql or another database management utility with your PostgrSQL server. You will be given the opportunity to set this access in the EC2 Launch Wizard. 

Install Docker & Docker-Compose on your EC2 instance. 

You'll need to SSH into your EC2 instance to do these next installs.

The Docker guides have great instructions on how to install Docker and Docker-Compose on Ubuntu.

Note that you will want to add the Ubuntu user to the Docker group on your EC2 server so you aren't having to sudo all the time.

Move your Dockerfiles, Docker-Compose script and .war file to your EC2 instance.  

We've come a long way, and we are almost done. The last steps are to move your files to the EC2 instance, and run docker-compose to spin up the Docker containers on the EC2 server.

Once you have the tools installed, exit out of your EC2 instance terminal.

Use the tar utility to compress your files. From the root folder (the folder that holds your docker-compose.yml file), run the following command: 

$ tar cvzf web.tar.gz

You should now have a compressed file in the root folder named web.tar.gz that contains all the necessary files for the server. 

Next, use the SCP utility to upload your files to the EC2 instance. 

scp -i xxx.pem web.tar.gz ubuntu@my_aws_ec2.compute-1.amazonaws.com:/home/ubuntu

The xxx.pem file in the command above should be the .pem file that you saved from AWS when you created your EC2 instance. The hostname should match the hostname of your EC2 server instance.

Once the files are uploaded to your EC2 instance, SSH into the EC2 server once again. Find the folder that holds your web.tar.gz file.  If you followed along exactly as I did it, the file should be in the /home/ubuntu folder.

Use the tar utility once again to extract your files:

$ tar xvf web.tar.gz

Run Docker-Compose to launch your containers.

Navigate to the folder that contains your docker-compose.yml file. Run the following command:

$ docker-compose up

Now, with luck, you should have the HAPI FHIR example server up and running. You can test talking to your server by navigating to it in your browser:

http://my_aws_ec2.compute-1.amazonaws.com:8080/fhir/

Add Some Example Patient Resources to HAPI.

If you are looking to learn a bit about FHIR, it would be useful to have some FHIR resources to play around with. I found that HAPI also comes with a nifty CLI that will allow you to load data from the HL7 site just for this purpose. Sadly, that example data doesn't work and according to the GitHub issue, no one intends to fix it. 

All is not lost. Crucible is a clever project that has a number of useful tools for testing your FHIR implementations, and they include a site for generating "realistic but not real" patient resource data. This is what I used to load some data into my new server. 

In a browser, navigate to the Load Test Data tool. Enter your server's base URL, select your format and the number of patients you would like generated, and let tool work its magic. 

I loaded 100 patients, and now have a large enough stash to start really learning something about FHIR. But I think that will have to wait until tomorrow, this was enough for one day!

Cheers, 

G

Today's the day:) Hello Hitachi Data Systems

Just over 10 years, and  it has finally happened.

Hitachi Data Systems intends to acquire our baby, Pentaho.

I couldn't be more excited.  Pentaho is the fast, maneuverable Destroyer coming alongside the Hitachi Battleship, eagerly pursuing dominance of the IoT and Big Data space.  I can't think of a better fit for us as a company ready to do big things in a big market, or as a culture of innovators, entrepreneurs and talented, hard-working engineers. So many people have committed themselves to the Pentaho vision for the past decade, people I know like family. Congratulations to my Pentaho family, and Hello Hitachi.

Looking forward to all we will achieve together :)

From Pentaho: http://www.pentaho.com/hitachi-data-systems-announces-intent-to-acquire-pentaho

From the CEO: http://blog.pentaho.com/2015/02/10/a-bolder-brighter-future-for-big-data-analytics-and-the-internet-of-things-that-matter-pentaho-hds/

Pedro Alves: http://pedroalves-bi.blogspot.pt/2015/02/big-news-today-hitachi-data-systems-hds.html

From Hitachi: https://community.hds.com/community/innovation-center

Bloomberg: http://www.bloomberg.com/news/articles/2015-02-10/hitachi-to-buy-pentaho-to-bolster-data-analysis-software-tools

MDX: Converting Second of Day to Standard Time Notation

Had a bit of fun with Pentaho Analyzer recently. In the release of Pentaho 5.2, we have introduced the ability to define filters across a range of time, which is really handy when your dataset is millions of records per second.

My use case included keying our time dimension on seconds per day, which results in 86,400 (60 seconds * 60 minutes * 24 hours) unique records; one to represent each unique second in a day. While this is great for simplifying query predicates, it does not help the usability or intuitiveness of the analysis report you are presenting to the user. For instance, who would intuitively understand that 56725 represents 15:45:25 in time?

So I came up with this user-defined calculation that will convert seconds in a day to standard time notation. Would love to hear from anyone who can optimize this:)  This is a valid MDX calculation that Mondrian will process. Since I needed to know the minimum and maximum second per hour in the display, I used the second of day number as a measure.

Format(Int([event scnd of day min]/3600), "00:") || 
  
  Format(Int(([event scnd of day min] - 
        (Int([event scnd of day min]/3600))*3600)/60), "00:") ||
    
    Format([event scnd of day min] - 
          ((Int([event scnd of day min]/3600)*3600) + 
          (Int(([event scnd of day min] - 
          (Int([event scnd of day min]/3600)*3600))/60)*60)), "00")


Here's what it looks like in Analyzer.  The columns Minimum & Maximum Second of Hour have the calculation applied to them. Note the time filter range in the filter panel. Super sweet.

Hello Docker.

Docker. Hmmmm. I really want to love it. Everybody else loves it, so I should right? I think maybe some of the "shiny" isn't so bright using DOCKER ON MY MAC.  Although, Chris J. over at Viget wrote this blogpost that singularly walked me through each Mac-Docker gotcha with zero pain.   Total stand-up guy, IMHO.

If you are not familiar, Docker plays favorites to Linux based operating systems, and requires a virtual machine wrapper called boot2docker in order to run on a Mac or Windows OS. Not a huge hurdle, but definitely feels heavier and a bit more maintenance intensive ... two of the core pain points in traditional virtual environment deployments that Docker proposes to alleviate.

Beyond that silliness, there is a whole lot more *nix based scripting than I expected. Somehow I thought the Dockerfile language would be richer, accommodating more decision-based caching. You know, something like cache this command but not this one.  As I looked around and read a few comments from the Docker enthusiasts and Docker folks-proper, it seems there is a great desire to keep the Dockerfile and it's DSL ... well ... simple. Limited? Is that a matter of perspective? I can appreciate simple I guess, but I still want to do hard stuff ... and thus I am pushed to the *nix script environment. This may just be a matter of stuffing myself into these new Docker jeans and waiting for them to stretch for comfort:)

One blessed moment of triumph I would like to share: I was able to write a Dockerfile that would accommodate pulling source from a private Github repository using SSH. This is NOT a difficult Docker exercise. This is a persnickety SSH exercise:) The Docker container needs to register the private SSH key that will pair with the public key that you have registered at Github. At least that is the approach I took. Please do let me know if there are easier / better / more secure alternatives.

So, the solution. The first few steps, I'm going to assume you know how to do, or can find guidance. They are not related to the container setup.

I'm going to tell you right up front that my solution does have a weakness (requirement?) that may not be altogether comfortable, and Github downright poo-poos it. In order to get the container to load without human intervention, you need to leave off the passphrase when you generate your SSH keys (Gretchen ducks.).  I planned to revisit this thorn, but just simply ran out of time. Would love to hear alternatives to this small snafu. Anyway, if you're still in the game,  then read on...

Here are the steps you should follow to get this container up and running.

1. Generate a pair of SSH keys for Github, and register your public key at github.com.
2. Create a folder for your Docker project.
3. Place your private SSH key file (id_rsa) in your Docker project folder.
4. Create your Dockerfile, following the example below.
5. Build your image, and run your container.
6. Profit:)

The Dockerfile

FROM gmoran/my-env
MAINTAINER Gretchen Moran gmoran@pentaho.com

RUN mkdir -p /root/.ssh

# Add this file ... this should be your private GitHub key ...
ADD id_rsa /root/.ssh/id_rsa

RUN touch /root/.ssh/known_hosts
RUN sudo ssh-keyscan -t rsa -p 22  github.com >> /root/.ssh/known_hosts

Running as root User

I am referencing the root user for this example, since that is the default user that Docker will use when you run the container. If you would like a bit more protection, you can create a user, and run the container with that user with the following command ...

USER pentaho

I created the 'pentaho' user as part of a Dockerfile used in the base image gmoran/my-env. IMPORTANT: Note that gmoran/my-env also downloads the OpenSSH daemon and starts is as part of the CMD Dockerfile command.

Adding the id_rsa File

The id_rsa file is the private SSH key generated as part of the first step in this process. You can find it in the directory you specified on creation, or in your ~/.ssh directory.

There are a number of ways to add this key to the container. I chose the simplest ... copy it to the container user's ~/.ssh directory. OpenSSH will look for this key first when attempting to authenticate our Github request.

Adding github.com to the known_hosts File

We add the github.com SSH key to the known_hosts file to avoid the nasty warning and prompt for this addition at runtime.

In my thrashing on this, I did find several posts in the ether that recommended  disabling StrictHostChecking, which hypothetically produces the same end result as manufacturing/mod'ing the known_hosts file. This could however leave this poor container vulnerable, so I chose the known_hosts route.

At the End of the Day ...

So at the end of the day, when I thought I would be honing my Docker skills, I actually came away a with a stronger set of Unix scripting skills. Good for me all in all.  I am excited about what Docker will become, and I do find the cache to be enough sugar to keep me drinking the Docker kool-aid.

I should say I appreciate not actually having to struggle with Docker. It is a nice, easy, straight-forward tool with very few surprises (we won't talk about CMD versus ENTRYPOINT). Any time-consuming tasks in this adventure were directly related to my very intentional avoidance of shell scripting, which I now probably have a tiny bit more appreciation for as well.

In the words of the guy I like the most today, Chris Jones ... Good Guy Docker :) 

Pentaho Analytics with MongoDB

I love technology partnerships. They make our lives as technologists easier by introducing the cross sections of functionality that lie just under the surface of the products, easily missed by the casual observer. When companies partner to bring whole solutions to the market, ideally consumers get more power, less maintenance, better support and lower TCO.

Pentaho recognizes these benefits, and works hard to partner with technology companies that understand the value proposition of business analytics and big data. The folks over at MongoDB are rock stars with great vision in these spaces, so it was natural for Pentaho and MongoDB to partner up.

My colleague Bo Borland has written Pentaho Analytics with MongoDB,  a book that fast tracks the reader to all the goodness at your fingertips when partnering Pentaho Analytics and MongoDB for your analytics solutions.  He gets right to the point,  so be ready to roll up your sleeves and dig into the products right from page 1 (or nearly so).  This book is designed for technology ninjas that may have a bit of MongoDB and/or Pentaho background. In a nutshell, reading the book is a straight shot to trying out all of the integration points between the MongoDB database and the Pentaho suite of products.

You can get a copy of Pentaho Analytics with MongoDB here.  Also continue to visit the Pentaho wiki, as these products move fast.

Pentaho's Women in Tech: In Good Company

I was honored this week to be included in a a blog series that showcases just a few of the great women I work with, in celebration of International Women's Day on March 8.

Check out the series, I think you'll find the common theme in the interviews interesting and inspiring. Pass on the links if you have girls in your life that could be interested in pursuing technology as a career. 

Pentaho's 12 Days of Visualizations

If you are interested in the ultimate extendability of Pentaho's visualization layer, you'll love this fun holiday gift from Pentaho: 12 Days of Visualizations.  Check back each date marked for a new plugin that demonstrates Pentaho leveraging cool viz packages like Protovis, D3 and more.

http://wiki.pentaho.com/display/COM/Visualization+Plugins

Today's visualization: the Sunburst!




Merry Christmas and Happy New Year!

Resolving "AppName is damaged and can't be opened." Don't move it to the trash!

I recently stumbled across this problem with one of Pentaho's applications. When the application was downloaded and installed on a Mac, launching the .app file resulted in "This app is damaged and can't be opened. Move to the trash".

 Relatively quickly with a few searches, we figure out that GateKeeper was the messenger, but why was she being so harsh? Our apps are unsigned (a signature improvement slated for the next release), but damaged? I was offended.

As it turns out, Apple has a decent support article that explains why you might get a "damaged..." message versus GateKeeper's standard message warning the user that the application is unsigned.

The answer to softening GateKeeper's tone (AKA getting her to only prompt with a security message rather than a "damaged" message) lies within the info.plist file within the .app. Kurtis, our .app builder, found that if he sets the following values, then the .app reverts to being a harmless unsigned .app.

<key>CFBundleSignature</key> 
<string>????</string>

I hope this solution saves someone else the heartache of deploying a"damaged" .app file.

kindest regards, 

Gretchen

Pentaho and OpenMRS Integration

We have a great opportunity to explore how Pentaho can provide ETL, analytics, and reporting benefits to OpenMRS, an open source medical records platform and community interested in global health care.

Check out the first projects underway, and decide if you have time to participate:

Pentaho ETL and Designs for Dimensional Modeling

Cohort Queries as Pentaho Reporting Datasource
This project still needs a lead developer; we'd like to have these projects run in tandem.

To get involved, feel free to email me directly, or contact any of the OpenMRS mentors listed in the projects.

kindest regards and in His grace,
Gretchen

PCM11: Continuity and Change @ Pentaho

Last week, I enjoyed my third (of four) Pentaho Community meetup, this year held in Rome (Frascati), Italy. Jan Aertsen did a fantastic job summarizing the presentations, you can review them all here, including access to the presentation materials. At this particular juncture, I find myself in my longest commitment to a single company in my career. The entire ride has this very cool thread of continuity through tides of swift and constant change that comes with being a bleeding edge software company.

When I look back over the past seven years, many times I focus solely on Pentaho milestones and growth, the markets we've entered and enjoyed success with, the new initiatives that take hold. PCM11 gave me a look at the global reach of success that Pentaho enjoys, creating opportunity and economy beyond the bounds of the company official. This is what makes open source make sense to me. This appeals to me.

The people that make up the Pentaho community are a talented, committed group of individuals who are growing in their own endeavors, many based on the community edition of the Pentaho BI Suite of tools. Many of our community colleagues have been committed to Pentaho from the earliest releases of 2004 and 2005. Their efforts are paying off, and while Pentaho the company doesn't get everything right, we've managed to earn the respect and partnership of some incredibly driven and talented people.



Another interesting phenomena - community members becoming Pentaho employees, blurring any lines that get drawn at times between community and corporate.

From the ranks of the Pentaho community a well of talent has sprung - Slawo, Roland, Jan, Jens, and a handful of others. Pentaho is incredibly savvy in hiring from the community. Our community is the hotbed of Pentaho, DBA, big data, analytic and reporting knowledge, both from a project development perspective and from a solutions development perspective. How many software projects suffer from the writers not understanding the use cases? Not eating their own dog food? Well, the newest Pentaho developers have been at that bowl for some time, and the internal developers can help them keep that commitment with internal initiatives delivering Pentaho solution driven information.

And what of the other direction? Those leaving the formal Pentaho realm and working entirely community based? Well, that would be me. It's not like this is new news - I'm now infamous for my off-again, on-again relationship with formal employment :) Don't mistake me for irresponsible; I just have higher priorities. We all should be so blessed, right?



The great news is I also have reaped the benefits of a long series of lessons in BI, big data, analytics, reporting, visualizations, problem solving and code writing. So I take these lessons learned into the community and can begin to give back a little. To my fellow community members, to other open source projects, to Pentaho.

One project that has caught my attention is the OpenMRS project. OpenMRS is a medical records system platform widely deployed throughout the compromised countries of the world. OpenMRS is open source, and has a thriving community of developers, implementers, users and observers from well established world health organizations.

I intend to spend the last quarter of this year investigating integration points between Pentaho tooling and OpenMRS. OpenMRS could use more insight into their data; Pentaho is an excellent set of tools for turning raw data into information. I see synergies here :)

Soon, there will be a project page to stay informed if you're interested or would like to participate. I'll post back as soon as I have the leg work done. In the meantime, checkout http://www.openMRS.org. It's a very rational site that gets you up to speed quickly on the project.

Cheers & all in His grace,
Gretchen

Jimmy D. takes a look at 2010 and where Pentaho is present

I couldn't resist re-posting this link to James' blog - these numbers are sooo exciting! More so for me since I remember when Pentaho was largely comprised of a small rented space and some beanbag chairs :)

James takes a look at 2010 and where Pentaho is present.

Kindest regards,
-G

Troubleshooting Localization

I've been gathering some interesting and useful information when dealing with Pentaho Reporting, Pentaho Metadata and characters not represented in the standard ASCII character set. This bucket of tips will make it into our documentation ASAP, but I thought it prudent to share it with our community even sooner.

IMPORTANT CAVEAT: Note that where I specify UTF-8, I am only doing that as a reference encoding... the encoding I speak of in most cases can represent any extended character set; UTF-8 is a common one for multi-national apps, because it represents multi-national characters.

Character encoding is key to displaying multi-byte or special characters from character sets outside of the standard ASCII character set. Any text-based files that contain special characters in their glyph form must be encoded as at least UTF-8, or in the character encoding for the language you are attempting to display.

The character encoding is significant no matter where these characters reside or travel - if the file or database stores the characters as UTF-8, then Java must handle those characters as UTF-8 and where ever the characters' destination is, be it a browser window or system file, the destination must also render the characters using the same character encoding.

So, this means:

1. Check any and all TXT or CSV files in an appropriate editor to verify that they are encoded in the correct character encoding. In a pinch, Notepad will do, but if you are seriously dealing with localization, it's in your best interest to invest in or download a good unicode text editor.

2. Make sure that your HTML and XML files have a meta tag specifying your chosen as the character set. For example:

<?xml version="1.0" encoding="UTF-8"?>
<meta http-equiv="Content-Type" content="text/html; charset="UTF-8" >

And, if it actually appears in an xhtml document (as suggested by the xml declaration) the content type should probably text/xhtml, and the meta tag should be closed in itself like so:

<meta http-equiv="Content-Type" content="text/xhtml; charset="UTF-8" />

3. The Pentaho BI Server allows you to specify a default encoding in a context parameter in the web.xml file of the webapp. This "default encoding" applies to any XML documents that the server generates. The platform adds an xml prologue to these documents and sets the encoding to that of the BI Platform, which comes from web.xml. By default, the server assumes this is UTF-8. If you want a different default encoding, specify it in the web.xml.

4. You also want to make sure that the default encoding that Java (specifically, the JVM that is running the Pentaho application) is using matches the encoding that the Pentaho application is using. We just mentioned that the default encoding for the Pentaho BI Server is UTF-8. So, what is the default encoding for the JVM? The JVM determines it's encoding from the system property "file.encoding". As of Java 1.4.2, this property is available and set from as the default OS locale. However, on Windows systems, this default locale may not exist, so Java makes a best guess. As you can see, knowing what the default encoding is can be a bit nebulous, so we recommend setting the encoding for Java on the command line:

java -Dfile.encoding=UTF-8

You will want to add this command line parameter to any Pentaho application startup script that you are attempting to use internationally. Specifically for the Pentaho BI Server, you would want to set this command line parameter in the start-pentaho.bat | .sh script.

It's important to note that we don't demand UTF-8. We do (for now) demand that whatever file.encoding is specified is what the web.xml context parameter "encoding" says. So - as long as this param says ISO-8859-1 and file.encoding says ISO-8859-1, you're still good.

Next, understand that common fonts do not have all of the characters possibly represented in the UTF-8 character set or other extended character sets. So, while your encoding may be correct, if you specify a font that doesn't include the glyph for a multi-byte character, it's likely to render as a square, question mark, or some other seemingly unrelated character.

A good test font on Windows systems is "Arial Unicode MS", which is distributed with MS Office and is claimed to have every UTF-8 character glyph available. It's ability to represent every character makes it a good TEST font, but comes with a price - the font is nearly 24 MB. You do not want to recommend this as a production font, since as a best practice guideline, we tell customers to embed their fonts with certain output formats, and this font would equate to staggering overhead in download sizes. The proper recommendation is to tell customers to find the font that best represents the consumer base's languages for that report.

So how do we control which fonts and encodings are used in Pentaho Reports? It's a bucket of valuable information I'm attempting to summarize here:

First encodings:

In Pentaho reports, there are global configuration properties for the different output formats. The global report engine configuration can be found in the Pentaho BI Server installation under the pentaho webapp: pentaho/WEB-INF/classes/classic-engine.properties.

org.pentaho.reporting.engine.classic.core.modules.output.table.html.Encoding=UTF-8
org.pentaho.reporting.engine.classic.core.modules.output.pageable.pdf.Encoding=UTF-8
org.pentaho.reporting.engine.classic.core.modules.output.table.csv.Encoding=UTF-8

And fonts:

1. If you have a metadata model in play, make sure that the metadata concept properties for the font-family are all set to a font that is installed on the server serving up the model and is capable of rendering the special characters you need represented. There is a Base concept (found in the Concept Editor) that has a default font-family that you will want to verify/modify is configured correctly.

2. If you are using any of the templates designed for Report Design Wizard or Web Adhoc Query and Reporting, you will want to verify/modify those templates to use a font that is capable of rendering the special characters you need represented. The templates for Report Design Wizard are found in the Report Designer's /templates directory. The templates for WAQR are found in the Pentaho BI Server solutions directory under pentaho-solutions/system/waqr/templates.

3. On Windows, what determines whether Pentaho can find an installed font? A few things! First, look in the Windows Control Panel (or modern equivalent), under Fonts... these are the fonts that should be available to the reports generated by the Pentaho BI Server. If for some reason you want to include a font not in the system fonts directory, you can add additional directories of fonts.

This is done using a configuration file that you would create and place in the Pentaho webapp WEB-INF/classes directory, which basically creates an override for the configuration file that is found in the libfonts-x.x.x.jar library in the Pentaho webapp primary classpath. The name of the libfont report configuration is libfont.properties. Create this file, place it in the classes directory and add the following configuration property to it, with your font location of course.

org.pentaho.reporting.libraries.fonts.extra-font-dirs.myNewDir=c:/myNewDir/myFonts

Note: There is an open issue with this property that should be fixed with the SUGAR release of the Pentaho BI Server: http://jira.pentaho.com/browse/PRD-2145.

4. Our best practice recommendation for ensuring the proper rendering of special characters in PDF reports is:
a. Embed the font. This can be accomplished using the following global reporting configuration property: org.pentaho.reporting.engine.classic.core.modules.output.pageable.pdf.EmbedFonts=true
b. The font should be a TrueType font.

Also important to note is that you can confirm what fonts the Pentaho BI Server is aware of, as the reporting engine creates a cache of the fonts it has registered. If you are at all concerned that the server hasn't correctly registered a new font from the system, you can blow away the cache, restart the server, and the reporting engine will load all system fonts anew.

The cache exists at $HOME/.pentaho/cache/libfonts.

Pentaho Architect's Bootcamp Training Now Available

Last week, I had the pleasure of offering our very first 5-day session of the Pentaho Architect's Bootcamp training, and overall, it was a big success!

There comes a point in many advanced deployments of the Pentaho BI Suite where some feature or requirement pushes the boundaries of the out-of-the-box product capabilities. Since the beginning of Pentaho time, we've marched to the beat of "make it possible first, then make it pretty/easy", and it's this scenario where our approach pays big dividends to our customers/community/users. Because the platform/server/tools were built for extensibility, there are numerous places where you can roll up your sleeves and leverage a simple API to implement a customization that suits your specific requirements.

The Pentaho Architect's Bootcamp is geared for developers, partners, customers, consultants that are ready to roll up their sleeves and understand the complex problems that are surfacing in large scale BI implementations, and how to extend the Pentaho suite of products to answer far more questions and customizations than the boundaries of the out-of-the-box product.

Here's a sampling of some of the questions that are answered during Architect Bootcamp training:

• How do I integrate my own custom visualizations (maps, charts, gauges, etc) into Pentaho?
• How do I accommodate multiple companies/groups/organizations' data in my solutions, while maintaining each companies/groups/organizations' personal point of view of the data?
  
• How do I dynamically drive row level security in Pentaho Analysis and Pentaho Metadata?
  
• How do I integrate Pentaho solution content into my own application?
  
• How do I customize security across the Pentaho platform and pillars?
  
• How do I create integrated solutions using ETL, reporting, analysis and metadata to deliver my customer's specific solutions?
• How do I plug my custom content into the Pentaho BI Server? How do I then integrate my custom data/functionality with the Pentaho pillars (ETL, reporting, analysis, metadata, etc)?
  

I really believe this course is a game-changer for Pentaho users and solution developers. I teach the course, so don't take just my word for it. Here's some feedback we received from the first course offering:

"We traveled from India to Florida for this course, and we are extremely glad that we did. The information is very valuable."

"The presentation was excellent, [I e]specially liked the interactive nature of the class."

"All of the lessons uncovered new boundaries; all topics had more priority!"

"YEEEEAAAA.. just built my first Pentaho BI Server plugin... Pentaho Architect's Bootcamp RULES!!! #Pentaho @Pentaho"

"Day 4 Pentaho Architect's Bootcamp brought so many excitements, I loved it. even though my brain feels like it's going to explode!! #Pentaho"

Barcelona Pentaho Community Meetup 2009 Pics

We fly home from Barcelona tomorrow, a lovely vacation and another fantastic Pentaho community gathering.

I'll be chatting with many of you online in the near future, and hopefully will see everyone again next year - Vienna is it? Sweet!

As promised, here's my pics :)

kindest regards,
G

Hola from Barcelona, Community Gathering 2009

We're close to restarting the sessions for the afternoon, just dropping in to update my fellow Pentaho colleagues on the gathering:)

We got a bit of a late start this morning, mostly because the community started the meetup last night, and the socializing lasted into the wee hours for the group. A great time was had by all :) The morning's speakers had great content, covering a variety of topics, from Mozilla statistics presentation with CDF to the latest revision of PAT, the community analysis tool. Roland and Jos are here, our celebrated Pentaho Solutions authors, signing books and presenting the basics of developing custom CDF components.

And in traditional European holiday style, we're late in getting the afternoon sessions started, most of our group (around 40 attendees with community and Pentaho included)is still in earnest roadmap discussions at the cervesseria :)

Next post, some pics for posterity.

buenos tarde!
Gretchen

Pentaho Community, Together in Barcelona

Doug and I arrived in Barcelona this morning, early enough to see some of this beautiful city before the Pentaho Community Meetup this weekend.

This is the second annual community meetup, an event that is organized and planned completely by Pentaho community for Pentaho community. No fluffy corporate speak, just a full weekend of Pentaho community developers and users showing off their stuff, talking through their current projects and solutions, and having a few beers and some fun. Many thanks to Tom Barber for planning and sponsoring much of this year's event.

We look forward to seeing familiar faces, and new community as well:0) See you all very soon!

Thanks Roland and Jos: Pentaho Solutions IN PRINT!

I received my pre-ordered copy yesterday of Pentaho Solutions: Business Intelligence and Data Warehousing with Pentaho and MySQL. A huge congratulations and thank you to Roland Bouman and Jos van Dongen, two long time Pentaho community members who wrote the book.

I can' t tell you how excited I am to see this book! For many years, developers and project managers that I've worked with have felt that a book like this one is the missing link to helping customers achieve success with their warehouse and business intelligence strategies. Most books on business intelligence are either too abstract or offer guidance only on select pillars (for example, only reporting solutions), which leave the reader with unfulfilled requirements and no direction for filling in the gaps.

With Pentaho Solutions, the reader gets a concrete explanation and best-of-breed Pentaho implementation of ETL, reporting, analysis, dashboarding and data mining solutions; 5 core pillars and their concepts that contribute to a healthy, whole, successful BI strategy and implementation.

You can pre-order your copy at Amazon.com :)

Roland and Jos, the team has already sunk their teeth in, and they love what they're reading. Well, the picture says it all :)

Development and Debugging with GWT and Javascript

Java code is the bread and butter of what I do, but as most Java developers know, there is a plethora of good frameworks and technologies that surround Java and provide a means to build very powerful and extendable software.

Lately, I've popped back into GWT land and have been dealing with lots of Javascript, both original and generated. I originally introduced myself to GWT building a small volunteer information submittal form for Brevard Rescue Mission. This tiny application only scratched the surface of what GWT could do in its earliest stages. The magic that the Pentaho development team have performed with Pentaho Dashboarding is a new level of web-goodness, fully capitalizing on the power of GWT. I've been dabbling in the chart rendering layers of dashboards, and have learned some simple, effective means of making life a bit easier when dealing with debugging and developing Javascript and GWT generated Javascript. I hope you find these tips useful, and it's certainly nice to have them aggregated in one place! I have to give credit for this info to Nick Baker and Mike D'Amour, two of my colleagues at Pentaho. Use this blog post as a starting point for googling the original sources for more details on each tip:)

Helpful tips for Developing and Debugging with GWT & Javascript in General

Limit the user.agent Property

GWT compiling is resource intensive due to the number of compilations that happen for the browsers supported. At times, you will run out heap space or other resources before the compile can finish (this usually manifests itself as a StackOverflowError).

The following entries in your *.gwt.xml file can help by only compiling for the single browser you may be testing on:

<inherits name="com.google.gwt.user.UserAgent"/> 
  <set-property name="user.agent" value="ie6" />

Valid values for the user.agent property are: ie6,gecko,gecko1_8,safari,opera

Limit the gwt.compile.localWorkers

You can also scale back the number of threads to use for running parallel compilation. While this may hurt performance, you will be able to finish the compilation without running out of resources. This property, gwt.compile.localWorkers, can be added to the compile option in your ant script.

Bump the GWT version from 1.6.4 to 1.7.0

GWT 1.7.0 seems to have resolved many of the compilation resource issues with GWT.

GWT Pretty Print Compile

By default, we obfuscate our GWT compiled Javascript. To debug readable GWT compiled Javascript, compile with pretty print turned on. This property, gwt-style, can be added to the compile option in your ant script. Valid values include OBF, PRETTY, and DETAILED.

IE Javascript Debugging Help

If you need to debug Javascript in IE, it is highly recommended that you get IE8. You can install IE8 for the duration of your testing, then uninstall it when you no longer need it, as it has conflicts with GWT. IE8 has a new set of features called Developer Tools that make debugging Javascript very easy.

Helpful In Line Javascript Alerts

You can use the following line of code to send alert windows whereever you like in your Javascript code:

$wnd.alert("Hello World")

A good example from Nick:

This is literally saving me hours. By adding a line to the the end of the
printStackTrace() function you can alert out the stacktraces that normally do nothing when compiled.

Open up the gwt script file (xxxxxxxxxxxxxxxxxxxxxxxxx.cache.html) for your particular browser. I find it by seeing what's loaded in firebug.

Search for "function $printStackTrace"

Add a new line right before the function returns:

$wnd.alert(msg.impl.string);

It should now look like this.

function $printStackTrace(this$static){
var causeMessage, currentCause, msg;
msg = $StringBuffer(new StringBuffer());
currentCause = this$static;
while (currentCause) {
causeMessage = currentCause.getMessage();
if (currentCause != this$static) {
msg.impl.string += 'Caused by: ';
}
$append_4(msg, currentCause.getClass$().typeName);
msg.impl.string += ': ';
msg.impl.string += causeMessage == null?'(No exception detail)':causeMessage;
msg.impl.string += '\n';
currentCause = currentCause.cause;
}
$wnd.alert(msg.impl.string);
}

In Code Breakpoints

Rather than sifting through the script debugger window trying ot figure out where to put a breakpoint, you can use the following line of code to embed a breakpoint:

debugger;

Feel free to comment and send your favorite tricks for working with GWT and Javascript.

Pentaho Analysis Tool Integrated as a Pentaho Plugin

I had the chance this week to play around with the still-under-construction Pentaho plugin architecture in the Citrus code line. The new architecture is just what BI developers have been waiting for: totally flexible with several new ways to integrate with the server, simple to use and allows for building nicely decoupled extensions.

With Aaron Phillip's help, I got my head around the new features in less than a day, and had my first plugin written shortly after: The Pentaho Analysis Tool (PAT) plugin. Before I get into the details of the PAT plugin, let's first talk about the new tools and capabilities in the Pentaho BI server's plugin layer.

The plugin architecture consists of several different fun ways you can hook into the Pentaho BI Server, without having to modify server code or disturb the platform deployment. All avenues for leveraging the plugin architecture expect that the necessary files and code will be found in the solutions folders. The layer currently has the following capabilities:

1. Customization of the menu system of the "classic" and more recent PUC (Pentaho User Console) user interfaces
2. Customization of various page contents (overlays)
3. New types of content to be added to the solution repository and operated upon in the user console
4. New Java classes that generate UI pages to be dynamically added to the server
5. (new in 3.0) Add your own BI Component to the platform without having to modify system files and paths

You can get more details about these features and how they work by reading the documentation here. Aaron also created a sample plugin that demonstrates each of these features in a simple plugin mockup, that also is a great template to use for new plugin creation.

So that's exactly what I did. Here's a screenshot of the results of my plugin:



Using the EchoPlugin sample as a guide, I created a new content type (.xpav, for Pentaho Analysis View) which is the first notion of a view definition file for PAT. When you "open" this new content type in PUC, it initializes and launches PAT, which is a separately deployed web application. This is accomplished by creating a new content generator in the plugin that delegates the generation to the PAT webapp. It takes a bit to put it all together: you need a bleeding edge Citrus BI Server download, the latest PAT code and the plugin project. If you are interested in seeing it in action, read the integration instructions here.

I only took advantage of a couple of the new plugin layer's capabilities in my first plugin. I'm looking forward to playing with the new web services as well as the component that allows my plain old Javabean to look like a BI component automagically. I can foresee great extensions coming fast for the Pentaho BI Server with this new architecture!

I've listed some good references for those who are ready to take a look at plugins:

Here is the documentation:
http://wiki.pentaho.com/display/ServerDoc2x/BI+Platform+Plugins+in+V2

and here is the Plugin Depot, where you can show others the cool new extensions you've built:
http://wiki.pentaho.com/display/ServerDoc2x/Plugin+Depot

and if you have questions, comments or problems, or think you may have spotted a bug, chat with some of the developers about it here:
http://forums.pentaho.org/forumdisplay.php?f=73

kindest regards,
Gretchen

Maven: The Definitive Guide

My one true nerdy tendency: I like writing technical documentation. It's ironic then (or a bit of a hypocrisy) that I loathe reading it. Chalk it up to my lack of passion for technology. I am a passionate problem solver; technology is a sometimes rewarding, sometimes frustrating means to help effectively get the problem solving job done.

Recently, I began reading Maven: The Definitive Guide while getting a pedicure at my local salon (laugh it up guys, I can guess where most of you do your leisure reading). I strongly recommend that any developer approaching Maven for the first (or tenth) time give chapters 3 through 8 a read. This guide is what you hope most technical guides or books would be, but then usually quite early on, they disappoint.

The guide starts with a quick, understandable introduction to Maven terminology and concepts, via a short step by step example. As I was reading this from a "make this worth my while" perspective, I had specific use case questions that immediately popped into my head ... and then, I was pleasantly surprised to find the answers in the next few paragraphs.

For example, the guide mentions early on that "support for transitive dependencies is one of Maven's most powerful features". To that my questions were "What about conflicts in dependency hierarchies?" and "What about compile time dependencies that I don't want to package?". The rest of the chapter addresses exactly those questions with explanations on dependency exclusions and scoping. Finally. A book that thinks like I do :)

A quick summary of the rest of the meat of the guide: chapters 4 through 8 build on the core concepts introduced in 3, with bite size chunks of additional functional explanation in each chapter. The material is presented as a hands on example, building in feature complexity little by little. Chapter 4 shows you how to add new dependencies to your project; 5 introduces simple web application features; chapters 6 and 7 cover multi-module project and enterprise project features. This presentation worked for me on a few different levels:

1. The graduated approach to the introduction of new materials made a large amount of Maven terminology, concepts and finally usage documentation digestible.
2. The authors take great care in describing WHY they arrange and refactor the projects as they do, in a very modular fashion. This approach, in practice, lends itself not only to Maven's default conventions, but also to best practices for software project layout. Note that this introduced complexity to the examples that wasn't necessary to explain the features at hand. But the authors bit that bullet in order to present a good and useful way of developing a project.
   
3. The example projects described in the guide are immediately relevant for me. I write Java code. I use Spring. I use Hibernate. This, of course, will not be the case for every reader, but it was a nice bonus for me.
   

So after all of my monologue thoughts, I leave you with a few tids:

• Read the guide (at least Part 1), then decide how problematic you perceive Maven to be. I know my perception changed dramatically.
• Note that it is a bit outdated, deprecated goals and such (the guide is updated for Maven 2.0.10, I downloaded Maven 2.1). This really didn't distract me at all.
• The chapters I reference are only the tip of the iceberg. Part 2 of the guide includes another 200 reference pages that I have yet to use. I'll let you know how that goes for me:)