Pentaho's 12 Days of Visualizations

If you are interested in the ultimate extendability of Pentaho's visualization layer, you'll love this fun holiday gift from Pentaho: 12 Days of Visualizations.  Check back each date marked for a new plugin that demonstrates Pentaho leveraging cool viz packages like Protovis, D3 and more.

http://wiki.pentaho.com/display/COM/Holiday+Visualizations

Today's visualization: the Sunburst!




Merry Christmas and Happy New Year!

Resolving "AppName is damaged and can't be opened." Don't move it to the trash!

I recently stumbled across this problem with one of Pentaho's applications. When the application was downloaded and installed on a Mac, launching the .app file resulted in "This app is damaged and can't be opened. Move to the trash".

 Relatively quickly with a few searches, we figure out that GateKeeper was the messenger, but why was she being so harsh? Our apps are unsigned (a signature improvement slated for the next release), but damaged? I was offended.

As it turns out, Apple has a decent support article that explains why you might get a "damaged..." message versus GateKeeper's standard message warning the user that the application is unsigned.

The answer to softening GateKeeper's tone (AKA getting her to only prompt with a security message rather than a "damaged" message) lies within the info.plist file within the .app. Kurtis, our .app builder, found that if he sets the following values, then the .app reverts to being a harmless unsigned .app.

<key>CFBundleSignature</key> 
<string>????</string>

I hope this solution saves someone else the heartache of deploying a"damaged" .app file.

kindest regards, 

Gretchen

Pentaho and OpenMRS Integration

We have a great opportunity to explore how Pentaho can provide ETL, analytics, and reporting benefits to OpenMRS, an open source medical records platform and community interested in global health care.

Check out the first projects underway, and decide if you have time to participate:

Pentaho ETL and Designs for Dimensional Modeling

Cohort Queries as Pentaho Reporting Datasource
This project still needs a lead developer; we'd like to have these projects run in tandem.

To get involved, feel free to email me directly, or contact any of the OpenMRS mentors listed in the projects.

kindest regards and in His grace,
Gretchen

PCM11: Continuity and Change @ Pentaho

Last week, I enjoyed my third (of four) Pentaho Community meetup, this year held in Rome (Frascati), Italy. Jan Aertsen did a fantastic job summarizing the presentations, you can review them all here, including access to the presentation materials. At this particular juncture, I find myself in my longest commitment to a single company in my career. The entire ride has this very cool thread of continuity through tides of swift and constant change that comes with being a bleeding edge software company.

When I look back over the past seven years, many times I focus solely on Pentaho milestones and growth, the markets we've entered and enjoyed success with, the new initiatives that take hold. PCM11 gave me a look at the global reach of success that Pentaho enjoys, creating opportunity and economy beyond the bounds of the company official. This is what makes open source make sense to me. This appeals to me.

The people that make up the Pentaho community are a talented, committed group of individuals who are growing in their own endeavors, many based on the community edition of the Pentaho BI Suite of tools. Many of our community colleagues have been committed to Pentaho from the earliest releases of 2004 and 2005. Their efforts are paying off, and while Pentaho the company doesn't get everything right, we've managed to earn the respect and partnership of some incredibly driven and talented people.



Another interesting phenomena - community members becoming Pentaho employees, blurring any lines that get drawn at times between community and corporate.

From the ranks of the Pentaho community a well of talent has sprung - Slawo, Roland, Jan, Jens, and a handful of others. Pentaho is incredibly savvy in hiring from the community. Our community is the hotbed of Pentaho, DBA, big data, analytic and reporting knowledge, both from a project development perspective and from a solutions development perspective. How many software projects suffer from the writers not understanding the use cases? Not eating their own dog food? Well, the newest Pentaho developers have been at that bowl for some time, and the internal developers can help them keep that commitment with internal initiatives delivering Pentaho solution driven information.

And what of the other direction? Those leaving the formal Pentaho realm and working entirely community based? Well, that would be me. It's not like this is new news - I'm now infamous for my off-again, on-again relationship with formal employment :) Don't mistake me for irresponsible; I just have higher priorities. We all should be so blessed, right?



The great news is I also have reaped the benefits of a long series of lessons in BI, big data, analytics, reporting, visualizations, problem solving and code writing. So I take these lessons learned into the community and can begin to give back a little. To my fellow community members, to other open source projects, to Pentaho.

One project that has caught my attention is the OpenMRS project. OpenMRS is a medical records system platform widely deployed throughout the compromised countries of the world. OpenMRS is open source, and has a thriving community of developers, implementers, users and observers from well established world health organizations.

I intend to spend the last quarter of this year investigating integration points between Pentaho tooling and OpenMRS. OpenMRS could use more insight into their data; Pentaho is an excellent set of tools for turning raw data into information. I see synergies here :)

Soon, there will be a project page to stay informed if you're interested or would like to participate. I'll post back as soon as I have the leg work done. In the meantime, checkout http://www.openMRS.org. It's a very rational site that gets you up to speed quickly on the project.

Cheers & all in His grace,
Gretchen

Jimmy D. takes a look at 2010 and where Pentaho is present

I couldn't resist re-posting this link to James' blog - these numbers are sooo exciting! More so for me since I remember when Pentaho was largely comprised of a small rented space and some beanbag chairs :)

James takes a look at 2010 and where Pentaho is present.

Kindest regards,
-G

Troubleshooting Localization

I've been gathering some interesting and useful information when dealing with Pentaho Reporting, Pentaho Metadata and characters not represented in the standard ASCII character set. This bucket of tips will make it into our documentation ASAP, but I thought it prudent to share it with our community even sooner.

IMPORTANT CAVEAT: Note that where I specify UTF-8, I am only doing that as a reference encoding... the encoding I speak of in most cases can represent any extended character set; UTF-8 is a common one for multi-national apps, because it represents multi-national characters.

Character encoding is key to displaying multi-byte or special characters from character sets outside of the standard ASCII character set. Any text-based files that contain special characters in their glyph form must be encoded as at least UTF-8, or in the character encoding for the language you are attempting to display.

The character encoding is significant no matter where these characters reside or travel - if the file or database stores the characters as UTF-8, then Java must handle those characters as UTF-8 and where ever the characters' destination is, be it a browser window or system file, the destination must also render the characters using the same character encoding.

So, this means:

1. Check any and all TXT or CSV files in an appropriate editor to verify that they are encoded in the correct character encoding. In a pinch, Notepad will do, but if you are seriously dealing with localization, it's in your best interest to invest in or download a good unicode text editor.

2. Make sure that your HTML and XML files have a meta tag specifying your chosen as the character set. For example:

<?xml version="1.0" encoding="UTF-8"?>
<meta http-equiv="Content-Type" content="text/html; charset="UTF-8" >

And, if it actually appears in an xhtml document (as suggested by the xml declaration) the content type should probably text/xhtml, and the meta tag should be closed in itself like so:

<meta http-equiv="Content-Type" content="text/xhtml; charset="UTF-8" />

3. The Pentaho BI Server allows you to specify a default encoding in a context parameter in the web.xml file of the webapp. This "default encoding" applies to any XML documents that the server generates. The platform adds an xml prologue to these documents and sets the encoding to that of the BI Platform, which comes from web.xml. By default, the server assumes this is UTF-8. If you want a different default encoding, specify it in the web.xml.

4. You also want to make sure that the default encoding that Java (specifically, the JVM that is running the Pentaho application) is using matches the encoding that the Pentaho application is using. We just mentioned that the default encoding for the Pentaho BI Server is UTF-8. So, what is the default encoding for the JVM? The JVM determines it's encoding from the system property "file.encoding". As of Java 1.4.2, this property is available and set from as the default OS locale. However, on Windows systems, this default locale may not exist, so Java makes a best guess. As you can see, knowing what the default encoding is can be a bit nebulous, so we recommend setting the encoding for Java on the command line:

java -Dfile.encoding=UTF-8

You will want to add this command line parameter to any Pentaho application startup script that you are attempting to use internationally. Specifically for the Pentaho BI Server, you would want to set this command line parameter in the start-pentaho.bat | .sh script.

It's important to note that we don't demand UTF-8. We do (for now) demand that whatever file.encoding is specified is what the web.xml context parameter "encoding" says. So - as long as this param says ISO-8859-1 and file.encoding says ISO-8859-1, you're still good.

Next, understand that common fonts do not have all of the characters possibly represented in the UTF-8 character set or other extended character sets. So, while your encoding may be correct, if you specify a font that doesn't include the glyph for a multi-byte character, it's likely to render as a square, question mark, or some other seemingly unrelated character.

A good test font on Windows systems is "Arial Unicode MS", which is distributed with MS Office and is claimed to have every UTF-8 character glyph available. It's ability to represent every character makes it a good TEST font, but comes with a price - the font is nearly 24 MB. You do not want to recommend this as a production font, since as a best practice guideline, we tell customers to embed their fonts with certain output formats, and this font would equate to staggering overhead in download sizes. The proper recommendation is to tell customers to find the font that best represents the consumer base's languages for that report.

So how do we control which fonts and encodings are used in Pentaho Reports? It's a bucket of valuable information I'm attempting to summarize here:

First encodings:

In Pentaho reports, there are global configuration properties for the different output formats. The global report engine configuration can be found in the Pentaho BI Server installation under the pentaho webapp: pentaho/WEB-INF/classes/classic-engine.properties.

org.pentaho.reporting.engine.classic.core.modules.output.table.html.Encoding=UTF-8
org.pentaho.reporting.engine.classic.core.modules.output.pageable.pdf.Encoding=UTF-8
org.pentaho.reporting.engine.classic.core.modules.output.table.csv.Encoding=UTF-8

And fonts:

1. If you have a metadata model in play, make sure that the metadata concept properties for the font-family are all set to a font that is installed on the server serving up the model and is capable of rendering the special characters you need represented. There is a Base concept (found in the Concept Editor) that has a default font-family that you will want to verify/modify is configured correctly.

2. If you are using any of the templates designed for Report Design Wizard or Web Adhoc Query and Reporting, you will want to verify/modify those templates to use a font that is capable of rendering the special characters you need represented. The templates for Report Design Wizard are found in the Report Designer's /templates directory. The templates for WAQR are found in the Pentaho BI Server solutions directory under pentaho-solutions/system/waqr/templates.

3. On Windows, what determines whether Pentaho can find an installed font? A few things! First, look in the Windows Control Panel (or modern equivalent), under Fonts... these are the fonts that should be available to the reports generated by the Pentaho BI Server. If for some reason you want to include a font not in the system fonts directory, you can add additional directories of fonts.

This is done using a configuration file that you would create and place in the Pentaho webapp WEB-INF/classes directory, which basically creates an override for the configuration file that is found in the libfonts-x.x.x.jar library in the Pentaho webapp primary classpath. The name of the libfont report configuration is libfont.properties. Create this file, place it in the classes directory and add the following configuration property to it, with your font location of course.

org.pentaho.reporting.libraries.fonts.extra-font-dirs.myNewDir=c:/myNewDir/myFonts

Note: There is an open issue with this property that should be fixed with the SUGAR release of the Pentaho BI Server: http://jira.pentaho.com/browse/PRD-2145.

4. Our best practice recommendation for ensuring the proper rendering of special characters in PDF reports is:
a. Embed the font. This can be accomplished using the following global reporting configuration property: org.pentaho.reporting.engine.classic.core.modules.output.pageable.pdf.EmbedFonts=true
b. The font should be a TrueType font.

Also important to note is that you can confirm what fonts the Pentaho BI Server is aware of, as the reporting engine creates a cache of the fonts it has registered. If you are at all concerned that the server hasn't correctly registered a new font from the system, you can blow away the cache, restart the server, and the reporting engine will load all system fonts anew.

The cache exists at $HOME/.pentaho/cache/libfonts.

Pentaho Architect's Bootcamp Training Now Available

Last week, I had the pleasure of offering our very first 5-day session of the Pentaho Architect's Bootcamp training, and overall, it was a big success!

There comes a point in many advanced deployments of the Pentaho BI Suite where some feature or requirement pushes the boundaries of the out-of-the-box product capabilities. Since the beginning of Pentaho time, we've marched to the beat of "make it possible first, then make it pretty/easy", and it's this scenario where our approach pays big dividends to our customers/community/users. Because the platform/server/tools were built for extensibility, there are numerous places where you can roll up your sleeves and leverage a simple API to implement a customization that suits your specific requirements.

The Pentaho Architect's Bootcamp is geared for developers, partners, customers, consultants that are ready to roll up their sleeves and understand the complex problems that are surfacing in large scale BI implementations, and how to extend the Pentaho suite of products to answer far more questions and customizations than the boundaries of the out-of-the-box product.

Here's a sampling of some of the questions that are answered during Architect Bootcamp training:

• How do I integrate my own custom visualizations (maps, charts, gauges, etc) into Pentaho?
• How do I accommodate multiple companies/groups/organizations' data in my solutions, while maintaining each companies/groups/organizations' personal point of view of the data?
  
• How do I dynamically drive row level security in Pentaho Analysis and Pentaho Metadata?
  
• How do I integrate Pentaho solution content into my own application?
  
• How do I customize security across the Pentaho platform and pillars?
  
• How do I create integrated solutions using ETL, reporting, analysis and metadata to deliver my customer's specific solutions?
• How do I plug my custom content into the Pentaho BI Server? How do I then integrate my custom data/functionality with the Pentaho pillars (ETL, reporting, analysis, metadata, etc)?
  

I really believe this course is a game-changer for Pentaho users and solution developers. I teach the course, so don't take just my word for it. Here's some feedback we received from the first course offering:

"We traveled from India to Florida for this course, and we are extremely glad that we did. The information is very valuable."

"The presentation was excellent, [I e]specially liked the interactive nature of the class."

"All of the lessons uncovered new boundaries; all topics had more priority!"

"YEEEEAAAA.. just built my first Pentaho BI Server plugin... Pentaho Architect's Bootcamp RULES!!! #Pentaho @Pentaho"

"Day 4 Pentaho Architect's Bootcamp brought so many excitements, I loved it. even though my brain feels like it's going to explode!! #Pentaho"