What Should be Next for Enterprise Mashups?

My friend over at JackBe, Chris Warner, recently sent me an email asking me what the next 2 to 3 years should hold for Enterprise Mashups. JackBe, in case you don't know, is the creator of the Enterprise Mashup software Presto.  We've used Presto on a project and it's a wonderfully crafted piece of software with many, many possibilities in front of it.  Presto has, at its heart, the creation of mashups, those elusive but so useful beasts which give you all kinds of new-fangled views into your data.

This is a topic I've spent a little bit of time thinking about, fancying myself to be at least a passable user and creator of mashups.  I'm going to change up the question a bit, though, and delve into territory that is slightly afield.  So, I propose these related questions:  Just what the heck are Enterprise Mashups?  And, what's next for them?

Any Ol' Programmer
Let's be honest, any programmer can get out there and create a mashup with a little elbow grease.  The technology is there such that, with a little bit of web service lifting and a little bit of UI know-how you can create fantastical mashups that will make everyone sit up and take notice.  What's more, if you do it right, your own mashup can, in turn, become a data source for someone else to consume.  It's a bit like Sir Elton John says, that whole Circle of Life thing.

But, that's just the problem, isn't it?  We're not all programmers, now, are we?  Not all of us even have access to programmers (or, as we like to style ourselves now-a-days, "software engineers"), so it's not like we can ask someone else to do it for us.  And, most important of all, programmer time is expensive.  If you do have access to a programmer and task them with spending hours creating your pet mashup, you're likely to get a whack on the wrists from the company dumby stick.  So, what use are mashups to the common-day man?  They're nice to look at, but not really practical beyond that.

Enter the Enterprise
No, not the ship, though that would be cool.  The Enterprise is (or should be) all about putting a useful tool into the hands of someone other than an engineer.  To a certain extent it's about lowering the technical barrier of entry.  Any programmer should be able to put a mashup together, but that doesn't mean that they are easy.  There is a lot of wiring together and hacking things up that takes place.

Enter the folks at JackBe, handing us Presto.  With software like Presto it's possible for someone of a less technical bent, someone that's not a programmer, to put together a mashup that's useful.  This Enterprise tool takes some of the difficulty out of the creation.  And that, in a nutshell, is what the Enterprise should be about.

What's Next?
Even though products like Presto lower the barrier of entry, there is still a long way to go.  Presto shines at doing the actual mashing, wiring the data together to get the new view.  But, one area where Enterprise Mashup software needs to tread next is into more robust presentation of that new view.

One thing that mashups excel at is the creation of data.  One thing that data desperately wants to do is be visualized.  There are so many tools out there for visualizing data, but most of them have a similar dilemma to non-enterprise mashups:  it takes specialized skills to use them.  Enterprise Mashups need to tread here, next, lowering the barrier of entry into rich presentation of the data that mashups are so good at generating.

To Infinity...
From there, well, who knows.  I'd like to see tools which are good at creating not just a good presentation but actual interaction, all done so that the common layman can create them with minimal know-how.  I've blogged before about how it's important for people like analysts to be able to do at least some kind of rudimentary software development.  I see that Enterprise Mashups can fill this role once they come to full maturity.  It's a big area, one that needs more exploration, and I'm quite happy that I get a peek into it at the ground floor.

An Analyst's Development Environment

Here in the land of academic research we're working with a "new" take on mashups. It seems like a no-brainer to me but a lot of people have expressed interest and surprise when I explain to them what we're doing. For now let's call it an analyst's development environment (ADE).

One thing that mashups are really, really good at is taking disparate data sources and allowing "momentary" relationships in the sources to be created. This in effect creates a new data source that is a fusion of the inputs. As is often the case in fusions, this new source tends to be more than just the sum of the parts. You often come up with new views on the data as you add extra sources.

Most people stop here at the fusion stage. Once they have the new view onto the data they rely on other tools outside the scope of a mashup to do interesting things. They might pipe that data into a tool such as Fusion Charts in order to visualize it or they might pipe it into an analysis tool such as a model or sim. But, why do they need to leave the scope of the mashup to do this? What if that analysis or the creation of the Fusion Charts XML was an automated part of the mashup itself?

Mashups deal with web services primarily (though there are some nifty products out there that allow you to mash more than just web services). A web service is usually considered to be a data source. But, in practice they are much more than that. Consider all of the specialized web services provided by Google for geolocation or Amazon for looking up aspects of books. The simplest example I can give you is Google's web service which converts an address to a lat and long pair (called geocoding). With these in mind let's take a different look at web services. Let's look at them as processing units.

A processing unit has 3 criteria: it takes input; does something interesting with that input; and provides output. Processing units are the basis of modern programming. They're known as methods, functions, procedures, etc. depending on context. We can most often build bigger processing units from simpler units.

Web services fit these 3 criteria handily. You can easily provide input, they can easily do something interesting with that input and then just as easily provide output. All communication is done in a standardized protocol driven environment.

The interesting thing about web services is that we can string them together (with the right tools) rather easily into processes. That's exactly what we're doing here. Each web service is either a data source or a processing unit. Given the ability to ferry data from one web service to the next (in an easy way) it is possible to create mashups that do more than just mash data. They actually do some form of processing.

Consider what it would be like if you had a web service endpoint attached to a model? You could pre-mash your data from various sources then run it all through the model and create a new output that would be very interesting. It would be so easy.

Using Presto we recently put together a demo which worked along these lines. It made our demo come together in several weeks rather than over several months. We used Presto to access databases then ferried that data (in XML format) into a custom built web service that took said data and ran XSL transforms on it. That produced Fusion Charts XML which we then piped into our presentation layer for visualization. It was easy.

Here is a diagram of what the actual flow of the mashup was.

Here is a screen shot of the actual chart produced by the generated Fusion Charts XML.


An ADE would work in a similar way. Using provided tools which allow for ferrying of data from one endpoint to another and given a grab-bag of analysis and transformation web services an analyst could create some amazing things with little effort or technical know-how. The only developer support would be in the creation of any custom web services. It could be a very powerful tool.

It's a Transforming Process!

So, right now I'm working on a gadget that takes in generic info and sends out Fusion Charts XML. It's a SOAP service and there will be many service endpoints, but right now there are only 2, one for a simple, single series bar chart, and one for a multi-series "drag node chart" (think network diagram with drag able nodes).

I chose to go about it in a different manner than I've seen a lot of people use for Fusion Charts, though. The prevailing way that I've seen people create Fusion Charts XML is to take the data in on the JavaScript side and create the XML, in string format, in the JavaScript. For this approach, I have only one thing to say: Building XML in JavaScript is less than optimal (translation: it sucks).

So, I decided to go about it in the web service itself. My web service is written in Java and once you get the question into Java a few, more palatable, alternatives suggest themselves. In my web service there are 3 distinct transformations: request object to traditional object; traditional object to simplified XML; simplified XML to Fusion Charts XML.

The request object to traditional object transformation is really the beast. The inputs for the endpoints are comma-delimited strings. A lot of work goes into parsing those strings and putting them into the more traditional object. I have my inputs be comma-delimited strings so that the Presto JUMP requests can invoke them effectively. I could just as easily have one of my endpoints be a direct invocation of the more traditional object, but as I understand it, that's a bit of a bad practice.

Once I have my traditional object the easiest step occurs. In this step I use XStream to serialize the traditional object into a simplified XML. If you've never used XStream, it's very simple, very powerful and I recommend it highly.

The last step is where the real magic happens, though. Here is where I transform the simplified XML into Fusion Charts XML. I use the Saxonica XSLT engine to do the transformation and it's a matter of using the right tool for the right job (with regards to using XSLT to transform XML).

XSLT is designed to transform XML, whether it be from XML to XML or XML to some other language. You write a transformation wherein you process the source XML and then create a document in the desired format. It's really not all that hard to take the simplified XML and transform it into the Fusion Charts XML.

Once I have the Fusion Charts XML document I send it back out of the service in a special response that contains the document in string format and the name of the Fusion Chart swf file that will correctly process that document. When my response arrives at its destination all that needs to be done is input the Fusion Charts XML document into the Flash engine with the correct swf file pulled up and voila, it's all done.

I like this approach in that it moves all of the heavy lifting out of the display side (the mashlet, in my case) and into a much more suitable environment, that being a Java web service. I don't have to do endless string concatenation that is hard to debug inside of the JavaScript presentation layer. As a matter of fact, I can write all of the pieces independently of each other and then put them all together in the end. It's a nice break up of all of the work.

An acknowledgment needs to go to @angleofsight for his help in getting this whole process set up. Without his paving of the way I wouldn't be anywhere near as far along as I am right now.

How Does the Magician Get What He Wants?

As I started developing the core of the "platform" I realized that I had undertaken a huge task, a task I was not sure I could finish in any short amount of time. This caused me some angst but I figured that I would just have to move on, doing my best to develop my way out of an impossible situation. Not a fun position to be in.

Then, sometime around the end of December, a buddy of mine pointed me in the direction of a product called Presto, developed by the company JackBe. I thought that I had stumbled into my own brain on the web!

Keep in mind that the concept I was working on was that the platform would map gadgets together in such a way that we could dynamically generate them with ease, rapidly prototyping processes to see how they worked. In effect, the gadgets would be web services and the thing the platform would create would be a mashup.

As I said, I had envisioned something similar to PopFly, the ability to visually create mashups and then do something meaningful with them in our efforts to prototype processes. Presto is this and so much more.

First and foremost, Presto has a beautiful visual mashup maker called Wires. In Wires, you drag services from a palette onto your canvas and connect them. You can also drag in actions, blocks that allow you to do something with the results of other services. So, in the basic example I worked with, you have two RSS feeds. You can take the data from each feed and "merge" them (where merge is an action block) so that you effectively have one RSS feed from two. Then, you can add a filter (another action) which can take a dynamic input. In this way, you can create very complex mashups from some very basic building blocks.

If the complexity of Wires is not enough for you, however, they have a markup language for creating mashups called EMML. What you do in Wires is distilled down to EMML, but Wires doesn't offer the full capabilities of EMML (which is not to say that Wires is not fully featured, there are just some rather tricky things you can do with EMML that you can't do with Wires as they don't have a visual representation).

What's even cooler, and something that spoke to a need we had, is that each mashup is published as a service itself. So, you can include mashups in other mashups! The modularity is great.

But, you might be asking, how do you get the services in so that you can use them from the palette?

Well, Presto has a "Service Explorer" which allows you to import services from wherever they may be and "publish" them in the Presto server. However, there's a little more to it than that. Presto comes complete with a user authentication system that can be standalone or hook into LDAP or AD. When I import a service, I can assign rights to it, and only those in the appropriate groups can even see the service, or any mashup in which the service exists. You can also assign rights to mashups themselves.

So, that gave us the mashing capability that we've been looking for, but the goodies in this bag didn't end there!

What we have up to this point is a unique view of the data, but no visualization of the data. Enter the mashlet!

A mashlet is a view of the mashup in a portable and embeddable package. It is created in JavaScript. Once you've created a mashup, you can attach a mashlet to it so that others can see what the mashup provides. There are 5 prebuilt mashlet types: RSS; grid; chart; Yahoo Map; and XML. If your data fits into any of these predefined views, creating a mashlet is as simple as selecting the mashup or service, selecting the view, then publishing it. If you need a more complex view of your data you can create a mashlet by hand. The process for creating a mashlet by hand is rather well thought out and not all that hard to grasp.

Once a mashlet is published you can do one of several things with it.

The mashlets are served up from the Presto server in much the same way that any JavaScript object is. Right out of the gate, you can view a mashlet standalone, if you so choose. However, the real fun comes when you realize that you can embed mashlets into any HTML page you wish by simply including a script tag. You can also embed them in a MediaWiki, NetVibes or as a GoogleGadget. You can't ask for more flexibility. From what I understand, there are more embeddable objects coming, including things such as JSR-168 portlets.

So, from our perspective, Presto offers us 3 incredible capabilities: the ability to capture services from across the web; the ability to create mashups in an easy and visual manner; and the capability to add a face to the services and mashups we create, then embed that face wherever we need.

We've committed to Presto as the core of our platform and it puts us months, if not years, ahead of where we were.