Mashing up RDF data with WSO2 Mashup Server

Okey so this is the fun part that I promised to write about . I managed to cook up a use-case to demonstrate RDF querying and making use of the semantic data. The data that I am using for querying, is the rdf data sources available in the UK data.gov site. With some analysis I [...]

Okey so this is the fun part that I promised to write about :D . I managed to cook up a use-case to demonstrate RDF querying and making use of the semantic data. The data that I am using for querying, is the rdf data sources available in the UK data.gov site. With some analysis I figured out that this task can be fundamentally archived using the combination of Mashup and Gadget Technologies. My choice of tools were WSO2 Mashup Server and WSO2 Gadget Server for their great flexibility and of cause for other obvious reasons :D . However the Mashup Server does not natively support RDF data retrieval, hence I had to do some work to get such functionality integrated. The great fact about the mashup server is its extensibility, the concept of host objects and the ability to write custom host objects and its pluggable nature comes handy in such cases. The high level architecture of what I am trying to achieve is as follows.

RDF data retrival with WSO2 Mashup server / WSO2 Gadget Server

To implement the above architecture with the tools at hand I created a custom host object that can be plugged to the Mashup Server. When dealing with semantic web related tasks and RDF data handling HP’s Jena java library comes in handy. With the use of Jena-ARQ (for SPARQL) api I managed to get the host object working with few lines of code.

.....
            Dataset dataSet = DatasetFactory.create(sparqlObject.rdfDataSource);
            // Create a new query form a given user query
            String queryString = sparqlObject.spaqrlQuery;
            Query query = QueryFactory.create(queryString);
            QueryExecution qe = QueryExecutionFactory.create(query, dataSet);
            ResultSet results = qe.execSelect();
.....
           resultString = ResultSetFormatter.asXMLString(results);
..... OR.....
           ByteArrayOutputStream bos = new ByteArrayOutputStream();
           ResultSetFormatter.outputAsJSON(bos, results);

With the host object in place, the next task was to create a Mashup in-order to query the rdf data with a given source (EndPoint or data source). The javascript service (Mashup) is created to serve this purpose, where the consumer can specify the RDF endpoint or the data source with the SPARQL query and retrieve the dataset in XML or JSON.

.....
function RdfDocQueryService(rdfDataSource, rdfQuery, resultType) {
   var sparqlObj = new SparqlHostObject();
   sparqlObj.rdfDataSource = rdfDataSource;
   sparqlObj.spaqrlQuery = rdfQuery;
   sparqlObj.resultType = resultType;
   return new XML(sparqlObj.getDataFromRdfSource());
}

Finally to bind everything together, lets try querying some data. My example usecase is to use the query at N2 blog to retrieve traffic monitoring points in UK roads. The query to retrieve the data set as follows,

#List the uri, latitude and longitude for road traffic monitoring points on the M5
PREFIX road:
PREFIX rdf:
PREFIX geo:
PREFIX wgs84:
PREFIX xsd:
SELECT ?point ?lat ?long WHERE {
  ?x a road:Road.
  ?x road:number "A4"^^xsd:NCName.
  ?x geo:point ?point.
  ?point wgs84:lat ?lat.
  ?point wgs84:long ?long.
}

To visualize these points I have created a gadget with the aid of Google Maps api. This gadget can be hosted in the Gadget Server, where it can dynamically retrieve traffic monitoring points for each road in the UK and display them in the map as follows.

Traffic points in A4 road, UK

The Semantic Web

For the past week or so I was able to work with data.gov.uk to find a possibility to make use of government/public data for a nice sample gadget, that will work in the Gadget Server. However it is quite disappointing to see most of these data are in lengthy PDF files, and without some expensive [...]

For the past week or so I was able to work with data.gov.uk to find a possibility to make use of government/public data for a nice sample gadget, that will work in the Gadget Server. However it is quite disappointing to see most of these data are in lengthy PDF files, and without some expensive processing these data could not be consumed easily. The happy scenarios in my mind was, that there will be a bunch of RESTfull web services where I can request for public data and in return I will get the result set in XML or (and if am really lucky) in JSON ;)

After some browsing over the data in the UK data site, There was one interesting area which is called SPARQL and that was my moment of joy :D . there were six government data endpoints which I can query against and the result set can be retrieved as XML or JSON. This was the ideal scenario, even the available data is limited, there it starts the story of the Semantic Web.

When I was reading for my bachelor’s thesis, under service orchestration and chirography, the concepts of semantic web (services) were discussed in length. However the idea is still in its early stages, and there is still no real business use-case for it. The very purpose of semantic web is that the resources of the WEB and there relationships can be self explained and explored.

Self Explainable RDF graph

I have a dream for the Web [in which computers] become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A ‘Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialize.

Tim Berners-Lee, 1999

The UK data site has offered chunks of eduction, transport, financial, environmental, analytical and statutory notices as RDF data endpoints which can be queried using SPARQL (SPARQL Protocol and RDF Query Language).

So why let this great opportunity to lay dry, my idea was simply to use these endpoints and query these data and display them in some gadgets, for instance with transport data we can draw traffic details in maps, or with financial and educational data we can create some graphs and charts.

But with some thought, WHY NOT go for mashing up, We have a great tool (WSO2 Mashup Server) just made for that. So I was cutting code for the past whole week to create a custom host object in the Mashup Server to query RDF (RDF Service endpoints or RDF Data sources) data, and am almost done, the host object is created, two mashup services are created (Query a RDF endpoint/and RDF data source) now its a matter of integrating the services with Google Maps API and visualize them as a gadget in the Gadget Server.

So yeah the good stuff will be in the next post. keep tuned :D

Shopping with ma lady :-)

I had to blog about it window.fbAsyncInit = function() { FB.init({appId: "108492862525832", status: true, cookie: true, xfbml: true}); }; (function() { var e = document.createElement("script"); e.async = true; e.src = document.location.protocol + "//connect.facebook.net/en_US/all.js"; document.getElementById("fb-root").appendChild(e); }());

I had to blog about it ;)

WSO2 Gadget Server 1.1.0, What to expect

WSO2 Carbon 3.0.0 – code name “Iridium” is just about to release in few more days. as of WSO2′s release strategy, all the products will graduate with their next version on top of carbon based platform. As for the newly released Gadget Server it would be version 1.1.0 Features were frozen for version 1.1.0 and [...]

WSO2 Carbon 3.0.0 – code name “Iridium” is just about to release in few more days. as of WSO2′s release strategy, all the products will graduate with their next version on top of carbon based platform. As for the newly released Gadget Server it would be version 1.1.0

Features were frozen for version 1.1.0 and that was in the end of February as I recall. We (the GS team) managed to squeeze in few new very important features to this release.

  • Upgraded shindig to the latest version

This was a bummer, since WSO2 Carbon platform is running on OSGi, if you had to use a non native OSGi project (i.e. Shindig) you will have to create an OSGi aware carbon orbit bundle and make use of it at run time. This was done some time back where shindig was on r734876 revision. In-order to leverage new functionalities such as OAuth, Pub-Sub etc. and to patch the bug fixes we thought of updating the shindig carbon orbit bundle. So now shindig is on r910768 and is quite up to date.

  • i18n internationalization support for gadgets

i18n is not a big deal for Google gadgets since the gadget API itself supports it, What we had to do is enable i18n support in shindig. (Which was already implemented by the shindig community)

  • Inter Gadget communication

Inter gadget communication, seemed to be the hot topic in our forums and webinars, all most in all tech talks we did about the Gadget Server, some person in the audience raised the question about gadget-to-gadget communication. Our answer was “it can be done at shindig level, and we are yet to support it”. So without a long await, with GS v1.1.0 we enabled this feature. The architecture is quite fascinating where there will be zero backend calls and all the communication is done simply on the front-end. Basically each publisher will have a publishing channel and the subscribers can subscribe to this channel. After that its simple pub-sub.

  • The portal will completely run on HTTP transport

This was a limitation we had in our fist release. The portal was running only on HTTPS, and the reason behind was that all WSO2 products are running on secure transport and the Gadget Server is also a combination of some specific components which ran on the same platform. For this release we went the extra mile and enabled HTTP for the portal. So once you go to the login page it will switch to the secure transport (HTTPS) and after successful login you will redirect back to the HTTP non-secure mode. (Of cause you can disable HTTP any time and run the portal purely on HTTPS, it all depends on the requirement.)

One disappointment I have is that we couldn’t integrate OAuth on time. even though it is supported at shindig level we have to do a considerable amount of work from the Gadget Server side to fully support it, hence it is postponed to our next release. You can try the sample OAuth GData Gadget, simply adding it by the URL, and that will work like a charm. (nothing useful but just to let you know that we are only few steps behind on it)

Okey so enough sneak peak ;) download the Gadget Server pre-beta, play with it, and help us to improve (even in the last minute) by reporting any issues.

Cheers !!

Reality of virtual trade

Internet is rushed with the term “virtual” these days. Every business strategy, every service and a software has the virtual term bind to it. It had come to a point  that we use several operating systems in the same desktop box using virtualization, and that’s what I know about the term “virtual”. But this post [...]

Internet is rushed with the term “virtual” these days. Every business strategy, every service and a software has the virtual term bind to it. It had come to a point  that we use several operating systems in the same desktop box using virtualization, and that’s what I know about the term “virtual”. But this post is not about Xen/VMWare virtualization nor application hosting on virtual private servers (VPS) it is more in to second life, yahoo’s avatar world or currently dead lively.

As we all know the harsh truth is that Business drives IT, but I like to believe that virtual worlds have created new avenues to do better and profitable business and not vice-versa. This post is dedicated to one such research, which was done by one of my collogues for her bachelors thesis. The aim of the project was to research on creating sustainable business environments in virtual worlds. Her test beds were mainly the popular public virtual worlds such as second life, Google lively and avatar city. The thesis contains alot of valuable data, for example the user density in these worlds, what are the brands who is actively doing business in virtual worlds, their success stories and if not how they failed, creative strategies to do business in such environments etc. Also as the research conclusions the thesis elaborate a set of best practises and a business model to conduct sustainable business in these platforms.

Distribution of Real World Brands

Distribution of Real World Brands in SL by 2007 (KZero 2007)

SL User Base (Linden Research, Inc 2009)

Second Life User Base (Linden Research, Inc 2009)

Even though, we do not see massive marketing campaigns or  business growth in these platforms yet, it seems that these virtual realities has alot to offer, for instance at WSO2 we can conduct a series of webnars in second life and we can ask our users to subscribe as avatars, I believe that would be a great experience. :)

Any how you might also get a slight chill or goose bumps, if you have already watched few related movies like Gamer (2009) or Hardwired (2009), as every technology has its dark side, I believe if we focus on its promising potential, definitely virtual worlds would be the future.

More details about the research and the thesis can be found at http://bit.ly/a4wr9W

Time for family and friends, Time to party and time to relax :D

Today I got holidays. yeah I know some of you might be wondering whether am still schooling (Yeah I got the same feeling first lol) !!! !@#$%^& But yeah we got a super awesome two whole weeks to have some fun, go some where, anywhere out from checking mail, and doing the stuff we do [...]

Today I got holidays. :D yeah I know some of you might be wondering whether am still schooling (Yeah I got the same feeling first lol) !!! !@#$%^& But yeah we got a super awesome two whole weeks to have some fun, go some where, anywhere out from checking mail, and doing the stuff we do day to day. Had the last-office-day party at flag and whistle dressed up like pirates, doing some crazy games and enjoying nice food. Dr. Sanjiva gave an inspiring speech, talked about what we achived as a team at WSO2 in 2009 and what we should do in the coming year.

I am still 5 months old at WSO2 and all these things inspires me, the friendly open culture (with a very Geeky backdrop), the challenging work, friendly people with an amazing passion and team work. All those make me think how great it is to be apart of it all. So now its holidays for us, till Jan 4th we take a break from work and enjoy with everything else but work. My plan is to sleep about two days, Going on a trip to North (Yeah north since there NO WAR any more) for like 3 ~ 4 days, visit some old friends and la..di..da..di..da… :) So yeah until then HAPPY HOLIDAYS

WSO2 Gadget Server is out… Download it !! Play with it !!

Yesterday (16th Dec) WSO2 Gadget Server graduated from its beta status and announced its release. WSO2 Gadget Server is designed to serve as a presentation middle-ware product in the SOA space to smoothly display chunks of service oriented data for the end users. The solution architecture is based on portal / portlet concept but making [...]

Yesterday (16th Dec) WSO2 Gadget Server graduated from its beta status and announced its release. WSO2 Gadget Server is designed to serve as a presentation middle-ware product in the SOA space to smoothly display chunks of service oriented data for the end users.

The solution architecture is based on portal / portlet concept but making it far more simpler. Since the enterprises are more and more leaning towards the cloud and service oriented mashups, visualizing those data should not be complicated. Hence the Gadget Server provides a simple platform to write the data visualization code just in HTML, JavaScript and XML the implementation of the presentation logic cannot make more simpler. It is exactly similar to writing a Google gadget (hosted in iGoogle / Gmail / orkut) to Tweet ;) .

As far as it goes the Gadget Server’s applicability for the enterprise is somewhat an enterprise dashboard that can be customized according to the user’s need and governed by the authorities. For an instance if you are a manager of a bank, wouldn’t it be great to have a dashboard forecasting and displaying current and future bank transaction stats and predications, while at the same time in a deferent view having your business schedule, calender, mail/IM, and news as small but clear and interactive portlets.

Yeah so thats, what the Gadget Server does, and the interesting fact is, its simplicity and extensibility, What all you need to know is some HTML and JavaScript. (no need to consult Java / .NET / PHP / SOA experts). So Download it !! Play with it !! Give us some feedback !!

Authoring, deploying and using XML Gadgets in WSO2 Gadget Server

We are about to release The long awaited WSO2 Gadget Server within few days of time. These few days I was doing some documentation and content writing about the Gadget Server, Apache Shindig and Google gadgets specification. My 1st article about authoring gadgets is now published on WSO2 Oxygen Tank as a help/Tutorial for Gadget [...]

We are about to release The long awaited WSO2 Gadget Server within few days of time. These few days I was doing some documentation and content writing about the Gadget Server, Apache Shindig and Google gadgets specification. My 1st article about authoring gadgets is now published on WSO2 Oxygen Tank as a help/Tutorial for Gadget server users. You can also download the Gadget Server release candidate 2 and play with it. Also Paul had written an interesting article about portals and Gadgets Server’s role.

Apache Shindig talk at Apache Con Asia 09

As I have mentioned in some previous post, today I did a talk about Apache shindig and its use cases (WSO2 Gadget Server) at Apache Con 09 with my Colleague Tyrell. So without writing about the whole talk let me share the presentation slides. The Talk turned up great, audience were quite interested, too bad [...]

As I have mentioned in some previous post, today I did a talk about Apache shindig and its use cases (WSO2 Gadget Server) at Apache Con 09 with my Colleague Tyrell. So without writing about the whole talk let me share the presentation slides. The Talk turned up great, audience were quite interested, too bad we had very limited time and missed to do a demo. :(

View more documents from Nuwan Bandara.

Apache Asia Road Show 2009 >> Keynote speech

Today I was at Apache Asia road show 2009 at Colombo, and feeling good about myself of attending . It was a great session with many valuable tech talks and more of a valuable community gathering. Am not writing this post to talk about Apache and all the open source movement but about a specific [...]

Today I was at Apache Asia road show 2009 at Colombo, and feeling good about myself of attending :) . It was a great session with many valuable tech talks and more of a valuable community gathering. Am not writing this post to talk about Apache and all the open source movement but about a specific talk which made me thinking.

Today, the 1st of 3 days event, the keynote speech was given by a remarkable Sri Lankan Scientist Prof Mohan Munasinghe on a topic if I remember correctly, “Making Development More Sustainable Will Help Address Climate Change“. I know !! It doesn’t sound much like open source or rather ICT in that matter, but the way he made most of the things related to ICT was quite remarkable. His solution to address bigger problems seems simple. His solution was to address the small problems, change the things what you can change at no cost, exactly quoting “Use electricity carefully, eat less meet :) ”.

Thinking it through, its the old method in solving the big problem, its just break it in to small problems and try to solve those small ones. The facts Prof Munasinghe talked about climate change was quite terrifying. But however his conclusion was optimistic, he believed as a community that we will stand up against these challenges, be less greedy and help the poor.

P.S : If anyone of you did find the presentation slides of the keynote speech please do link in comments :)