Skip to content

Gadgets On the Cloud

2010 June 3

There is no doubt that JavaScript/XML gadgets make a great presentation layer over the web with increasing amount of data floating around. The ability of which these gadgets can be embedded in any place over the web, provides a great flexibility, and a wider reach. Google does this quite nicely with their iGoogle gadgets, enabling the gadgets to be embedded in almost any web page. The success of this great idea, would be only logical if all the data, services and mashups are also available over the web with open access or maybe authenticated access. This is where a cloud story fits-in, and this the very reason why Google can do it quite easily.

However, what if you want to do everything from the scratch and also provide a great presentation layer for the users. For an instance, lets say you have a lot of financial data within your enterprise, and you need to provide some of these to your customers, to general public and some for your employees. To do this, you will have to create appropriate data services, maybe mediate or transform some data, integrate with some legacy data sources, create some business work flows, mashup them with some 3rd party services like Google finance or charts and finally expose the end results to the targeted user group. This is where WSO2 Stratos PaaS comes for your rescue :)

If your requirements are such, you will need a strong middle-ware platform to full fill all the above tasks, and if its all on the cloud, you will not have to worry about anything other than writing your business logic. Once the business logic is correctly compiled, you can Mashup some of your data with external service APIs, and then write the presentation logic purely on javascript and xml as XML Gadgets and expose them to the users you need. Once the gadgets are published on WSO2 Cloud Gadget Server its just a matter of linking them up in any web page you want over the web.




The above two gadgets are taken from WSO2 Cloud Gadget Server and have linked in to this blog, to convince about the great flexibility and reach it can add-up. You do not need to use the Cloud Gadget Portal as the only place for your data to be presented (Of cause if you are not using other gadget server specific privileges such as inter-gadget communication etc). You can simply use the Gadget Server as your own gadget repository, and encourage users to discover the gadgets and embed them into their own web pages over the web.

To sum up the story I would say, try-out Stratos, try out the available services and you will definitely find out more use cases, and creative ways to use the platform and leverage the advantages of the cloud

WSO2 Stratos – A true cloud story

2010 June 2
by Nuwan Bandara
Stratos Services

Yesterday (1st of June), A little over a 12 developer team at WSO2, took a great middle-ware platform in to the cloud. It’s not just putting all our server products on an EC2 instance, but embedding all cloud-native features into them. The PaaS (Platform as a Service) is named as WSO2 Stratos, which is based on award winning WSO2 Carbon middle-ware platform. As the Alpha-1 release, Stratos offers number of WSO2 products integrated, namely Governance Registry (GREG), Identity Server (IS), Business Activity Monitor (BAM), Mashup Server (MS) and WSO2 Gadget Server (GS).

Stratos is also offered as a downloadable version for the private cloud within your enterprise. If you are quite serious about using SOA for your enterprise and do not need to worry about deployment, scalability and server maintenance, Stratos would be the ideal solution for you.

Vesak @ Bandos’

2010 May 28
by Nuwan Bandara

Linux for human beings

2010 May 17
by Nuwan Bandara

This is a post that should have been written few days back, On April 29th Ubuntu 10.04 – (Lucid Lynx) was released, and I was counting days till it did but couldn’t try it out because of the release work I was doing in my machine. Last weekend (even the work wasn’t quite over) my anxiousness couldn’t rest. I installed the new beast keeping my /home safe. To tell you the truth, the installation process was hardly “linux like”. There were no screens showing any commands executing, nor view of the terminal. What you see is a breath taking slide show (very much windows like, but much sweeter) trying to show off the beauty ;) and market itself.

Ubuntu - 10.04 (Lucid Lynx)

Well I know many of you already have tried Lucid, and there is nothing new for me to mention, hence I thought of writing few words about Ubuntu’s slogan “Linux For Human Beings”. My history as a Linux user is not that ancient, even though I have used it side by side with a  Windows OS, I’ve never gone total Linux, There were reasons behind. {1} I was afraid it will all break down in the middle of something. {2} It looked like a dark cave with alot of never ending tunnels {3} It was hard for me to troubleshoot on my own (I was a novice). But thanks to Sandaruwan and the never ending virus problems, performance degradation with time in windows, I jumped in to the deep-end. Ever since am a happy man ;)

Before Ubuntu, I have tried Debian and Suse, but with Ubuntu I felt quite safe. There were number of reasons, among them, Canonical released frequent updates and bug fixes (mainly fixes related to hardware drivers). Ubuntu’s is doing major release twice a year with a considerable amount of improvements, and nevertheless its Virus free, ultra fast and very stable for a software development environment.

Being those my reasons, I believe Linux is the answer for many 3rd world countries, to obtain a high IT literacy rate. When Microsoft and other proprietary software vendors are investing in millions and billions to put a full-stop to software piracy (Which as a software engineer I encourage), and when personal computers are sold with these proprietary software pre installed (Ofcause not for free) with prices automatically goes sky high, resulting a normal user to look at a computer as if it is a Jet plane ;) .

But for some (many) reason free and open source software is hidden to the normal user world. Of-cause there are few myths associated, One popular myth is that Open source and free software doesn’t come up to the standard of proprietary software, If we forget for an instance that I work for a 100% open source company ;) , and look at Apache software foundation, where there are plenty of great products (Apache HTTP server, Tomcat, Maven, Synapse et al.), but normally the argument is “A normal user doesn’t care about the server space. What does free software offer for them ?, to listen to music, edit some photos, check email and browse internet” ? And as far as I see this is where Ubuntu places itself. I believe this is why it displays a pretty screen while its being installed and I think so far Ubuntu had done a great Job, and it is ready with a strong and shiny armour, to battle with any proprietary operating system and make it’s mark.

So if you are a normal user reading this article, try new Ubuntu, and when you are buying your next personal computer, ask you computer supplier to give the machine with Ubuntu. I promises you, you will save a hell a lot of money, and it will set you free. :)

Sharing HTTPS, HTTP sessions in tomcat hosted web-apps

2010 May 7
by Nuwan Bandara

The requirement is to only serve the login page securely and once the user is authenticated (s)he should be redirected to non-secure http mode. I was struggling to do this quite some time back, and just thought of documenting about it.

The requirement

The idea I had was; “It should be quite simple”, Facebook does that, Google does that and why is it still not well documented ?, However the almost all Google search results for my queries were about simply redirecting HTTP traffic to HTTPS for certain URLs, some were using URL rewriting (mod_rewrite), and some have used server configuration via Tomcat’s server.xml.

What I really wanted to achieve is to preserve the state between the protocol switch. After some considerable amount of searching I found out this is not achievable (in a very clean manner) with tomcat or rather it is a conflict between security and state management in the servlet spec itself, hence there only exist a dirty hack (not sure if this works) to get it done, but even that hack couldn’t be applied to my scenario.

So after some thinking I came up with my own hack (I think its even dirtier ;) ) to solve the issue; Its quite simple, and involves cookie manipulation. My approach was simply read the HTTPS cookie and set it as the HTTP cookie, what I need was one jsp which is served with HTTPS and few lines of Java code.

The solution

The solution

True enough it certainly looks like a hack, but security wise its as same as the Tomcat user group has suggested. so until the new servlet specification answers this question we have to live with this. the code of converting the cookies are as follows.

    Cookie[] cookies = request.getCookies();
    String sessionId;
    if (cookies != null) {
        for (Cookie c : cookies) {
            if (c.getName().equals("JSESSIONID")) {
                sessionId = c.getValue();
            }
        }
    }

    Cookie k = new Cookie("JSESSIONID", sessionId);
    k.setPath(request.getContextPath());
    response.addCookie(k);

Basically what the code does is, reading the secure cookies while inside the middle.jsp and setting them without security (k.setSecure() is not mentioned hence by default its false), and that’s about it, once this is done you can simply redirect to the HTTP page.

response.sendRedirect("http://foo.com:8080/index.jsp");

and now the cookie which originally set via HTTPS is accessible to the HTTP requests, hence the session is shared.

Web Scraping & Parsing HTML to XML in Javascript

2010 April 28

Today I was working on a customer POC and happened to create few Google gadgets to visualize selected data sets from *.gov.uk sites. The scenario which is implemented was, mixed with inter-gadget communication and content search over data.gov.uk sites. I created three simple gadgets which communicates with each other, and one acted as the controlling gadget which pushed the search parameters to other two gadgets. The two content gadgets showed UK (1) primary school information and (2) electoral information. The pushed parameter was the postal code of different parts of UK. The direct.gov.uk has a form based implementation of this.

The Requirements for the POC was, simple and we already had working samples of such a scenario at WSO2 library.

  1. Show how one gadget can pass the context to other gadgets
  2. How gadgets can harvest data in various formats (in my previous post I explained on how to get data from RDF endpoints, which are also available in *.gov.uk sites)

The building blocks for the implementation was the search url, which was quite straight forward. for all the requests based on postal codes the direct.gov site served in the same manner (because of this important fact, the automation process became trivial). for an instance the url for primary school information retrial was,

http://local.direct.gov.uk/LDGRedirect/LocationSearch.do?LGSL=13&searchtype=1&LGIL=8&Style=&formsub=t&text=SE1+7DU

Where the param “text” changed according to the postal code. So far everything seemed straight forward, however at implementation, while using Gadgets API for content retrial, I faced problems in parsing text with javascript. Hence the gadgets.io.makeRequest supported HTML as text and the API method returned the retrieved HTML document as string making it quite impossible to process.

With some thinking and advise, I brought the Mashup Server in to the picture and used it to retrieve the data from the gov site and returned the result in XML format. Using the Mashup Server web scraping seems to be a piece of cake, We created a simple mashup using the scraper host-object and captured the result set in the search result page. The mashup code as follows,

function search(searchUrl) {
	var scraper = new Scraper(
		
		    {searchUrl}
			
			    
				
				   
				
			     
			
		
	);
	return new XMLList(scraper.response);
}

And finally the two gadgets were making service calls to the mashup service and retrieved the data as an XML object, making the data processing painless. The final version at the Gadget Server looked quite appealing.

WSO2 Gadget Server with UK gov data

Gadget Server look - in the end

Special thanks goes to Ruchira for helping me out with the mashup service :) You can download the Gadget code and the Mashup service and try the scenario yourself.

I integrated the facebook “like” button to my blog

2010 April 25
by Nuwan Bandara

I just integrated the facebook’s new like button, there was an easy to use plug-in for wordpress so I didnt have to make any code level changes to my template. So if you guys also thinking of integrating the social feature you might want to read this, this and this. :)

And for all you people who are too lazy to comment on posts, can now, just click on “like” and give me a heads up :D

Too many “Big Brothers” – Back to stone age ?

2010 April 23
by Nuwan Bandara

I am sure you have heard about the phrase “Big Brother is watching you”, and maybe read “Nineteen Eighty-Four” by George Orwell. But how many of you, realized that we are the very civilization Orewell describes in his novel which is completely kept under surveillance. Well it may sound bit harsh for now, but that day seems to be not that far away.

We live in the information era where we need to understand, that measurement of power is defined by the amount of knowledge and information we possess. We start our day, by checking our mails before we brush our teeth (Yeah that’s me, and I know few more like me :P ). We tweet about what we do, what we read, what we hear tagged with our location using geo-tagging. What ever we hear that we do not know, or if we are curious about something (maybe, Paris Hilton’s Sex Tape or about government corruption) we simply google. and as social beings, rather than hanging out with few friends on a Friday night, we comment and gossip around in facebook. So by doing all this we gather information about others, about things and knowledge about new development. As the result we become strong and well connected and that’s the beauty of it.

But the bigger picture is quite scary. Even how strong we become with information and knowledge there will always a stronger person, who will know all about you, from the day you started gathering information and that is the reality of the information age. There will always be a “big brother” who is watching you.

The funny thing is, at present the internet space is full of “big brothers”. Every e-business is trying to know about you and me and graph our social connections. However we should also understand that “Nothing in Life is Free”, nor service is a charity. Google provide us with search results and they make money out of advertising on us. facebook and twitter are the same and now they are at war (With facebook like, connect pushing out to the entire web) simply to own our activities on the web. Whoever wins, each one of them possess great power with regard to information they own. I think this is where we can say “Great power comes with great responsibility” and be blind about the facts.

In my opinion, going back to the stone age is not the answer. as member of the information society we should be aware of our rights, we should read the privacy statement of the e-Business we consume and one of my colleagues said today, “if you are planing to be the next president of the state, do not build up a facebook profile with all your most personal details, it might backfire” :D

Mashing up RDF data with WSO2 Mashup Server

2010 April 14

Okey so this is the fun part that I promised to write about :D . I managed to cook up a use-case to demonstrate RDF querying and making use of the semantic data. The data that I am using for querying, is the rdf data sources available in the UK data.gov site. With some analysis I figured out that this task can be fundamentally archived using the combination of Mashup and Gadget Technologies. My choice of tools were WSO2 Mashup Server and WSO2 Gadget Server for their great flexibility and of cause for other obvious reasons :D . However the Mashup Server does not natively support RDF data retrieval, hence I had to do some work to get such functionality integrated. The great fact about the mashup server is its extensibility, the concept of host objects and the ability to write custom host objects and its pluggable nature comes handy in such cases. The high level architecture of what I am trying to achieve is as follows.

RDF data retrival with WSO2 Mashup server / WSO2 Gadget Server

To implement the above architecture with the tools at hand I created a custom host object that can be plugged to the Mashup Server. When dealing with semantic web related tasks and RDF data handling HP’s Jena java library comes in handy. With the use of Jena-ARQ (for SPARQL) api I managed to get the host object working with few lines of code.

.....
            Dataset dataSet = DatasetFactory.create(sparqlObject.rdfDataSource);
            // Create a new query form a given user query
            String queryString = sparqlObject.spaqrlQuery;
            Query query = QueryFactory.create(queryString);
            QueryExecution qe = QueryExecutionFactory.create(query, dataSet);
            ResultSet results = qe.execSelect();
.....
           resultString = ResultSetFormatter.asXMLString(results);
..... OR.....
           ByteArrayOutputStream bos = new ByteArrayOutputStream();
           ResultSetFormatter.outputAsJSON(bos, results);

With the host object in place, the next task was to create a Mashup in-order to query the rdf data with a given source (EndPoint or data source). The javascript service (Mashup) is created to serve this purpose, where the consumer can specify the RDF endpoint or the data source with the SPARQL query and retrieve the dataset in XML or JSON.

.....
function RdfDocQueryService(rdfDataSource, rdfQuery, resultType) {
   var sparqlObj = new SparqlHostObject();
   sparqlObj.rdfDataSource = rdfDataSource;
   sparqlObj.spaqrlQuery = rdfQuery;
   sparqlObj.resultType = resultType;
   return new XML(sparqlObj.getDataFromRdfSource());
}

Finally to bind everything together, lets try querying some data. My example usecase is to use the query at N2 blog to retrieve traffic monitoring points in UK roads. The query to retrieve the data set as follows,

#List the uri, latitude and longitude for road traffic monitoring points on the M5
PREFIX road:
PREFIX rdf:
PREFIX geo:
PREFIX wgs84:
PREFIX xsd:
SELECT ?point ?lat ?long WHERE {
  ?x a road:Road.
  ?x road:number "A4"^^xsd:NCName.
  ?x geo:point ?point.
  ?point wgs84:lat ?lat.
  ?point wgs84:long ?long.
}

To visualize these points I have created a gadget with the aid of Google Maps api. This gadget can be hosted in the Gadget Server, where it can dynamically retrieve traffic monitoring points for each road in the UK and display them in the map as follows.

Traffic points in A4 road, UK

The Semantic Web

2010 April 8
by Nuwan Bandara

For the past week or so I was able to work with data.gov.uk to find a possibility to make use of government/public data for a nice sample gadget, that will work in the Gadget Server. However it is quite disappointing to see most of these data are in lengthy PDF files, and without some expensive processing these data could not be consumed easily. The happy scenarios in my mind was, that there will be a bunch of RESTfull web services where I can request for public data and in return I will get the result set in XML or (and if am really lucky) in JSON ;)

After some browsing over the data in the UK data site, There was one interesting area which is called SPARQL and that was my moment of joy :D . there were six government data endpoints which I can query against and the result set can be retrieved as XML or JSON. This was the ideal scenario, even the available data is limited, there it starts the story of the Semantic Web.

When I was reading for my bachelor’s thesis, under service orchestration and chirography, the concepts of semantic web (services) were discussed in length. However the idea is still in its early stages, and there is still no real business use-case for it. The very purpose of semantic web is that the resources of the WEB and there relationships can be self explained and explored.

Self Explainable RDF graph

I have a dream for the Web [in which computers] become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A ‘Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialize.

Tim Berners-Lee, 1999

The UK data site has offered chunks of eduction, transport, financial, environmental, analytical and statutory notices as RDF data endpoints which can be queried using SPARQL (SPARQL Protocol and RDF Query Language).

So why let this great opportunity to lay dry, my idea was simply to use these endpoints and query these data and display them in some gadgets, for instance with transport data we can draw traffic details in maps, or with financial and educational data we can create some graphs and charts.

But with some thought, WHY NOT go for mashing up, We have a great tool (WSO2 Mashup Server) just made for that. So I was cutting code for the past whole week to create a custom host object in the Mashup Server to query RDF (RDF Service endpoints or RDF Data sources) data, and am almost done, the host object is created, two mashup services are created (Query a RDF endpoint/and RDF data source) now its a matter of integrating the services with Google Maps API and visualize them as a gadget in the Gadget Server.

So yeah the good stuff will be in the next post. keep tuned :D