Collaborate, Create, Communicate

Google Maps, Open Source and Plotting Ongoing Research: Demonstrating Innovation

It is the perfect time to celebrate CGX 2.0’s birthday by talking about the tangible differences made by ICT-KM’s choice to shift away from proprietary technical architectures and move towards a more ample and sustainable technical landscape.  This technical landscape is epitomized by what is called Enterprise 2.0, “a system of web-based technologies that provide rapid and agile collaboration, information sharing, emergence and integration capabilities in the extended enterprise” (AIIM – What is Enterprise 2.0?).

The CGIAR Ongoing Research system is a practical example of how we apply the Enterprise 2.0 and  collaboration principles to the CGIAR’s core work: research. The way the system was designed, the technologies we selected to implement it, the collaboration process with the Regional Plan for Collective Action, all tell a story of how the Program has chosen to innovate the way to connect, share and collaborate online.

I was first given the challenge of opening the doors to Ongoing Research in late 2008 with the Eastern and Southern Africa (ESA) map, a proof-of-concept prototype using Google Maps and Data APIs.

My goals in designing the ESA system were to: facilitate the contribution of information regarding ongoing research in Africa, expose this information to a broad public, and link the information to that which is published in CGMap project plans (MTPs).

In a nutshell, adhere to ICT-KM’s Triple A Framework.

So for all of 2009, the ESA Prototype Map was launched as a catalyst for meeting these goals, as well as to prove the concept that project leaders and researchers throughout the CGIAR are eager to connect and give visibility to their work.  We had a great response from scientists and researchers who found that contributing to the map was easy and time-effective.   The Google Map interface was well appreciated for navigating projects by country.  A survey was performed which asked researchers about the usefulness of the prototype map, and again the results were quite good and verified that the Who is doing What, Where and with Whom is paramount.

Surveys and feedback also highlighted the need to go beyond the prototype, with extended roles for contribution, site search and several analysis features.  But where should this information “live”, and how should it be managed?  Could we and should we use one of our existing systems or frameworks?

In the latter half of 2009 I assessed various possibilities of integration with our systems, but I finally decided on an Open Source solution.  Coming from a solid corporate (banking) background, it felt almost wrong to choose Open Source, but I must say that the choice of Linux, MySQL (Sun Microsystems, now Oracle), and Symfony, an open-source Web application framework, was a perfect fit for the technical requirements and budget.  I was already familiar with Linux, having worked solely on Unix-based systems in the corporate world.  Also Sun’s products have always been a part of my foundations in technology.  Google Maps (the Google Maps API) has been a fundamental component in CGMap, opening access to research project factsheets.  The Google Maps API allows using maps within Web applications, providing a vast set of customizable geo-features.

The introduction of these technologies was not the only innovation in this project.  We used Agile software development, a group of software development methodologies which uses iterative development to allow requirements and development to constantly evolve.  This catered to the need to express requirements  and changes while the system was being built.

To talk more about Symfony and Agile software development, let me introduce you to Jacopo Romei, Agile development coach and specialist in open standards and open source.  Jacopo developed the recently launched Ongoing Research Web application.

Before handing over to Jacopo, I want to say that working with Jacopo on this project was a great pleasure, and he proved to be passionate not only about building for the need, but also about understanding the need.

Over to you Jacopo …

“Thanks Michael.

First I’d like to clarify what the whole point is in using an agile methodology. Put in simple words it’s all about turning changes into opportunities. The traditional approach to software development aims to ossify requirements until the next release, exposing the project to becoming unfit for the context it’s released into at the end of the development cycle. While we are used to say requirements should reflect market I strongly believe we should stick to a far better motto: requirements are the current market.

Obviously it doesn’t come for free, but the agile development community spent the last 15 years minimizing the cost of change and then, most importantly, removing the fear of change.

The prototyping phase of the ESA map and the rich analysis it brought with it was a goldmine that allowed us to start developing a system close to users’ needs. Of course, we also wanted to make sure we could face a change in requirements or an emergent and previously overlooked aspect of the design.

Symfony granted us enough modularity to break the project into low-coupling, high-coesion modules to develop a full rewrite of the prototype on top of MySQL. It also made room for one of the pillars of agile development: feedback. We adopted a full and continuous quality approach. With its full fledged automated tests library, Symfony provided the perfect environment to write a complete suite of automatic tests to keep the bug rate low and regressions rare. Every change in the code was locked in a cage of executable tests providing us with relentless feedback on the quality of our code letting us focus on real productivity – ever thought about debugging as a completely non-productive activity?

Even more important tests acted as a solid reference to drive development when requirements changed. Yes, requirements changed along the road. Once again. Even after having spent months on a prototype. Even with close-to-perfection sharing of project goals among all stakeholders.

Agile development methodology helps teams to embrace change to the point that the design phase is diluted over the activity and changing requirements is never a showstopper.

We split the development cycle into 12 very short iterations (each was 5 workdays long). Each iteration was meant to share results among developers, usability experts, information architects and project managers. We met the goal of releasing a working-bug-free though not-complete-yet system at the end of each iteration.

I paid most of my attention to keeping communication fluent between me, Michael and Antonella to reduce the risk of unexpected outcomes on both sides. We expressed user requirements in a very fine-grained way, called user stories, expressing software features from the user point of view, hiding any technical detail not useful to express the final user experience. Those small deliverables were the meeting point among all of us throughout the iterations, so we were all up to speed with what had been done and what was left to do.

Eventually, we met our common goal within the initially planned 12 iterations with just one extra day. Over the first week from the beta release of Ongoing Research, the user community reported only one serious bug. The system is working and keeps growing with an increasing collection of CGIAR projects that contributors create, manage and maintain.

Agile methodology is a way to drive software development if you already believe in values like communication, feedback, simplicity and respect. Removing the fear of change is a key aspect too. Everything was possible here at ICT-KM, since we just let those values drive decisions, day by day, face to face. Even through a Skype call! ;)”