At a glance

2019
Completed
CKAN
State government
Build & migration
GovTech, Open data, Open platform, Open source, Whole of government
Tools & systems, Open standards & common platforms, Open source

The players

The Victorian Department of Premier and Cabinet (DPC) manages several centralised digital services, including data.vic.gov.au, a CKAN-based open data platform where public servants, industry, researchers, and citizens can access government data.

The problem

When DPC was looking at renewing its hosting agreement for Open Data Portal, both DPC and Salsa thought it would be a great opportunity to bring the portal onto an open source platform, making the entire stack open source. We spoke to our open platformExternal Link partners amazee.ioExternal Link , and realised we could bring the portal across to Lagoon and deliver cost savings. Salsa presented this option to the Data Vic Team. The team was excited by the proposal, already having an understanding of the benefits of the whole-of-government digital solution Single Digital PresenceExternal Link and the success of the underlying open platform (Lagoon).

However, CKAN had never been integrated with Lagoon before, so it was new (and exciting) territory. In fact, as far as we know this is a world-first — the first time open data has been built with open source software and hosted on an open source platform.

The solution

The Open Data Portal (that we built and currently manage with DPC) was already using open source software, CKANExternal Link . However, it was hosted on traditional cloud infrastructure. So ‘all’ we needed to do was migrate it to the open source hosting platform, LagoonExternal Link . Lagoon had been created as an open source platform with an initial focus on website content management systems such as Drupal. This solution represented an opportunity to extend Lagoon support for CKAN.

Lagoon uses containerisation technology (see our post What is containerisation?External Link for more information), which relies on ‘images’ that are run within the container. So the first step in this project was to create images (using open source software Docker) that were for CKAN rather than Drupal.

CKAN has services/functions that need to be supported, like harvesting services (pulling data from other CKAN instances) and data pushing (‘pushing’ data to the database so it can be manipulated in various ways, such as through APIs). See our blog on Victorian Government APIsExternal Link for a definition of APIs.

Once the platform was set up, Salsa moved onto deploying and configuring CKAN on Lagoon, working in consultation with amazee.io. We used the existing Data Vic CKAN database and CKAN code, and deployed it onto Lagoon. Salsa CKAN experts worked with amazee.io Lagoon experts to successfully execute this complex merging of CKAN and Lagoon.

As part of this project we laid some foundations to ‘open up’ this work, so that others can re-use the CKAN architecture we’ve set up in Lagoon. The intention is to make it easier to set up CKAN on Lagoon and bring the benefits of containerisation to CKAN projects. We wanted to do this by making the code and accompanying documentation available as a starter kit.

The original contribution to Lagoon we made can be found here: https://github.com/amazeeio/lagoon/pull/897External Link

This was accepted and merged and is now part of the Lagoon offering.

We also provided a CKAN Lagoon starter kitExternal Link (based on CKAN version 2.7.5 and soon to be updated) that will help anyone who wants to get up and running with CKAN on Lagoon.

It’s certainly not polished but it's a start we hope will help others get going.

Benefits

The benefits of bringing Victoria’s Open Data Portal onto Lagoon are:

  • A fully open source solution, in line with the open government movementExternal Link , that is transparent, flexible, extensible and with no proprietary lock-in (which typically comes with higher related costs).

  • Strong developer workflow processes with automated deployments of ‘local’ feature branches so that developers can test their features and code before merging them into the UAT or production environments.

  • Inherent benefits of KubernetesExternal Link (which is what Lagoon is built on) such as auto-healing, auto-scaling and distribution across multiple availability zones for truly enterprise-grade resilience.

  • Centralised logging and notifications on deployments and failures for streamlined and efficient software development, error maintenance and bug fixing.

  • Flow-on benefits to open data, open source and open government — hopefully other government organisations will follow Data Vic’s lead and use a fully open source solution for their open data portals.

Why Salsa Digital?

Salsa Digital held the existing hosting contract and we’ve built a strong relationship with the Data.Vic team and wider DPC team over the past few years. In addition, we’ve successfully used the Lagoon platform before and co-created and co-built Single Digital Presence with DPC. We’ve also got an ongoing relationship with DPC, working on continual enhancements to Single Digital Presence and general program support. DPC is an innovative and forward-thinking department that’s always open to new and innovative ways to deliver projects and benefits. DPC was also the first significant government department to invest in and adopt Lagoon, and so they knew firsthand about the platform and its benefits.