IATI Tech update: Q4 2021/Q1 2022

This post has been written by Wendy Thomas, IATI Technical Lead based at Development Initiatives.

Welcome to the IATI Technical Team’s quarterly update. Every quarter, we update our community on the progress made on IATI tech and share our plans for the next quarter. The Technical Team’s quarterly plans are reviewed by IATI’s Governing Board technical focal points. You can read the last quarter's update here.

Overview of 2021

Before we begin with our review of quarter 4, IATI Software and Systems Architect Alex Lydiate has provided an overview of 2021.

In 2021, the Technical Team focused on rebuilding the fundamental technical infrastructure that underpins the delivery of IATI’s publishing and data use services. At the beginning of 2021 we aimed to implement IATI’s new Unified Platform (and launch the IATI Validator version 2 on the platform) by quarter 3. Creating IATI’s Unified Platform was an essential recommendation from IATI’s 2020 Stocktake, which found that IATI had no existing architecture design, infrastructure or application code.

By the middle of 2021 the Technical Team was asked to implement the next version of the IATI Datastore on the Unified Platform. This required significant additional work by the Team. We committed to delivering a production-ready public API by the end of quarter 4 2021 and an accompanying web application called Datastore Search, by the end of quarter 1 2022.

Despite these challenges, the Technical Team was able to build both the new version of the IATI Validator and Datastore on IATI’s Unified Platform by the end of 2021. The Unified Platform went live in September 2021, along with the second version of the IATI Validator. We now have a robust platform that centralises the tracking, validation and indexing of all IATI documents on a largely serverless and wholly microservice Azure cloud architecture.

The IATI Unified Platform has:

• Reduced the timescale of processing IATI documents, from discovery through validation to the Datastore, from 10 days and more to a matter of hours
• Reduced the monthly hosting costs to less than half that incurred previously
• Represented all valid documents in the Datastore exactly as they were published, using unique, dynamic and recursive algorithms
• Provided complete transparency and understanding of the journey of each published document, both in terms of where they are in the discovery-validation-index pipeline and also their history of validation.
• Provided best-in-class API management and discovery via the API Gateway and Developer Portal

The Team also launched the new IATI API Gateway and Developer Portal to provide a single point of access for all of IATI’s API data services. Both offer state of the art analytics, security and discovery tools for all public IATI APIs.

The new Validator API was the first service offered on the API Gateway. It was also the first time IATI has provided a public API for the Validator. This means organisations are able to access an API to integrate IATI validation into their internal data systems and external tools. IATI’s new publishing tool being developed in 2022 will be reliant on this service to ensure all IATI documents published through the application are validated and valid before publishing. This is absolutely key to driving improvements in the quality of data from a structural standpoint, and all IATI publishers are encouraged to integrate the API into their own workflows.

Quarter 4 2021 updates

Datastore

As mentioned above, in quarter 4 the Technical Team finalised the work to bring the Datastore in-house on IATI’s new unified platform and successfully delivered a ‘soft’ launch of its API as scheduled in December. The new Datastore API has been entirely rebuilt to a higher specification, with lower costs and greater efficiency than was possible with previous versions. The Team also prepared redirects and extensive user documentation in advance of the full launch of the Datastore API version 3 on 11 January 2022.

Validator

We have been updating the front end of the Validator, ready for release in early 2022 along with the full launch of the IATI Datastore. Updates include displaying the validation status for each published file. This allows users to easily check which of their files are critically invalid, contain errors, have warnings or have no issues, from a single page. We are also adding in times and dates for when each file is identified in the Registry, validated and becomes available in the IATI Datastore.

We’ve been delighted to support IATI Validator API users to make over 12,000 requests in the last quarter of 2021 to explore the Validator API. We are also helping them to embed this into their publishing processes, which will improve the data quality of their files pre-publication.

Publishing Tool

In quarter 4 we worked with our Secretariat partner UNOPS who led a competitive procurement process to find an organisation to develop IATI’s new Publishing Tool. We were pleased to announce that Nepal-based technology company Young Innovations was chosen (see announcement here). We look forward to working with a company that has exceptional IATI knowledge and experience of developing their own publishing tool, AidStream.

d-portal

In quarter 4 we noticed that some larger d-portal queries made via d-query were slowing d-portal’s overall response time. We worked with d-portal supplier, Wet Genes to switch d-portal to stream data back to those users, allowing them to download larger datasets without impacting front-end users.

At the November 2021 IATI Governing Board meeting, we discussed options for d-portal version 2. The Board recommended that we work with Wet Genes to refactor and improve d-portal and this work will commence in 2022.

Registry

We improved the security of the Registry API by switching the API authentication to use tokens instead of keys. In addition to maintenance work, we progressed six user experience improvements, which will enhance the publishing journey for organisations. These were released at the end of January. You can read more about this work here.

How publishers publish

A joint activity with UNDP at the intersection of data quality and data use, this project is helping the IATI Secretariat understand how different organisations publish to IATI. This research will enable us to eventually bridge the gap between publisher practices and data user needs. In quarter 4, we hosted a session to engage publishers at the IATI Virtual Community Exchange 2 (12-13 October). We also held webinars with each publisher stakeholder group in the project (bilateral, multilateral, DFIs, NGOs and private sector) and offered one-to-one meetingss for organisations that were unable to join or had additional questions. Recordings from the webinars are available on IATI Connect. We want to thank everyone who got involved and found the analysis helpful!

Data Quality Index

We worked with Secretariat colleagues to conclude Phase 1 of the Data Quality Index consultation. We held a well-attended session at the Virtual Community Exchange in October and an additional webinar consultation later that month. We shared a summary document outlining a way forward for each measure and are now preparing the draft technical methodology, to launch Phase 2 of the DQI consultation in February. You can read about this next phase here.

Helpdesk queries

We continue to receive a steady number of Helpdesk requests (636 received from September to December) and feedback shows a high satisfaction rate (98%). The number of data use queries continues to increase, with these being jointly supported by the Technical Team and UNDP.

Staffing updates

Sadly we have bade farewell to Petya Kangalova, IATI’s Senior Business & Data Analyst. After five years with IATI, she has moved on to a new challenge, working for the Humanitarian Openstreetmap. Recruitment for her role is underway.

Quarter 1, 2022

With our Secretariat colleagues, we are detailing our high-level workplan for this year, to finalise quarterly deliverables, which will be discussed with the IATI Governing Board in early March. Meanwhile, we are implementing the work planned for quarter 1 of this year.

At the start of 2021 we also said a fond farewell to Melinda Cuzner of SIDA, who was one of our IATI Governing Board Technical Focal Points and held the Technical Seat on the Board. Melinda was a source of great strategic support to the Technical Team and Secretariat and will be very much missed.

Datastore launch

Following its soft launch in December 2021, the IATI Datastore version 3 was launched officially to the IATI Community in January. Read more about the launch here. We are now designing a new web interface for the Datastore and exploring what functionality should be included in the Datastore and what should be incorporated in d-portal v2.

The Team is now busy preparing to finalise and launch the Datastore Search user interface and I look forward to this becoming available to IATI data users.

IATI Publishing Tool

In January, the Technical Team began working with Young Innovations to develop IATI’s new publishing tool. This quarter we are currently focusing on designing the tool, ensuring that it will be simple, intuitive and make publishing easier for organisations. We will provide updates on the IATI website as this project progresses.

Data Quality Index phase 2 launched

In quarter 1, we have also been busy working with our Secretariat colleagues to prepare for the launch of the second phase of the IATI Data Quality Index. In Phase 2, the IATI Secretariat will be consulting with the IATI community on the proposed methodology and how each measure will be weighted and combined to generate an overall score for each publisher.

Phase 2 of the consultation was launched on 15 February 2022 and will run until the end of June. I encourage IATI’s community to visit IATI Connect to read the methodology papers, join the upcoming webinars and post comments on the consultation.

d-portal version 2

The Technical Team will be working with our Secretariat colleagues to kick off the project to deliver a new version of d-portal. We will begin with a small user experience project to pull together all the existing inputs, from the Semantic Data Layer project and our previously gathered data use requirements, and then identifying what additional inputs are needed, e.g. user stories, screen designs. These will then inform a Terms of Reference and implementation plan for d-portal v2.

Stay updated

Stay updated with the Technical Team’s work by signing up to IATI’s Newsletter using the form below or by engaging with IATI’s Technical Community of Practice on IATI Connect.