NAB is returning to standardize the way it builds data pipelines for its Ada data platform, which will reduce operational costs and enable new recovery capabilities.
Dheeraj Puli from NAB.
Head of data assurance Dheeraj Puli told the Databricks Data+AI Summit in San Francisco that the bank is standardizing Spark Declarative Pipelines (SDP) across more than 1,600 data pipelines.
Ada is considered the “second chapter” in the bank’s data journey, first breaking out of hiding back in 2022.
It is based on Databricks on AWS and has replaced several large data platforms, including the NAB Data Hub (NDH) and the 26-year-old Teradata framework.
However, there are already opportunities for its modernization.
As is common among Databricks clients, NAB data on the platform is organized or structured according to what is known as a medallion architecture.
The raw, unmodified data goes to the bronze layer; It is then refined at the silver level, ready for use by AI/ML models and the like, or processed for more specific business use at the gold level.
Data typically moves through a multi-step pipeline as it passes through a medallion system.
The transfer of raw data to the bronze layer is already standardized, Pooley said.
“When we started, bronze actually used SDP from day one,” he said.
The opportunity is to modernize the data pipelines used to convert data for the silver and gold layers.
These pipelines are still based on Spark, but were built in different ways – and it is this customization and complexity that the bank now wants to remove from running Ada.
SDP is another way of constructing pipelines. Notably, the build is “declarative,” meaning that instead of coding each extract, transform, or load (ETL) step, engineers “define what the data should look like,” and then Databricks—or more accurately, its Lakeflow platform—performs the necessary “orchestration, state management, and incremental processing” in the background to make it all happen.
Work on the silver layer is already well advanced, with new pipelines being built using SDP and old ones being upgraded.
“All the new silver (pipelines) are going through SDP,” Pooley said.
“We have completed most of the migration of existing custom Spark SQL pipelines to SDP.
“We are now in the process of testing these jobs to ensure they can safely switch between custom Spark and SDP.”
A case study published by Databricks at the same time as the US presentation shows that work on the silver layer is about 50 percent complete; At the event itself, Pooley stated that “we currently have 3,800 SDP pipelines running in silver.”
The next step will be to extend the use of SDP to gold layer data.
“The long-term goal is a fully declarative, end-to-end streaming architecture from bronze to gold,” the case study states.
“Looking to the future, NAB aims to become the first bank to run 100 per cent of its pipelines on SDP, setting a new standard for how large financial institutions build, operate and scale data platforms.”
In addition to consistency and efficiency in pipeline development and operation, the bank expects savings from implementing SDP.
“We expect operating costs to drop by 15 percent if we disable all (custom) Spark jobs and then run (pipelines) only on SDP,” Pooley said.
“Currently we are running both jobs in parallel – SDP and then special jobs.”
The move to SDP also allowed NAB to privately review new features and conduct pilot testing.
One of these functions, “rewind and replay,” allows users to restore a failed pipeline to a previously consistent state.
Pooley said the bank decided to join the private preview because of the emergence of a finance-specific use case.
“A few months ago we had a business requirement (with) one of the financial systems (where) they didn’t want to load the data until it was fully consistent,” he said.
“For whatever reason, if that data is not consistent, they want us (to be able to) go back to a previous state where the data is consistent.”
Without a ready-made solution at the time, Pooley said NAB “created a tiny module of custom code to fit our timeline and business requirements.”
However, it appears that the “rewind and replay” feature may provide a natural way to meet specific business rollback requirements.
Ry Crozier attended the Databricks Data+AI Summit in San Francisco as a guest of Databricks.