The Development of Kowalski: Our Market Intelligence and Structural Monitoring Engine

When we looked at the Road User Charges landscape in New Zealand, we saw a lot of noise. Pricing, fee schedules, and service terms were scattered across dozens of different provider sites, often buried in inconsistent tables or deep within PDF disclosures. For the average driver, comparing these was nearly impossible. We decided to build a way to fix that, and the result was Kowalski, our proprietary market intelligence and structural monitoring engine.

Kowalski isn't just a basic web scraper. It is a purpose-built data pipeline designed specifically for the complexities of the RUC ecosystem. Because provider layouts and pricing structures change without notice, we couldn't rely on "off the shelf" tools. We needed a system that was "source-aware" and could handle the irregular way that RUC data is published.

"The challenge with RUC data is that there is no standard schema," says Briyarne, CTO. "One provider might list a fee in a clean table, while another embeds it in a three-page policy document. Kowalski was engineered to handle that inconsistency. We built it with AI-assisted parsing and pattern-based logic so it can identify meaningful changes even when a website’s layout shifts. It’s about turning fragmented information into a single, queryable source of truth."

The engine powers our public-facing tools like RUC Hub and RUC Compare, ensuring the data New Zealanders see is both current and accurate. But getting that data isn't a "free-for-all." We designed Kowalski with a strict "Default to Caution" principle.

Adam Johnston, CEO, explains the philosophy: "We operate Kowalski with a huge amount of respect for the sources we monitor. It only engages with publicly available commercial data, and we’ve built in intelligent rate limiting and request pacing. We aren't interested in personal data or anything behind a login. We are strictly interested in facts: pricing, fees, and service parameters. By being respectful in how we acquire data, we ensure that our market visibility is built on a high-integrity foundation."

Behind the scenes, the technical mahi was led by our data engineering team. Marcus, our Lead Systems Architect for the project, and Sarah, our Quality Assurance Lead, developed the layered validation controls that every piece of data must pass through. "We don't just extract and publish," says Marcus. "Every record goes through a validation workflow. We check for schema matches, range consistency, and structural outliers. If a price update looks like an anomaly, the system flags it for manual review rather than pushing it live. This ensures that the intelligence Kowalski provides isn't just fast, it’s reliable."

Kowalski also maintains a full "Revision History" for every provider it monitors. This temporal tracking allows us to see how the market is evolving over time, supporting the deep-dive research and trend analysis we provide to our users.

While Kowalski is a highly automated system, it remains under human governance. Every monitored source goes through an internal approval process before it’s added to the pipeline. It is a sophisticated piece of kit that does the boring, repetitive work of data monitoring so that we can focus on what matters: giving New Zealand drivers the transparency they deserve.