How 1.8 Million Ball Bounces Shaped an Evolvable Data Pipeline

Every data system running longer than a few months eventually hits this problem. Business requirements change, someone wants a new field, someone wants a type changed and the system has to handle it without downtime or data loss. In a classic relational database, adding a column to a table with a huge number of rows can lock production for hours and changing a data type often means rewriting the entire historical table.

Why it matters

• A rigid schema means a fixed cost that scales with data volume. As history grows, so does the cost of migrating it.
• A system meant to run for years will eventually face changes nobody can predict today. Someone will query five-year-old data and expect it to be readable in today's query format even though the schema looked completely different back then.

Tennis and Hawk-Eye

Hawk-Eye, the technology used for electronic line calling in tennis, is a computer vision system that tracks the ball's trajectory using multiple cameras to determine whether a shot landed in or out. A network of six or more calibrated cameras surrounds the court, each one tracking the ball independently and the system triangulates those views into a single 3D reconstruction of the ball's flight path, accurate to a few millimeters and delivered in a fraction of a second.

The historical Hawk-Eye dataset, covering matches from 2005 to 2009, already contains over 1.8 million recorded ball bounces from more than 1,800 men's singles matches. That data includes the position of every bounce, the serving player, the ongoing score and the point winner, all captured at millisecond granularity. It's been used since in academic research on topics ranging from risk management in tournament play to mixed strategy modeling, which only works if the old data is still queryable in a consistent way alongside everything collected since.

Two decades later, the same type of data is still flowing in, just at a greater scale, with more courts and extra fields nobody planned for back in 2005. Electronic line calling has since expanded from a challenge system, where a player could request a review of a single call, to full automated officiating at every court of major tournaments. Ball spin, player biomechanics and additional tracking metadata have all been added over the years, on top of a system that was never designed to be torn down and rebuilt every time a new sensor or a new sport gets added to the platform. This is exactly the kind of long-lived, ever-growing dataset that makes schema evolution a real engineering problem instead of a theoretical one.

It's worth walking through how those 1.8 million bounces actually get collected, since the process explains a lot about why the resulting dataset is so consistent across two decades. Before a match, each of the six or more cameras around the court is calibrated against the exact geometry of that court, since a clay court behaves differently on impact than a hard or grass court and the calibration has to account for that. During play, every camera captures the ball independently in its own 2D view. Software then detects the ball in each of those views frame by frame. Those independent detections get triangulated into a single 3D trajectory, giving a position accurate to a few millimeters at every point along the ball's flight.

That trajectory, together with metadata like the server, the score and the point outcome, is what gets written into the dataset as one structured record per point.

Technical solution: Amazon S3 + Apache Iceberg

Modern infrastructure handling this kind of sports data stores it on Amazon S3 using the Apache Iceberg table format instead of classic, rigid tables. Three mechanisms are worth breaking down:

• Schema evolution without rewriting data. Adding a new field doesn't require migrating historical records. Old data stays readable, and the new column is simply empty for older entries.
• Time-travel queries. Iceberg keeps a version history of the table, so you can query data exactly as it looked at a past point in time.
• ACID transactions at scale. Reads and writes stay consistent even while new match data is being written and historical data is being queried at the same time.

Tech stack

The full pipeline behind this kind of system typically has four layers.

Ingestion. Cameras on the court stream raw video to cloud-hosted tracking services. Those services analyze the footage to determine ball and player position, then convert the result into a compact binary message format, commonly Protocol Buffers, before pushing it into a streaming broker such as Apache Kafka.

Stream processing. From there, data first passes through a stream processing layer built on Apache Flink, typically split into a few parallel jobs. One job handles real-time analytics for live dashboards, another enriches and standardizes the raw tracking data for downstream consumers and a third prepares the data for long-term storage by buffering writes and optimizing cost before it lands in S3.

Storage. From there it moves into Iceberg-format tables on Amazon S3, which lets the system handle historical queries and live writes from ongoing matches without the two workloads conflicting. Live dashboards typically read from a separate fast-indexing layer for immediate results, while Iceberg on S3 handles the cost-effective long-term archive with full historical query support.

Consumers. The same processed data feeds several downstream systems at once, each with its own latency needs. Officiating needs a call within a fraction of a second. Broadcast graphics and replay overlays can tolerate a few seconds. Tournament and ranking systems aggregate results across simultaneous matches for live leaderboards. Distributors such as Sportradar package point-by-point data into APIs for statistics and betting platforms on a longer delay, and coaching firms turn the same official data into tactical breakdowns hours or days later. One dataset ends up serving all of these audiences from the same underlying records.

Day to day impact

It's worth looking at what this architectural choice actually delivers in practice, not just in theory. According to the AWS case study on Hawk-Eye's infrastructure, moving to this kind of managed, Iceberg-backed storage layer produced a 50 percent reduction in storage costs, alongside an 80 percent reduction in consumer lag from better partitioning across multiple downstream consumers. On the operational side, the team reported roughly 80 percent less time spent on infrastructure management and a 60 percent reduction in total cost of ownership compared to the previous self-managed setup.

Those numbers matter because they show the benefit isn't purely architectural elegance. A flexible schema and cheaper long-term storage translate directly into less engineering time spent firefighting migrations and more time available for building new features on top of the data, whether that's a new tracking metric, a new sport added to the platform or a new downstream consumer that needs access to years of historical results.

Limitations

• The Iceberg table format adds a metadata layer and a level of complexity that a simple file-based write doesn't have, so for smaller, short-lived projects it can be overkill.
• Teams need to understand the split between the streaming layer (fast but expensive) and the table layer (cheaper but with some delay versus live data) and design deliberately around which data goes where.