Simulation of End-to-End Data Ingestion Strategies under Continuous Schema Drift

Abstract

Modern analytical platforms ingest data from heterogeneous and rapidly evolving sources, including microservices, third-party APIs, log streams, and event buses. In these environments, schemas rarely remain stable for long periods and tend to change incrementally as product features evolve, data producers refactor internal representations, or governance rules require new attributes. The resulting continuous schema drift challenges traditional end-to-end data ingestion pipelines that implicitly assume stable contracts between producers and consumers. When schemas drift while ingestion logic remains static, data loss, ingestion failures, and semantic inconsistencies can occur, especially for systems that enforce schemas eagerly and rely on manual change management processes. This work examines how different end-to-end ingestion strategies behave when exposed to continuous schema drift by constructing a simulation framework that models sources, pipelines, and storage targets over long horizons. The framework encodes schemas, transformations, and adaptation policies and replays drift scenarios with controllable frequency, magnitude, and correlation across sources. Several classes of ingestion strategy are considered, including schema-on-write with strict validation, schema-on-write with relaxed evolution, schema-on-read with late binding of structure, and hybrid patterns that separate physical and logical schemas. For each strategy, the simulation measures ingestion coverage, failure rates, lag between producer changes and consumer visibility, and resource consumption associated with reprocessing or schema migration. The study focuses on qualitative behavior and structural trade-offs rather than optimizing any single metric. The simulation results highlight how drift characteristics, such as directionality and concentration on specific fields, interact with pipeline design choices and suggest design considerations for constructing ingestion systems that remain operational under persistent schema evolution.