Data modernization involves simplifying, automating, and orchestrating data pipelines, as well as improving the claim and settlement process using various AWS SaaS services, converting large data settlement files to a new business-required format. The task involves processing settlement files from various sources using AWS data pipelines. These files may be received as zip files, Excel sheets, or database tables. The pipeline applies business logic to transform inputs (often Excel) into outputs (also in Excel).
Our inputs typically come from a variety of sources. Utilize inputs from existing AWS tables and external inputs in Excel format. These diverse inputs are ultimately converted to Parquet format. This documentation outlines the process and includes the AWS data pipeline ETL jobs architecture for replication purposes.