CS-10 — Data platforms
ETL and financial data quality on a banking Big Data platform
ETL processes with Spark and data quality validations over large volumes of banking information on BBVA's Big Data platform.
- My role
- Data Finance Engineer (GFI / Bittek).
- Period
- 2018–2019
- Disclosure
- Anonymised details
- Technologies
- Apache SparkPythonScalaSQLJupyterHadoopHDFSParquetAvro
- Last reviewed
- 13 November 2026
Context
Big Data initiatives at a financial institution: processing architectures for large volumes of banking data, focused on ingestion, ETL transformations, modelling and quality.
Challenges
- 01Ensuring that financial information is processed correctly at scale.
- 02Nulls, missing fields, incorrect records and inconsistent data.
Solution & architecture
Distributed ETL and ingestion flows with Spark (Python and Scala) on Hadoop/HDFS with Parquet and Avro, and validation and transformation logic: completeness checks, type validation, null handling, referential consistency, cleansing, transformation validation, anomalous record detection, business rules, quality monitoring and, where appropriate, reconstruction or imputation of missing values.
Key decisions
Not documented yet
Trade-offs
Not documented yet
Results
Results will be published once verified
Lessons learned
Not documented yet
No specific certifications or regulatory compliance are claimed.
Ask about this case study
Questions here are answered with the case study “ETL and financial data quality on a banking Big Data platform” as context.