Data Engineer Advisor

Role Summary NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads. This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines. Experience 10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark. ________________________________________ Key Responsibilities Migration Framework Engineering • Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code. • Build reusable Hive metadata/schema discovery and migration-configuration components. • Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history. • Implement PNC-defined Data Movement Controls including: o checkpointing, o restart/recovery, o idempotency, o retries, o control totals, o audit logging, o exception handling. PB-Scale Migration Execution • Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale. • Design migration patterns for small, medium, large and very-large tables. • Tune parallel Spark execution across tables, partitions and chunks. • Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables. • Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks. • Support migration-wave execution and production issue resolution. Cyber / Tokenization Integration • Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines. • Implement secure processing of sensitive fields. • Handle service failures, retries, throttling and restart scenarios. • Measure and optimize tokenization impact on end-to-end migration throughput. • Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules. Apache Iceberg Engineering • Create/configure Iceberg tables using approved target standards. • Implement appropriate partitioning and file-sizing strategies. • Optimize compaction and small-file handling. • Work with Iceberg snapshots, schema evolution and metadata. • Troubleshoot Iceberg write/commit/catalog issues. • Support table maintenance and performance optimization. Data Validation & Reconciliation • Develop automated source-target validation covering: o row counts, o schema conformity, o partition completeness, o control totals, o aggregates/checksums, o null/duplicate checks, o tokenization validation where applicable. • Isolate reconciliation failures at table/partition level. • Implement selective retry/reprocessing of failed data. • Produce auditable migration evidence. Production Engineering • Monitor and troubleshoot distributed Spark jobs. • Analyze Spark UI, executor logs and resource utilization. • Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues. • Build operational telemetry covering migration status, throughput, retries and failures. • Participate in code reviews, testing, CI/CD and engineering standards. • Mentor mid-level/junior Data Engineers. ________________________________________ Mandatory Skills Strong hands-on experience with: • Python • PySpark • Apache Spark / Spark SQL • Hadoop / HDFS • Hive / HiveQL / Hive Metastore • SQL • Apache Iceberg • Parquet • Linux • Git • distributed data processing • large-volume ETL/data migration • source-target reconciliation • Spark performance tuning. Practical Iceberg experience should include: • table creation/read/write, • partitioning, • snapshots, • schema evolution, • file sizing, • compaction, • Spark integration. Strong understanding of: • parallelism, • checkpoint/restart, • idempotency, • retry/recovery, • large-table processing, • small-file problems, • distributed-system troubleshooting. ________________________________________ Preferred Skills • Apache Polaris/Nessie/Glue or another Iceberg catalog. • Kubernetes/OpenShift/Spark Operator. • S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred. • TaaS/Protegrity or equivalent tokenization/masking technology. • PostgreSQL for migration configuration/control/audit metadata. • Trino. • Control-M/Airflow or equivalent enterprise scheduler. • CI/CD for Python/Spark applications. • Banking/financial-services experience. • PB-scale migration experience. • HiveQL→Spark SQL modernization experience. • Bronze/Silver/Gold lakehouse experience.