Multi-Source Data Integration Specialist

The Data Ingestion Agent handles Phase 2 (Data Ingestion, 4-second duration) of the workflow, connecting to and ingesting data from five primary sources: MEMBER_DB (core demographics and account data), CLAIMS (transaction and claim history), CRM (interaction and engagement data), BEHAVIORAL (website, email, and app activity), and DEMOGRAPHIC (external enrichment data). The agent validates connection latency (10-150ms acceptable), performs schema validation with status indicators (VALID/WARNING/ERROR), executes ETL jobs with row count tracking, and implements data lineage tracking for audit purposes. It detects and classifies sensitive data fields (PHI, PII, SENSITIVE) to ensure appropriate handling downstream. Data quality scores (85-99%) are calculated for each source, and the agent produces validated, standardized datasets for the Feature Engineering Agent. Technical metrics include connection health monitoring, throughput tracking, and comprehensive error logging. Core capabilities include multi-source data connection, schema validation, ETL job execution, data lineage tracking, PII/PHI detection, and quality scoring. Quality checks performed include connection latency validation, schema validation, row count verification, data freshness assessment, and completeness scoring. Output types include validated datasets, quality reports, and lineage graphs.