Healthcare generates enormous volumes of data — clinical encounters in EHRs, claims in payer systems, device readings from RPM and IoMT platforms, documents in HIEs, and operational data across every administrative system. But most of this data sits in silos — locked in proprietary formats, spread across disparate systems, and inaccessible to the analytics, population health, and AI workloads that could transform it into actionable insight.
A healthcare data lake consolidates these sources into a single, scalable, analytics-ready platform. This guide covers the architecture from source ingestion through production analytics.
1. Why a Data Lake for Healthcare
Data warehouse vs. data lake. Traditional data warehouses require defining the schema before loading data (schema-on-write). Healthcare data is too diverse and too unpredictable for rigid schemas — HL7v2 messages, FHIR resources, C-CDA documents, DICOM metadata, claims files, and flat CSVs all need a home. A data lake accepts data in its native format (schema-on-read), storing raw data first and applying structure when queried.
The modern approach: lakehouse. Combine the flexibility of a data lake with the performance and governance of a data warehouse. Platforms like Databricks, Snowflake, and cloud-native lake formations (AWS Lake Formation, Azure Synapse, BigQuery) enable this pattern — raw data lands in the lake, transformation layers apply clinical schemas, and a query engine serves analytics, reporting, and AI/ML workloads.
2. Data Sources and Ingestion
EHR Clinical Data
Bulk FHIR export. The emerging standard for population-level EHR data extraction. The FHIR server produces NDJSON files — one FHIR resource per line — covering Patient, Condition, Observation, MedicationRequest, Procedure, Immunization, and other USCDI data classes. Use incremental export (_since parameter) for daily updates after the initial full load.
HL7v2 real-time feeds. ADT messages (admissions, discharges, transfers), ORU messages (lab results), and ORM messages (orders) provide real-time clinical events. Route HL7v2 messages through Mirth Connect to the data lake — parsing, transforming, and landing each message as a structured record.
C-CDA documents. Clinical documents from HIEs, referrals, and care transitions. Parse C-CDA XML, extract structured sections (medications, problems, allergies, procedures), and land as structured records alongside the raw XML for future re-processing.
Claims and Administrative Data
X12 837/835 transactions. Professional and institutional claims, remittance advice, eligibility responses. Parse EDI transactions into structured claims records with service lines, diagnosis codes (ICD-10), procedure codes (CPT), and payment amounts.
Payer data feeds. Attribution files, quality measure data, and risk adjustment scores from CMS and commercial payers. Often delivered as flat files (CSV, fixed-width) with payer-specific schemas.
Device and Monitoring Data
RPM and IoMT data. Blood pressure readings, glucose levels, weight measurements, pulse oximetry — high-frequency time-series data from connected medical devices. Ingest through device platform APIs or MQTT/HL7v2 streams. Code with LOINC observation codes for interoperability with clinical data.
SDoH and External Data
SDoH screening data. Patient social determinant assessments — food insecurity, housing, transportation. Ingest from EHR SDoH modules or community referral platforms.
Public health data. Census data, Area Deprivation Index, CDC surveillance data, and environmental datasets that enrich patient records with community-level context.
3. Data Lake Architecture Layers
Landing Zone (Raw Layer)
Store data in its original format — NDJSON from Bulk FHIR, raw HL7v2 messages, X12 EDI files, CSV extracts. Organized by source system and ingestion date. Immutable — raw data is never modified in the landing zone. This preserves the audit trail and enables re-processing when transformation logic changes.
Storage: Cloud object storage (S3, Azure Blob, GCS) with partitioning by source and date. Encryption at rest with customer-managed keys.
Curated Layer (Transformed)
Apply transformations: parse raw formats into structured tables, map codes to standard vocabularies (SNOMED CT, LOINC, ICD-10, RxNorm), resolve patient identity across sources through the MPI, deduplicate records, and apply data quality rules.
Schema design options:
OMOP Common Data Model. The OHDSI (Observational Health Data Sciences and Informatics) community’s standard schema for observational health data. OMOP provides a well-defined relational schema for clinical events, medications, conditions, procedures, measurements, and costs — with built-in vocabulary mapping. Ideal for research, population health, and multi-site analytics.
FHIR-native schema. Store data in a schema that mirrors FHIR resource structures — useful when your primary consumers are FHIR-aware applications. SQL-on-FHIR initiatives (Google’s FHIR analytics, SMART on FHIR Cumulus) are defining standard query patterns for FHIR-structured data.
Custom domain schema. Designed for your specific analytics needs — clinical operations, financial performance, quality measurement. More work to design and maintain, but optimized for your query patterns.
Analytics Layer (Consumption)
Serve data to consumers: BI dashboards, quality measure calculations, risk stratification models, AI/ML training pipelines, and operational reports. Materialized views, aggregation tables, and data marts optimize performance for common query patterns.
4. HIPAA Compliance Architecture
Encryption. All layers encrypted at rest (AES-256, customer-managed keys) and in transit (TLS 1.2+). Encryption applies to storage, processing, and query results.
Access control. Row-level and column-level security where supported. Restrict PHI access to authorized roles. Implement data masking for non-production environments. Separate access policies for clinical data, claims data, and de-identified research data.
Audit logging. Log every data access — who queried what, when, and what data was returned. Cloud-native audit logging (CloudTrail, Azure Monitor, Cloud Audit Logs) captures storage and compute access. Application-level logging captures business-layer queries.
De-identification pipeline. Build an automated de-identification pipeline that produces Safe Harbor or Expert Determination-compliant datasets for research, analytics, and AI training. De-identified data can be stored in a separate zone with broader access policies.
Data retention. Implement retention policies aligned with HIPAA requirements (minimum 6 years for covered entity records) and organizational policies. Automated lifecycle rules archive or delete data per policy.
5. Technology Stack
Cloud Platforms
AWS: S3 (storage) → Glue (ETL/catalog) → Athena (serverless query) or Redshift (data warehouse) → SageMaker (ML). AWS HealthLake for FHIR-native analytics.
Azure: Blob Storage → Data Factory (ETL) → Synapse Analytics (query/warehouse) → Azure ML. Azure Health Data Services for FHIR/DICOM.
GCP: Cloud Storage → Dataflow (ETL) → BigQuery (analytics) → Vertex AI (ML). Google Cloud Healthcare API for FHIR analytics.
Processing Frameworks
Apache Spark (Databricks, EMR, Dataproc). Distributed processing for large-scale ETL, FHIR resource parsing, vocabulary mapping, and feature engineering. The standard choice for healthcare data lake processing at scale.
dbt (data build tool). SQL-based transformation framework for the curated and analytics layers. Manages transformation dependencies, testing, and documentation. Increasingly popular for healthcare analytics teams.
Orchestration
Apache Airflow / Cloud-native orchestration. Schedule and manage data pipeline workflows — ingestion, transformation, quality checks, model training, and report generation.
6. Common Pitfalls
Ingesting everything without a use case. A data lake that collects everything and serves nothing is a data swamp. Start with specific analytics use cases — quality measure reporting, risk stratification, financial performance — and build ingestion and transformation for those use cases first. Expand incrementally.
Underinvesting in data quality. Garbage in, garbage out applies doubly in healthcare. Build data quality monitoring into every pipeline stage — completeness checks, vocabulary validation, referential integrity, and temporal consistency. A risk model trained on incomplete data produces dangerous predictions.
Ignoring patient identity resolution. Data from multiple sources must be linked to the correct patient. Without MPI integration or probabilistic matching, you’ll have duplicate patient records, split clinical histories, and unreliable analytics.
Building the lake before building the team. Technology is 30% of a data lake project — the other 70% is clinical informaticists who understand the data, data engineers who build the pipelines, and analytics consumers who define the use cases.
How Taction Helps
At Taction, our team designs and builds healthcare data lake infrastructure for health systems, payers, ACOs, and health IT vendors.
- Data lake architecture — We design HIPAA-compliant data lake architectures on AWS, Azure, or GCP with ingestion, transformation, quality, and analytics layers.
- FHIR and HL7 ingestion pipelines — We build pipelines that ingest Bulk FHIR NDJSON, HL7v2 messages, C-CDA documents, and claims data into the data lake with vocabulary normalization and patient matching.
- OMOP/FHIR schema implementation — We implement OMOP CDM or FHIR-native schemas in the curated layer — enabling standardized analytics, research, and quality measurement.
- Population health analytics — We build analytics platforms on top of the data lake — risk stratification, care gap dashboards, quality measure engines, and financial performance tracking.
- AI/ML data infrastructure — We build feature engineering pipelines and training data preparation for healthcare AI models — de-identification, cohort selection, and feature extraction from clinical data.
