In this book we argue for the need for a new approach to data provenance and explain how recent advancements in data processing workflow automation present an opportunity to address this need. We introduce descriptive dataflow operators - a novel approach based on integrating descriptive workflow languages with a data modeling Domain-Specific Language (DSL). We review the current workflow automation technologies and propose a DSL that supports complex data transformations, enhances reproducibility, and enables precise data lineage tracking. Within the framework we introduce the concept of descriptive dataflow operators for more flexible and expressive data transformations.

Modern healthcare increasingly relies on complex data pipelines to process diverse diagnostic information, clinical records, and research data. This growing complexity, combined with emerging AI/ML applications and stricter regulatory oversight, demands sophisticated approaches to data preparation, documentation, and validation. Healthcare organizations face mounting pressure to ensure granular traceability and reproducibility of their data transformations while maintaining regulatory compliance. These challenges are particularly acute in research settings, where data provenance and quality validation become critical for scientific reproducibility and regulatory adherence.

Given the increasing complexity of healthcare data, data ingestion and transformation workflows present significant technical challenges, particularly in ensuring the reproducibility and seamless integration of diverse datasets for ML and AI model development.

We introduce the Dorieh Data Platform as an exemplar implementation of a DSL, providing a comprehensive framework for reproducible research. The platform's infrastructure supports robust data lineage documentation, validation, and error logging, making it a powerful tool for healthcare data analysis by ensuring transparent, auditable data processes and regulatory conformance.

We show how to apply this framework to analyze healthcare claims data quality, revealing insights into inconsistencies and deficiencies. Our approach demonstrates the potential for improved data management and accountability in scientific research, underscoring the necessity for precise, reproducible data transformation methodologies to produce reliable research outcomes.

Argues for the need for a new approach to data provenance Introduces the novel approach of descriptive dataflow operators Applies framework to analyze healthcare claims data quality, revealing insights into inconsistencies and deficiencies

Autorentext

Michael Bouzinier is a Senior Research Software Engineer within University Research Computing. He has over 30 years of diverse experience in software research and development and 10 years as a professional educator. His intellectual interests include semiotics, natural language processing and text analytics, data visualization, evolutionary and medical genetics, computer simulations, and explainable AI. Throughout his career, he has worked and led diverse international teams, successfully collaborating with developers and researchers from within the US, UK, Sweden, Finland, Belgium, The Netherlands, and Japan.

Dmitry Etin is a digital health strategist specializing in interoperability and health data management at the intersection of technology, medicine, and policy. He guides organizations and policymakers through the complexities of health data governance and large-scale interoperability initiatives, bringing practical solutions to critical challenges in digital health.He is deeply involved in shaping the European Health Data Space, working with the European Commission and supporting the European Medicines Agency in enabling an interoperable European Medicines Regulatory Network. Dmitry is also involved in several Horizon-funded research initiatives, accelerating the adoption of interoperable EHR systems in the EU. As a co-founder of Forome, an open-source initiative for genomics, health data management, and regulatory compliance, he helps develop data provenance and analysis tools for complex healthcare and clinical research cases.

Naeem Khoshnevis is a Research Software Engineer within University Research Computing. In this role, Naeem designs, builds, and optimizes software applications for researchers across Harvard University. Naeem has a superior mathematical and numerical analysis background and has developed, documented, debugged, extended, and refactored numerous scientific software applications for research groups, helping them successfully carry out their projects.

Max Shad is the Director of Engineering at the Kempner Institute for the Study of Natural and Artificial Intelligence and University Research Computing and Data (RCD) at Harvard University. In this role, he leads the computational program of the Kempner Institute, ensuring the provision of advanced Research Computing (RC) tools/services and expert Research Software Engineering (RSE) support. His efforts are instrumental in leveraging High-Performance Computing (HPC), particularly in Machine Learning (ML) and AI research, to facilitate pioneering discoveries in AI, ML, and computational biology.

Scott Yockel is the University Research Computing Officer at Harvard. In this role, Scott works with researchers across campus to develop and champion a university-wide research computing strategy in support of Harvard's research mission. He is focused on identifying emerging needs, engaging with faculty, school, and university leadership to articulate those needs, and identifying possible solutions and funding mechanisms. He is spearheading the implementation of these initiatives and articulating their success with concrete measures.

Inhalt

Chapter 1. Introduction.- Part I. The Need, the Opportunity and the Solution.- Chapter 2. The Need for Complete Data Provenance.- Chapter 3. The Opportunity: Advancement of Workflow Automation.- Chapter 4. Solution: Descriptive Dataflow Operators.- Chapter 5. Language Design.- Part II. How to Implement It.- Chapter 6. Proof of Concept Implementation.- Chapter 7. Sample Application: Building ML-Ready.- Chapter 8. Dorieh Medicare Claims Data Pipeline.- Chapter 9. What is Next?.- Part III. The Architecture of Trust: Regulation, Provenance and Compliance-As-Code.- Chapter 10. Trust and Regulation as Delegated Understanding.- Chapter 11. The Burden Spiral: When Technology Exceeds Human Oversight.- Chapter 12. The Provenance Imperative.- Chapter 13. From Provenance to Computable Trust.- Part IV. Classification of Data Transformations.- Chapter 14. A Taxonomy of Data Transformations.- Appendix.