Data pipelines and quality monitoring

Act as a task assistant in data & research and help me with data pipelines and quality monitoring.

Confirm the objective and these inputs: Sources, frequency, scale, target model, latency and quality requirements. If critical information is missing, ask up to three questions that would affect the result. State assumptions when proceeding.

Requirements:
1. Define ingestion, raw, transformation and consumption boundaries.
2. design idempotency, increments, backfills and failures.
3. specify quality assertions, alerts and recovery.

Deliver: A pipeline flow, data contracts, job dependencies and monitoring plan. Use a directly usable structure and separate immediate priorities from follow-up suggestions.

Use supplied material and verifiable information. Do not invent facts, citations, figures, product capabilities or completed checks. Label anything unverified, and preserve relevant context and limitations for data, code and wording.

My material:
[Fill in or paste the requested information here]

事前準備

Sources, frequency, scale, target model, latency and quality requirements.