83 saved links, 2017–2026 (peak: 2024). One of the oldest threads in your vault, and one of the clearest arcs: it starts with Azure Data Lake retweets in 2017, runs through a long Kafka apprenticeship, settles into the Databricks/Snowflake lakehouse era — and then, from 2023 on, stops being about moving data and starts being about feeding it to AI. By 2026 nearly every new save here is really an AI Agents save wearing a data-engineering hat. That’s not tag drift; that’s the thesis of your Enterprise AI Architect arc — agents are only as good as the data platform underneath them.
Related: Azure · Databases · Data Science · Machine Learning · MLOps
Where it started: the Azure data plumbing years (2017–2020)
The earliest layer — retweet-era saves from your Azure days, when “data engineering” meant Data Factory, Data Lake, and Databricks-on-Azure. Historical strata more than reading list, but they explain where your platform instincts come from.
- Azure Data Lake online training — saved in September 2017, then again three weeks later. The tag’s very first entries.
- Application patterns using Azure SQL Data Warehouse — pre-Synapse warehouse thinking.
- The Azure Databricks onboarding run: intro to ML for developers on Azure Databricks, accessing your data, running your first code, and the free-tier signup — plus early ML-on-Spark saves: TensorFlowOnSpark on Databricks, sentiment analysis on streaming data, MLflow + TensorFlow + Keras with PyCharm.
- Architecture patterns of the era: Azure Event Grid in a modern data warehouse · Lambda architecture with Cosmos DB and Databricks · the simplified Lambda take · massively scalable IoT pipelines on Cosmos DB · a serverless big-data pipeline powered by a single Azure Function.
- The DevOps crossover — your two worlds meeting early: DevOps in Azure with Databricks and Data Factory and CI pipelines for Azure Databricks (video).
- Pipeline-builder milestones: Cathrine Wilhelmsen’s ADF blog series — saved off a tweet celebrating a first ever Data Factory pipeline built from it. Later: Managed Airflow lands in ADF · the ADF / Databricks / Synapse comprehensive guide · Durable Functions vs. Apache Airflow — orchestration compared honestly, by the Durable Functions author.
- Late echoes: the Azure Databricks training video series and migrating an on-prem data pipeline to Azure.
The Kafka thread
The most persistent single technology in this tag — a decade-long apprenticeship that ends somewhere unexpected: Kafka as the event backbone for multi-agent systems.
- Apache Kafka in 15 minutes, with Neha Narkhede — the 2019 entry point.
- exactly-once processing in Kafka — the semantics everyone gets wrong.
- ByteByteGo: why is Kafka so fast? — and their deep dive on Cloudflare’s trillion-message Kafka infrastructure, saved a year later under the same source.
- Production war stories: kafka-in-production — tech blogs from companies running Kafka at scale · Allegro tackling Kafka tail latency (the fix: migrate brokers from ext4 to XFS).
- kafka-zero-to-production — complete learning journey with Docker Compose cluster and production scripts; the DevOps-friendly on-ramp.
- Streaming meets AI: real-time anomaly detection with Kafka and vectors · Confluent on getting started with generative AI · a real-time news search engine with serverless Kafka and a vector DB.
- The agent-era payoff: why event-driven multi-agents with Kafka, Flink & LangChain and Kai Waehner: agentic AI with A2A and MCP, Kafka as the event broker — “real foundations are forming,” and they look like the streaming platforms you already know.
Lakehouse & architecture war stories
The platform layer: Databricks and Snowflake as they matured, plus the reference architectures worth stealing.
- Ali Ghodsi on the evolution of data architectures — “one of the best conversations I’ve ever had on the future of data,” says the save; business, architecture, and operations in one podcast.
- Uber’s real-time data infrastructure journey — the paper-length war story, “a fascinating read for all data engineers.”
- ML reference architectures from Google, Facebook, Uber, and Databricks · a guide to data-driven design and architecture · TraditionalModernDW — a data warehouse architecture that “just works” (KISS, delivered consistently).
- Databricks in daily practice: lakehouse orchestration with Workflows · applying software development and DevOps best practices to Delta Live Tables — the save that treats pipelines as software, which is exactly your instinct.
- Google on systems engineering as the foundation of SRE — reliable data pipelines and SLOs in the same breath; the bridge between this tag and your DevOps life.
- Snowflake’s arc in three saves: the 2021 Summit → running Llama 2 in Snowpark Container Services (the demo failed live — “had so much fun” anyway) → Cortex LLM functions go GA (“makes you feel like you got superpowers”).
- Databricks Assistant goes GA — the first hint of where the next section goes.
Data meets GenAI (2023–2024)
The pivot year is visible in the timestamps: from mid-2023, “data engineering” in your vault means engineering data for LLMs. This is the RAG supply chain viewed from the platform side.
- improving RAG response quality with real-time structured data — Feature & Function Serving; unstructured retrieval plus fresh structured context.
- long-context RAG performance — Matei Zaharia and co. on why long context isn’t a silver bullet, because LLM reasoning mimics human reasoning.
- RAFT on top of Databricks fine-tuning to outperform RAG — retrieval-aware fine-tuning, applied.
- personalizing LLM apps with LangChain and Tecton — the feature store meets the prompt.
- Platform mechanics: LLM inference performance engineering best practices · Mosaic AI Gateway — governance and credential management in front of the LLM API · Mosaic AI’s compound-AI launch: Agent Framework, Tool Catalog, Vector Search, evals.
- Governance for the whole stack: the Big Book of MLOps, updated for generative AI and managing AI security risks — the Databricks CISO workshop.
Agentic data engineering (2025–2026)
The current chapter, and the densest overlap with your day job ambitions: every major data platform shipping agents that do data engineering, and data engineers learning to evaluate them.
- “Why traditional ETL is dead — the rise of agentic data engineering” — the manifesto version of the shift.
- The platform agents: the BigQuery Data Engineering Agent (“easy-to-follow examples that even came with their own data”) · Google’s open-source Data Agent Kit for your IDE or CLI · ontology-grounded reasoning with Snowflake Cortex Agents · Snowflake’s practical guide to AI agents.
- How Databricks builds agents on itself — the AI-on-Databricks series: Mosaic AI, tools, and function calling · the agentic AI workload analyzer · agent evaluations in LLMOps with GitHub Actions · generative AI for retail media optimization — plus the coSTAR pattern for shipping agents fast without breaking things.
- how agentic software development will change databases — agentic coding changes infrastructure requirements; “every service will have these capabilities over time.” Read this next to your agentic-DMS notes.
- Robin Moffatt: evaluating Claude’s dbt skills — building an eval from scratch — the craft of judging an AI data engineer; directly reusable for your own agent evals.
- Context: InfoQ’s 2025 AI, ML and data engineering trends report · Databricks’ first student-fellow cohort for agentic AI.
Learning the craft
The credential-and-curriculum shelf — mostly pre-AI fundamentals, which is what makes it durable.
- the data engineering handbook — “super helpful”: certifications, courses, communities, whitepapers, podcasts, all in one repo.
- the official Databricks data engineer learning path.
- Zach Wilson’s DataExpert material, three ways: the LLM-driven data engineering lecture with its hands-on lab (free, ~2 hours), and the scale stories — processing 2,000 TB/day at Netflix with Spark and Airflow · data lake modeling at Airbnb: 100 TB into 5 TB with Parquet and run-length encoding.
- Real Python on what a data engineer actually is — saved with an admission of “throwing around the term without fully grasping the domain.” Honest starting points age well.
- the Google Professional Data Engineer certification guide — the cert path from your cloud-credential era.
- The GenAI upgrade path: Generative AI for Data Engineers (Coursera specialization) · Databricks’ Generative AI Fundamentals videos · free courses spanning data engineering to LLMOps.
- Small but useful: a lightweight ETL pipeline with Airtable and Python (free tier only) · the Databricks real-life ML examples ebook — the 2018 ancestor of everything above.