News | Joshua Berkowitz

12 Articles

data engineering ×

Amazon MWAA Serverless: Simplifying Workflow Orchestration for Data Teams

Managing Apache Airflow clusters has historically been complex and resource-intensive. The introduction of Amazon Managed Workflows for Apache Airflow (MWAA) Serverless changes that equation by lettin...

Airflow AWS cloud computing cost optimization data engineering security serverless workflow automation

Nov 18, 2025

0 4235

Conversational AI and AWS MCP Are Revolutionizing ETL Pipelines

In the fast-evolving world of data engineering, the demand for efficient ETL (Extract, Transform, Load) processes continues to grow. Traditionally, building robust ETL pipelines involved manual script...

Amazon Q AWS conversational AI data engineering ETL Model Context Protocol Redshift S3 Tables

Nov 15, 2025

0 1188

Google BigQuery’s Data Engineering Agent Is Automating the Future of Analytics

Manual pipeline maintenance can drain your team’s creativity and slow down innovation. Google’s new Data Engineering Agent in BigQuery, now in preview , aims to automate routine data engineering tasks...

AI agents automation BigQuery cloud analytics data engineering data migration pipeline management

Nov 10, 2025

0 3905

Hugging Face Introduces Dataset Streaming for Machine Learning

If you’ve ever been frustrated by long waits to download massive datasets for model training, you’re not alone. Hugging Face has introduced a groundbreaking way to stream multi-terabyte datasets direc...

data engineering dataset streaming huggingface machine learning parquet performance optimization Xet storage

Nov 7, 2025

0 9548

Empowering Developers: Microsoft Fabric Extension for VS Code Goes Open Source

Microsoft is inviting developers to take the reins with the open-source Fabric Core extension for Visual Studio Code. This strategic move not only demonstrates the company’s commitment to transparency...

community data engineering developer tools extensions GitHub Microsoft Fabric open-source VS Code

Oct 30, 2025

0 1760

Databricks Data Orchestration: Table Update Triggers in Lakeflow Jobs

Start eliminating the guesswork from your data workflows where pipelines launch the moment fresh data lands, and insights never lag behind reality. With Databricks’ latest table update triggers in Lak...

automation Databricks data engineering data orchestration event-driven Lakeflow Jobs table triggers Unity Catalog

Oct 21, 2025

0 9680

Docker MCP Toolkit Supercharges Codex: From Code Generation to Data Engineering

Today’s AI coding assistants are impressive, but what if they could do more than just suggest code? By connecting OpenAI’s Codex to Docker’s Model Context Protocol (MCP) servers with the Docker MCP To...

AI assistants automation Codex data engineering Docker knowledge graphs MCP Toolkit Neo4j

Oct 20, 2025

0 8019

dbt Agents and the Remote MCP Server Are Shaping Trusted AI Analytics

Analytics is entering a new era, driven by AI-powered agents and robust governance frameworks. dbt Labs recently unveiled the remote dbt MCP Server alongside dbt Agents, a suite of secure, task-specif...

AI agents analytics data engineering dbt Fusion engine governance MCP server self-service analytics

Oct 15, 2025

0 3377

Databricks Variant: The Open Standard For Semi-Structured Data in the Lakehouse

Organizations increasingly face the challenge of managing vast amounts of semi-structured data, such as logs and telemetry, in analytics and AI workflows. Historically, teams had to choose between slo...

Apache Iceberg data engineering data performance Delta Lake lakehouse Parquet semi-structured data Variant

Oct 10, 2025

0 3553

Hugging Face’s FinePDFs Dataset For AI Training

AI research has long relied on web-scraped content, but Hugging Face’s FinePDFs dataset is set to change the landscape. By sourcing over 475 million documents directly from PDFs, often considered too ...

AI data engineering datasets Hugging Face language models machine learning open source PDF

Sep 19, 2025

0 9119

AI-Driven Vibe Coding and Rust Are Changing Data Engineering

Building high-performance, reliable data tools is more crucial than ever and Rust is at the forefront of this movement. Thanks to the surge in AI-powered development, a new approach called vibe coding...

AI coding asynchronous processing data engineering parallel computing performance optimization Rust vibe coding

Sep 9, 2025

0 5731

Rust Is Disrupting Data Engineering and Outpacing Python and Java

Data engineering is undergoing a quiet revolution. Rust, a language once known mainly to systems programmers, is rapidly gaining traction in data engineering and challenging Python and Java’s dominanc...

data engineering ELT pipelines Java memory safety performance Python Rust software development

Jun 30, 2025

0 6127

Our latest content

Check out what's new !

See all

Ads

Prompt Maker Image Generator

Struggling with the perfect AI image prompt? My free app helps you generate brilliant ideas and instantly creates an image to match. Go from concept to creation in two clicks!

Try It

Most Popular Articles

Check out what the hot topics are!

See all

Follow us

Our latest content

Prompt Maker Image Generator

Most Popular Articles

Every shirt tells a story—and every story

#ClothingForACause