Fabric Runtime 2.0: Unleashing Advanced Data Engineering in Microsoft Fabric

Fabric Runtime 2.0 Experimental (Preview) is designed for advanced data engineering and data science. The new runtime empowers teams to process big data workloads efficiently and unlock scalable analytics on the Microsoft cloud platform. For data professionals seeking innovation, Fabric Runtime 2.0 signals a bold step forward in Spark-based development.

Major Upgrades and Technology Highlights

• Apache Spark 4.0: Delivers productivity and performance enhancements, making Spark workloads faster and more intuitive.
  
  
• Delta Lake 4.0: Supports enhanced features for open lakehouse architectures and improved interoperability.
  
  
• Azure Linux 3.0 (Mariner 3.0): Provides a secure, modern foundation for cloud-native operations.
  
  
• Latest Language Support: Java 21, Scala 2.13, and Python 3.12 ensure compatibility with today’s coding standards.

These updates offer seamless integration with Microsoft Fabric, enabling users to harness the latest APIs and data engineering features.

Getting Started: Enabling Fabric Runtime 2.0

Adopting Runtime 2.0 is simple and flexible, giving organizations control over deployment:

• Workspace Level: Enable Runtime 2.0 by updating workspace settings, applying it across all Spark workloads.
  
  
• Environment Item Level: Assign Runtime 2.0 to specific notebooks or Spark Job Definitions, overriding workspace defaults as needed.

Note that session startups may take 2-5 minutes due to the preview’s experimental status.

Preview Features and Current Limitations

As an experimental public preview, Fabric Runtime 2.0 offers early access to advanced Spark and Delta Lake capabilities. However, several important limitations exist:

• Supports Spark 4.0 sessions, PySpark, Scala, and Spark SQL; R language is not supported.
  
  
• Library installation via pip and conda is available; Spark settings can be configured in notebooks and job definitions.
  
  
• Delta Lake 4.0 supports basic reads/writes; advanced features such as V-order and time travel are not yet enabled.
  
  
• Spark Advisor is unavailable, but Spark UI and logs offer monitoring alternatives.
  
  
• Integrations with Copilot and many data connectors are not currently supported.
  
  
• PySpark offers a basic Conda environment without additional data science libraries.
  
  
• No Visual Studio Code integration or support for some Azure Storage protocols (WASB/ABFS).

Microsoft encourages community feedback to guide future development and prioritize enhancements.

Innovations in Apache Spark 4.0 and Delta Lake 4.0

Apache Spark 4.0

Spark 4.0 introduces significant advancements:

• Spark SQL: New data types, user-defined functions, session variables, and improved syntax.
  
  
• PySpark: Native plotting API and unified profiling for user-defined functions.
  
  
• Structured Streaming improvements for more robust state management and debugging.
  
  
• Deprecation of SparkR, streamlining supported language options.

These changes increase expressiveness and productivity for developers.

Delta Lake 4.0

Delta Lake 4.0 enhances performance and interoperability for open lakehouses:

• Optimized for Spark experiences like notebooks and job definitions.
• Expanded performance features, with evolving support across Microsoft Fabric workloads.

Before enabling new Delta features, review interoperability documentation to ensure compatibility with your environment.

Shaping Tomorrow’s Data Engineering

Fabric Runtime 2.0 Experimental (Preview) positions Microsoft Fabric as a leader in scalable, cloud-based data engineering. While some features are still in development, this runtime offers an exciting opportunity for professionals to experiment, innovate, and contribute feedback. By engaging early, users help drive the evolution of a robust, production-ready platform for the future.

Related Microsoft content

• Apache Spark Runtimes in Fabric - Overview, Versioning, and Multiple Runtimes Support
• Spark Core migration guide
• SQL, Datasets, and DataFrame migration guides
• Structured Streaming migration guide
• MLlib (Machine Learning) migration guide
• PySpark (Python on Spark) migration guide
• SparkR (R on Spark) migration guide

Source: Microsoft Learn – Runtime 2.0 in Fabric