During the dual online event on November 1st and 2nd, Snowflake unveiled a deluge of updates, providing insights into the development status of features announced earlier in the year at SUMMIT in Las Vegas. This event, tailored for diverse global audiences, served as a platform for Snowflake to provide crucial updates on features unveiled earlier this year at SUMMIT in Las Vegas and to introduce an array of new functionalities.
Christian Kleinerman, Senior Vice President of Product at Snowflake, took center stage to unveil a myriad of innovative features in the Snowflake ecosystem. Let’s delve into each segment:
- Data Foundation: Elevating the Core of the Platform
- Robust Snowflake Data Governance: Introducing SNOWFLAKE HORIZON
- Cost Management: Enhancing Financial Control
- Machine Learning Integration: A Paradigm Shift
- SNOWFLAKE CORTEX: Elevating AI Accessibility
- Scale with Applications: Native Integration Evolution
Data Foundation: Elevating the Core of the Platform
The Data Foundation segment stood out for its implications in everyday platform use, introducing interface enhancements and features designed to simplify user interactions.
Unistore, for mixed OLTP and OLAP Workloads
Unistore, initially introduced as a new Workload at the SUMMIT, progressed to a Public Preview of HYBRID TABLES, scheduled for late 2023. This feature allows the fusion of OLTP tasks with traditional OLAP usage, enabling real-time analytics. For instance, an e-commerce company could leverage Unistore to record user interactions in real-time and fuel real-time analytics, enhancing the personalization of the user experience.
Furthermore, Unistore has the potential to streamline operations, reducing the number of databases and pipelines by allowing fast single-point operations in Snowflake. This, in turn, accelerates the development of new features and use cases.
More Data Lake versatility with, managed or not, iceberg tables
Another highly anticipated feature is the integration with Apache ICEBERG TABLES standard, entering Public Preview with two integration options: UNMANAGED and FULLY MANAGED ICEBERG TABLES. This integration provides additional versatility to all workloads within the platform, whether in Data Warehousing, Data Lakes, or Data Lakehouses.
Performance Improvements everywhere for traditional workloads
Focusing on corporate DATA LAKES, Snowflake announced the General Availability of features for handling semi-structured data and introduced Dynamic File Processing with Snowpark in Python and Scala. The improvements extend to traditional DATA WAREHOUSES with innovations like automatic clustering cost estimation, materialized view refresh enhancements, and support for INSERT statements for the query acceleration service, all in private preview.
These enhancements, often unnoticed by the common eye, significantly contribute to the enhancement of the SNOWFLAKE PERFORMANCE INDEX. This index, incorporated into the platform’s web interface since August 2022, indicates a 15% improvement in platform performance compared to a year ago.
Robust Snowflake Data Governance: Introducing SNOWFLAKE HORIZON
The highlight of this block revolves around the Data Governance framework named SNOWFLAKE HORIZON, addressing Compliance, Security, Privacy, Interoperability, and Access—the five pillars of Horizon. This framework aims to provide control over data across all available Snowflake platforms and regions, ensuring compliance with local and international standards.
Many announcements were made regarding the Security of the platform:
- ENHANCED NETWORK SECURITY (Public Preview): Improvements in the management of blacklists and whitelists for IP access.
- IMPROVED AUTHENTICATION (Public Preview coming soon): Enhancements in authentication mechanisms.
- DATABASE ROLES (General Availability): Introduction of new roles at the database level for expanded Role-Based Access Control (RBAC) capabilities.
However, what stood out for me in terms of Security Governance was the announcement of CIS: SNOWFLAKE FOUNDATION BENCHMARK. This serves as a catalog of recommendations and best practices, ensuring a consistent and robust security policy. Coupled with the upcoming TRUST CENTER (Private Preview coming soon), and a new section in the graphical interface, Snowflake is bolstering its security and compliance features.
Privacy also takes center stage with the announcement (still in development) of DIFFERENTIAL PRIVACY POLICIES. These policies aim to add layers of “noise” to data, making it less identifiable as granularity increases, contributing to improved data privacy.
In terms of Interoperability, features already mentioned in other sections of the Keynotes focus on the platform’s compatibility with new cataloging standards such as external catalogs, Iceberg Tables catalogs, and access to Iceberg tables from Rest APIs for Snowpark.
The final pillar of Horizon, Access, was addressed through features like AUTO-CLASSIFICATION (still in development) and CUSTOM CLASSIFIERS (Private Preview), both leveraging the capabilities of LLM and AI to assign custom classifications to objects automatically.
SNOWFLAKE COPILOT (in Private Preview) caught my attention for its potential to bring natural language interaction features, akin to ChatGPT, through LLM. This promises to facilitate building SQL statements from natural language requests, ushering in a new era of interaction within Snowflake’s graphical environment.
Cost Management: Enhancing Financial Control
The introduction of the COST MANAGEMENT INTERFACE empowers administrators with tools for enhanced financial control. Notable features include COST INSIGHTS, offering real-world examples for optimization, and the BUDGETING view (in Public Preview on AWS), providing a comprehensive overview of budgeted and actual spending.
Visibility, through charts and dashboards showing workload metrics per warehouse, is useful for evaluating scaling or considering increasing capacity through multi-clustering. Cost-per-query charts to identify queries that may need reengineering and more.
Putting efforts into Optimization, they focus on optimization through a COST INSIGHTS section, providing real use cases in our account where a situation that could be improved has been detected. It offers an explanation of the best applicable practice for that specific case and a guide on how to apply it. It is extremely interesting and a lifesaver for many account administrators.
As a final point in this section, they revisited the BUDGETING view (in Public Preview on AWS), where users can preview and compare budgeted and actual spending over time. This budget tracking can be automated with email or message notifications when certain non-compliance thresholds are exceeded. Budgets themselves can be configured individually for resources (by Database, Schema, Table, or Warehouse).
From this point on, we delve into the trending topic of Artificial Intelligence, Large Language Models, and Machine Learning. We all know a trend is here to stay and has long become the new paradigm shift that will impact all aspects of our lives.
Snowflake wasn’t going to be left behind, reacting months ago with the acquisition of Neeva and swiftly incorporating the expertise of its professionals into the core of the Data Cloud.
Machine Learning Integration: A Paradigm Shift
Snowflake’s foray into Machine Learning (ML) was marked by the imminent entry into the General Availability of the ML MODELING API. Key features such as FEATURE ENGINEERING, TRAINING, SNOWFLAKE MODEL REGISTRY, and SNOWFLAKE FEATURE STORE (in Private Preview) promise to redefine ML capabilities within the Snowflake platform.
To aid the transition for Data Analysts and Data Scientists accustomed to conducting their samplings and training from a Notebook, the folks at Snowflake will introduce their own interface, SNOWFLAKE NOTEBOOKS (In Private Preview). Developed in Streamlit, it will provide the characteristic development in cells that can contain Python code, Streamlit, SQL, and Markdown.
For more complex ML applications where this API falls somewhat short in terms of functionalities and power, they propose the use of SNOWPARK CONTAINER SERVICE (Public Preview Soon). In this approach, the complete application developed in Snowpark is published in a container similar to Kubernetes and runs within Snowflake, leveraging its optimized capabilities for flexible computing.
From my point of view, it’s fantastic to see how Snowflake incorporates all these functionalities that traditionally have had a high burden of infrastructure management or a lot of experience in configuring and deploying systems (such as installing compatible Python libraries, deploying containers, etc.) effortlessly as a genuine SaaS platform should.
SNOWFLAKE CORTEX: Elevating AI Accessibility
Snowflake’s total commitment to bringing the adoption of Artificial Intelligence to its customers’ workloads (AI for Everyone) has materialized in the CORTEX engine.
Comprising many features of services managed by Snowflake (Serverless) to provide access to cutting-edge LLM and AI models in the industry easily and quickly, helping democratize their use.
SPECIALIZED FUNCTIONS (In Private Preview): These functions aim to incorporate Translation, Sentiment Analysis, Summarization, and Extraction of Answers capabilities into queries and applications.
GENERAL FUNCTIONS, incorporating existing functions in leading LLM standards, such as LLAMA2, were also introduced. Well-known functions like LLM Inference, Vector as Native Data Type for Llama2, Complete, Txt2SQL EMBED_TEXT, Vector_L2_Distance, etc., are part of this offering.
The demonstration of these features showcased Snowflake’s substantial effort in making CORTEX a differentiating element compared to its competitors.
Scale with Applications: Native Integration Evolution
As a follow-up to the announcements made at the SUMMIT regarding the integration of native applications in Snowflake, they provided a few more updates:
As part of the application development lifecycle, they announced DATABASE CHANGE MANAGEMENT (in Private Preview). This feature will allow the execution of scripts directly against our account from a Git repository (thanks to the integration with GitHub announced at the SUMMIT). It even supports DML statements, and it enables us to manage CI/CD for development and the underlying Snowflake data model with incremental change analysis, etc.
Another feature related to this application integration announced at the event was the NATIVE APP FRAMEWORK (in Private Preview, soon to be in General Availability on AWS). It’s an environment for deploying and consuming native applications within the Data Cloud itself. To accelerate its development, they announced a $100 million investment in startups to assist in the development of these Native Apps.
Conclusion: A Year of Unparalleled Innovation
As we eagerly anticipate further announcements at the year-end partner event, Snowflake has undeniably delivered robust features. These developments solidify Snowflake’s position as a trailblazer in the cloud data platform space. The platform’s unwavering focus on performance, governance, AI integration, and financial control positions Snowflake as a comprehensive solution for tech experts and organizations navigating the evolving landscape of data management.
Snowflake’s continuous innovation is transforming the data landscape, and we are committed to keeping you informed at every step of this transformative journey.
