Nimbly Logo
Snowflake: A Comprehensive Solution for Modern Data Management

Snowflake: A Comprehensive Solution for Modern Data Management

Chris Ballment, April 03, 2023.

As the world becomes increasingly data-driven, businesses are constantly exploring innovative ways to manage and utilise their data. Snowflake is a platform that stands out for its cutting-edge features and adaptability to modern data principles. It is for this reason that it is recommended in Nimbly’s target architecture, and supported by our data accelerators for Snowflake.

In this article, we will delve into various themes, such as deployment, accessibility, multi-cloud support, scalability, data lake integration, and transformation capabilities, and discuss how Snowflake’s features support and enable these key areas.

Deployment: Streamlining and Efficiency

Snowflake's "Zero Copy Clone" [1] feature significantly improves the testing process in build automation. By programmatically creating a zero-copy clone of the production environment, developers can deploy code changes (e.g., SQL, dbt, etc.) to an exact replica of the current production database and run tests without any impact on the production database. As the ZCC name suggests, this clone is not a physical copy of the data, but rather references immutable data in-place from the source database, quickly and cost-effectively.

Additionally, Snowflake's compatibility with Terraform [2] enables organisations to employ Infrastructure-as-Code (IaC) for consistent and reliable implementation and deployment of infrastructure across multiple environments. This approach streamlines the usually convoluted process of managing environments and also minimises errors, enhancing deployment efficiency.

Increased Accessibility: Empowering Users and Developers

Snowflake uses standard SQL commands for data access, requiring no other skills which makes data within the platform highly accessible. The increasing use of SQL for both developers and business users alike means that more people are able to access data without any further uplift in technical skills.

If Snowflake is used as a data lake, analysts, developers, or consumers no longer need to be proficient in Spark, Presto, or other frameworks of this nature. By using a single homogenous skill set for most components and enabling all the data to be stored in the one platform, users can access and manage their data without leaving the platform which increases productivity and lowers the barrier to entry for new users of the platform.

Multi-Cloud: Flexibility and Innovation

Snowflake's availability across the three major cloud service providers (AWS, Azure and GCP) ensures that organisations can choose their preferred cloud provider without sacrificing the benefits of the platform.

Additionally, Snowflake has a new feature in private preview called Iceberg tables [3], which will offer first-party table support for Parquet and Iceberg storage. This will allow standard CRUD (create, read, update, and delete) operations with similar performance to native Snowflake tables, while reducing vendor lock-in. This will also give organisations with highly sensitive data or regulatory constraints the flexibility to keep their data within their existing cloud tenancy instead of housing it within Snowflake's storage layer.

Scalability: Performance and Cost-Effectiveness

Snowflake's hybrid of the "shared disk" and "shared nothing" architectures [4] sets it apart from traditional platforms by isolating compute and storage resources. This means that even the most resource-intensive workloads cannot impact other workloads on the platform by design.

Snowflake offers auto-scaling, both horizontally and vertically so capacity can more accurately match the demands of the system. Additionally, Snowflake's compute model, resource monitoring capabilties and "auto-suspend" feature ensures that users only pay for what they use, making it a cost-effective solution for organisations of all sizes.

Storage: Versatility and Affordability

One of Snowflake's major advantages is its storage pricing, which is practically the same as the underlying cloud service provider's blob storage costs (e.g., AWS S3, Azure Blob Storage and GCS). This means that organisations can store data in Snowflake as cost-effectively as they can in a cloud provider’s blob storage service, in some cases moreso as Snowflake also compresses its data. As a result, some modern data architectures that include Snowflake use it as both the analytical database and the data lake.

Furthermore, Snowflake's variant data type enables storage of semi-structured formats (e.g., JSON, Avro, ORC, Parquet, or XML). This enables organisations to store raw semi-structured data and apply the schema-on-read, parsing attributes out from the variant columns during consumption. This removes the traditional need to perform analysis and modelling prior to storing data in a database.

Transformation: Automation and Scalability

Snowflake offers several features that enhance its transformation capabilities. The platform supports automatic ingestion from blob storage with its "Snowpipe" feature [5], ensuring that new files are automatically ingested by Snowflake.

The “Time Travel” feature [6] allows the querying of data from an object “as at” a prior point in time. This duration can be set from 1 up to 90 days worth of history for a table with Time Travel enabled. It is helpful for both data recovery (if a table was updated incorrectly, truncated, or dropped) and for comparing the state of a table at two points in time (enabling a reliable change data capture process by performing a logical minus of the prior data from the current data set).

Another feature, Snowflake "Streams" [7] generates a list of DML operations like an Oracle Redo log without impacting insert, update, or delete operations on the source table. This feature enables transformations to only process data with changes since the last stream consumption, negating the need for complex change detection logic or “high water marks.”

Scheduled transformations are also possible with Snowflake Tasks [8], which can be executed using SQL, Snowflake's scripting language (like Oracle PL/SQL), or by calling stored procedures. This feature allows the co-location of transformations in the database, and the use of internal scheduling prevents the need for other orchestration tools. Tasks also run serverlessly, negating the need for provisioning transformation compute.

Finally, the Snowpark [9] library offers an additional layer of transformation capabilities. The library allows developers to query and process data at scale within Snowflake using supported languages such as Java, Python, and Scala. This means that complex data processing can be performed without moving data to another system, making the process more secure and efficient. By leveraging the elastic and serverless Snowflake engine, organisations can process data at scale with ease.

Conclusion

Snowflake is a powerful and versatile platform that enables and supports modern data principles. Its robust deployment features, increased accessibility, multi-cloud support, scalability, seamless integration with data lakes, and advanced transformation capabilities make it an excellent choice for organisations looking to optimise their data management processes.

In a world where data is increasingly becoming the lifeblood of businesses, Snowflake offers a future-proof solution that can adapt to the ever-evolving landscape of data management. By embracing Snowflake's cutting-edge features, organisations can empower their teams to unlock the full potential of their data and drive innovation in their respective industries.

Discover the power of Snowflake with Nimbly, a trusted Snowflake partner. Join us on the journey to transform your data management today!

References