Traditional data warehouses coupled compute and storage on the same nodes. Running a large query required buying more hardware: more CPU and more storage together, even if you only needed more CPU for a week. Compute-storage separation decouples them: data lives in cheap object storage (S3, GCS, Azure Blob), and compute clusters spin up on demand to query it. Pay for compute only while queries run. Scale each dimension independently.

Architecture#

Storage layer: immutable columnar files on object storage. S3 at $0.023/GB/month. No provisioning, no maintenance, infinite scale. All compute clusters read from the same storage layer, so there’s no data movement when scaling compute.

Compute layer (virtual warehouses): clusters of nodes that execute queries. Snowflake calls these virtual warehouses. You can have multiple warehouses simultaneously: one for BI dashboard queries, one for ETL jobs, one for a data science team running experiments. They don’t interfere with each other because they’re reading from shared immutable storage, not shared mutable disk.

Auto-suspend: a warehouse with no queries suspends after N minutes of idle time. Credits stop accumulating. Resume is fast (30-60 seconds for a small cluster). Run queries only when needed, pay only then.

graph TD A[Shared Object Storage: S3 Parquet files, immutable] --> B[Compute cluster 1: BI dashboards, 4 nodes] A --> C[Compute cluster 2: ETL jobs, 16 nodes] A --> D[Compute cluster 3: Data science, 8 nodes] B --> E[Local SSD cache: hot data for repeated queries] C --> F[Local SSD cache: ETL working set] D --> G[Local SSD cache: ML feature data] E --> H[Dashboard query result] F --> I[Transformed silver layer written back to S3] G --> J[Feature extraction complete] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style J fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

The Local Cache Layer#

Object storage latency (50-100ms per request) would make query performance terrible without caching. Each compute node has local SSD that caches recently read file chunks. On the first query, data reads from S3 at 500MB/s per node. On the second query for the same data, reads from local SSD at 3-5 GB/s.

Cache invalidation: since storage is immutable (new data writes new files, doesn’t overwrite old ones), cache entries remain valid indefinitely. A file once written to S3 never changes. No cache invalidation logic needed.

Consistent hashing distributes file chunks to the same nodes across queries. If 16 nodes each cache different parts of the dataset, a query requiring the full dataset touches all 16 nodes in parallel rather than causing all nodes to independently re-read the same popular files from S3.

Scaling Up and Scaling Out#

Scale up: use a larger warehouse (more nodes per cluster) for a single query that needs more parallelism. Partition the query across more nodes.

Scale out: spin up additional warehouses for additional workload types. Multiple teams run queries simultaneously with no contention on storage or compute.

This is the fundamental advantage over shared-nothing MPP databases (Redshift, traditional Teradata): adding a second workload type doesn’t require provisioning a second cluster with its own storage copy.

At Salesforce#

Salesforce’s analytics infrastructure used a shared-nothing MPP cluster for customer usage data. During end-of-quarter reporting season, dashboard queries and ETL jobs competed for the same nodes. ETL jobs starved dashboards of resources: p99 dashboard query time went from 8 seconds to 4 minutes. After migrating to compute-storage separation with dedicated virtual warehouses per workload type, ETL and dashboards stopped competing. Dashboard p99 query time stayed under 12 seconds through end-of-quarter.

What I’m Learning#

Compute-storage separation solves the multi-tenancy problem for analytics: different workloads with different resource profiles can coexist without interference. The local SSD cache is what makes it performant despite the latency of object storage. Immutability of the storage layer is what makes the cache correct without invalidation logic.

Have you dealt with workload interference in a shared analytics cluster, and how did you isolate workloads from each other?