Topics Covered

On this page

1. The unit where the money is
2. 3.1 The three storage types — block-based, file-based, object-based
3. 3.2 Object storage, in detail
4. 3.3 Storage classes, and the arithmetic that decides
5. 3.4 Egress, and data gravity
6. 3.5 Use cases
7. 3.6 Key-value databases
8. 3.7 Batch against streaming for ML pipelines
9. 3.8 Cloud data warehouses
10. Practice problems
11. Exam questions from this unit
12. Mistakes that cost marks

Syllabus topics: Cloud storage: introduction, benefits, use cases (backup, archiving, disaster recovery, content delivery). Cloud storage systems: block-based, file-based, object-based. Key-value databases: features and limitations. Batch against streaming data for ML pipelines. Cloud data warehouses: AWS Redshift, Google BigQuery.

---

The unit where the money is

Storage decisions are where cloud bills are made and lost, and they are the most examinable arithmetic in the course. Every figure below is computed in 04_storage.py and 09_etl_warehouse.py.

---

3.1 The three storage types — block-based, file-based, object-based

The benefits of cloud storage are durability you do not engineer, capacity you do not provision, and access from anywhere. What you choose between is the shape.

---

3.2 Object storage, in detail

⚠️ There are no directories. None.

LIST prefix 'raw/' with delimiter '/':
  objects at this level : (none)
  common prefixes       : ['raw/2026/']

LIST prefix 'raw/' with NO delimiter:
  raw/2026/01/sales.csv
  raw/2026/02/sales.csv
  raw/2026/03/sales.csv

raw/2026/01/sales.csv is ONE KEY containing three slashes. The console's folder tree is drawn from common prefixes computed at list time. Delete every object under a "folder" and the folder is gone, because it never existed.

And a prefix scan is the only query an object store supports. No WHERE, no index, no search by content.

⚠️ There is no rename

'rename' README.md -> docs/README.md
  bytes read 1,024, bytes written 1,024, API calls 2

It is a copy plus a delete. Renaming a 5 TB dataset "to tidy up the folders" moves 10 TB and is billed for it — and on a filesystem it would have been a metadata edit.

⚠️ Versioning bills for every version

versioning ON, then overwrite and delete:
  older versions kept : 2
  current object      : DeleteMarker

A delete with versioning on writes a delete marker; the data is still there and still billed. That is the feature (you can undelete) and the bill (you pay for every version until a lifecycle rule removes them).

Set the lifecycle rule when you enable versioning, not later.

---

3.3 Storage classes, and the arithmetic that decides

⚠️ And the reversal that catches people

The same 1 TB, retrieved twice a month:

Class	Storage/yr	Retrieval/yr	Total
Standard	$282.62	$0.00	$282.62
Standard-IA	$153.60	$245.76	$399.36
Glacier Instant	$49.15	$737.28	$786.43

Standard is now the cheapest, by a wide margin. The "cheap" tiers charge per GB retrieved, and at two retrievals a month the retrieval fee exceeds everything the storage discount saved.

Storage class is a bet on your access pattern, and the minimum durations and retrieval fees are what make the bet real.

---

3.4 Egress, and data gravity

Transfer	Cost
1 TB in from the internet	$0.00
1 TB out to the internet	$92.16
1 TB between availability zones	$20.48
1 TB S3 → EC2, same region	$0.00

---

3.5 Use cases

Use case	What it needs	Right service
Backup	cheap, write-mostly, occasional restore	S3 Standard-IA → Glacier by lifecycle
Archiving	very cheap, rarely read, long retention	Glacier Deep Archive
Disaster recovery	cross-region, tested restores	S3 Cross-Region Replication
Content delivery	low latency, high egress volume	S3 + CloudFront
Data lake	cheap, queryable in place	S3 + Parquet + Athena/Glue

⚠️ The DR distinction that is examined

Term	Means
RPO — Recovery Point Objective	how much data you can afford to lose
RTO — Recovery Time Objective	how long you can afford to be down

An RPO of zero requires synchronous replication, which costs latency on every write. An RTO of minutes requires a warm standby, which costs money while nothing is wrong. Both are business decisions with a price, and "we back up nightly" is an RPO of 24 hours whether or not anyone said so.

And a backup you have never restored is not a backup. It is an untested hypothesis.

---

3.6 Key-value databases

⚠️ The limitations that decide against it

Limitation	Consequence
No ad-hoc queries	every access pattern must be designed in advance
No joins	you denormalise, and maintain consistency yourself
No aggregation	counts and sums are maintained as counters, or exported
Item size caps	DynamoDB 400 KB, Cosmos 2 MB
Secondary indexes cost	storage and write throughput, per index

Design the access patterns first, then the key. In a relational database you design the schema and the queries follow; in a key-value store the queries come first, and getting that backwards is the most common failure.

The comparison across the catalogue

	DynamoDB	MongoDB (Document Oriented Database)	HBase (Big Data Technologies)
Model	key-value + document	document	sparse sorted map
Query by value	with a GSI	yes, natively	no
Managed	fully	Atlas, or self-hosted	self-hosted, mostly
Scales by	partition key	shard key	region split
Billed on	read/write units	instance	instance

---

3.7 Batch against streaming for ML pipelines

	Batch	Streaming
Arrives	on a schedule	continuously
Completeness	a whole period, consistent	whatever has landed
Late data	impossible	normal, and must be handled
Re-runnable	yes, idempotent	no — events are consumed
Latency	minutes to hours	seconds
Failure means	re-run the job	a gap in the data
Cost	bursty, predictable	continuous
Services	Glue, Dataflow batch, EMR	Kinesis, Pub/Sub, Dataflow

⚠️ And the failure that has no error message

Training on batch and serving on streaming is how training/serving skew happens. The batch job computes a feature over a complete day; the streaming path computes it over whatever arrived. The model then sees, at serving time, a feature distribution it never trained on — and every metric stays green.

A feature store exists to solve exactly this: one definition of a feature, computed once, served to both.

---

3.8 Cloud data warehouses

ETL against ELT

	ETL	ELT
Transform runs	on separate compute	in the warehouse
Lands in the DW	clean data only	raw data, then transformed
Fix a transformation bug	re-extract from source	re-run SQL on raw
Needs	an ETL server, or Glue	a warehouse that scales
Suits	limited warehouse capacity	cheap elastic compute

ELT won because warehouse compute got cheap and elastic. Landing raw data means a bug is fixed by re-running SQL rather than re-extracting from a production database that may no longer hold the old rows — which is exactly the DELETE problem Big Data Technologies found in Sqoop.

---

Practice problems

1. A 50 TB dataset is written once and read about twice a year. Which storage class, and what would you check first?

Glacier Flexible or Deep Archive, and the thing to check first is the retrieval pattern, not the size.

Per year, per TB: Deep Archive storage is $12.17 and one retrieval is $20.48. At two retrievals, retrieval already costs 3.4× the storage. Scaling to 50 TB:

• Deep Archive: 50 × ($12.17 + 2 × $20.48) ≈ $2,657/year
• Glacier Flexible: 50 × ($44.24 + 2 × $10.24) ≈ $3,236/year
• Standard: 50 × $282.62 ≈ $14,131/year

Deep Archive wins, but check two things first: the 180-day minimum (fine here — data is kept for years) and the 12-hour retrieval time. If a read must complete in minutes, Deep Archive is disqualified regardless of price, and Glacier Instant is the answer.

2. Explain why an object store cannot rename a file, and what to do about it.

Because the key is the identity. An object store is a flat map from key to bytes; there is no directory entry to edit and no inode to relink. "Renaming" means writing the object under a new key and deleting the old one — a full read and a full write.

Measured: a rename of a 1 KB object read 1,024 bytes, wrote 1,024 bytes and made 2 API calls. Scale that to 5 TB and it moves 10 TB.

What to do:

• Get the key layout right at write time. Partition prefixes (dt=2026-08-01/) should be decided before the first write.

• Never restructure a lake by renaming. Write to the new layout from the pipeline, and expire the old prefix by lifecycle rule.

• Use a table format — Iceberg, Delta, Hudi — where the layout is metadata and reorganisation does not move bytes.

3. Your BigQuery bill jumped from $40 to $900 in one month with no new data. What do you look at?

Bytes scanned, per query, per user. The INFORMATION_SCHEMA.JOBS view gives total_bytes_billed by query.

The usual causes, in order of likelihood:

1. A dashboard on auto-refresh. A SELECT * on a 10 TB table costs $62.50; every 15 minutes for a month is 2,880 runs.

2. A new query without a partition filter. The 500× difference measured above is entirely partition pruning and column projection.

3. A view hiding a SELECT *. The view looks cheap; what it scans is not.

4. SELECT * in a LIMIT 10 query. LIMIT does not reduce bytes scanned in a columnar store — it reduces bytes returned.

The fixes: --maximum_bytes_billed on every automated job, --dry_run before every large one, require_partition_filter on large tables, and materialised views for dashboards.

4. Design the storage for a model-training pipeline: raw logs arrive continuously, features are computed nightly, and eight workers train in parallel.

• Raw logs → S3 Standard, partitioned by date, in Avro or Parquet. Cheap, append-only, and the pipeline reads them once. A lifecycle rule moves them to Standard-IA at 30 days and Glacier at 90.

• Nightly features → S3, Parquet, partitioned. Column projection and predicate pushdown make the training reads cheap, and the same files serve ad-hoc analysis through Athena.

• The eight training workers → read from S3 directly. Not EFS. Each worker reads its shard; there is no shared-write requirement, so the 13× EFS premium buys nothing.

• Checkpoints during training → S3, so a spot interruption does not lose the run.

• EBS only for each worker's OS disk and scratch space.

The decision that carries it: is there a shared-write POSIX requirement? There is not, so the answer is object storage throughout — and "we put it on EFS because it was easy to mount" is the mistake this question is testing.

5. When would you choose provisioned Redshift over serverless BigQuery?

When you scan more than about 254 TB a month — the measured break-even — because above that a fixed cluster is cheaper and each extra query is free at the margin.

And four non-price reasons:

1. Predictable spend. A fixed monthly cost is easier to budget than a variable one, even if the variable one is lower on average.

2. No per-query blast radius. A careless query on a provisioned cluster costs time; on on-demand it costs money.

3. Sustained, concurrent workloads. Hundreds of dashboard users hitting the same tables all day is exactly the profile a cluster suits.

4. Data locality with the rest of an AWS estate, avoiding cross-cloud egress.

And the counter: if the workload is spiky — heavy month-end, quiet otherwise — serverless costs nothing during the quiet weeks, and a cluster costs the same every hour.

---

Exam questions from this unit

Two marks

1. Name the three cloud storage types.
2. Which of them has no directories?
3. Why is there no rename in an object store?
4. What is the minimum billing duration for Glacier Deep Archive?
5. Is ingress or egress charged?
6. State RPO and RTO.
7. Name one limitation of a key-value database.
8. What is BigQuery billed on?

Five marks

1. Compare block, file and object storage on access unit, sharing and cost.
2. Explain storage classes and the trap in the "cheap" tiers.
3. Explain egress pricing and data gravity.
4. Explain the features and limitations of key-value databases.
5. Compare batch and streaming ingestion for an ML pipeline.

Ten marks

1. Explain cloud storage in full — the three types, the classes, the use cases and the cost model — with worked arithmetic.

2. Compare a cloud data warehouse with a traditional RDBMS, explain serverless against provisioned pricing, and work out the break-even.

---

Mistakes that cost marks