Unit 4 — Cloud Platforms for Data Science and ML

Topics Covered

Syllabus topics: Machine learning in the cloud: benefits and limitations. Cloud-based ML services: AIaaS, GPUaaS. Managed ML platforms: overview and advantages. Cloud ML platforms: AWS SageMaker, Azure ML Studio, Google Cloud AutoML.

4.1 Machine learning in the cloud — what changes, and what does not

4.2 Benefits, honestly weighed

4.3 Limitations

⚠️ The one that costs real money

A training job ends and stops billing. An endpoint runs until you delete it, at hourly rates, whether or not anything calls it.

An ml.m5.large endpoint is about $70/month, called or not.

Every "surprise cloud bill" story is a resource nobody switched off. Set a budget alarm on day one, before creating anything else.

⚠️ And the one that costs credibility

Nothing about the cloud fixes the data. A wrong target, a leaked feature, an unexamined base rate or a drifted distribution is exactly as wrong on eight GPUs. The DummyClassifier at 84.33% is unmoved by hardware.

4.4 AIaaS and GPUaaS

The syllabus names these as if they were new layers. They are not, and saying so is worth the mark:

AIaaS — pre-trained services

GPUaaS — and the honest version

Renting a GPU is renting an instance. The interesting content is which one and why:

And the two questions before renting any of them:

1. Does the algorithm have a GPU code path? If not, the answer is no.
2. Is the GPU actually busy? A GPU at 15% utilisation because the data loader is the bottleneck is a CPU problem you are paying GPU rates to ignore.

4.5 Managed ML platforms

  data  →  prepare  →  train  →  tune  →  register  →  deploy  →  monitor
            │           │        │         │            │          │
         Data Wrangler  Training HPO    Model      Endpoint   Model
         / Glue         Jobs           Registry              Monitor

The three platforms, compared

The pieces worth knowing by name

The feature store is the one that prevents a real, silent failure: training/serving skew, where a feature is computed one way in the batch job and another way at request time. The model then sees a distribution it never trained on, and every infrastructure metric stays green.

4.6 AutoML

⚠️ And read the top of it carefully

0.9334 against 0.9288 is a gap of 0.0047, with standard deviations of 0.0210 and 0.0196. The difference is inside the noise.

Declaring a winner here is not supported by the data, and "AutoML picked X" is not a reason to prefer X. If two models are indistinguishable, choose on something else: inference cost, interpretability, training time.

Practice problems

1. A team reports "our model is 97% accurate in the cloud". What do you ask next?

Three questions, in order:

1. What is the base rate? 97% on a 3%-positive problem is what DummyClassifier scores by predicting "no" every time — measured here at 84.33% accuracy and 0.0000 F1 on a 15%-positive problem.

2. What is the F1 or recall on the positive class? Accuracy on an imbalanced problem tells you almost nothing.

3. Was the test set held out before any preprocessing? Scaling or imputing on the full dataset leaks the test distribution into training.

And "in the cloud" is irrelevant to all three. The infrastructure has no bearing on whether the evaluation is sound, and mentioning it suggests the speaker thinks it does.

2. When would you use a pre-trained AIaaS API instead of training your own model?

When the task is generic and your data is not special. Extracting text from a scanned invoice, transcribing English speech, detecting whether an image contains a face — a model trained on billions of examples will beat what you can build, and the API costs cents per call.

Train your own when:

• the domain is specific — your defect types, your forms, your vocabulary
• you must explain the decision, or audit it
• the data may not leave your environment
• the behaviour must be stable — the provider updates the model, and your thresholds move underneath you

The decision rule: would a knowledgeable stranger do this task well without knowing your business? If yes, use the API.

3. Your AutoML run returns a model with AUC 0.91 and your hand-built one scores 0.89. Do you switch?

Not on those numbers alone. Ask:

1. What is the variance? In the measured run, standard deviations were 0.0196 and 0.0210 — a 0.02 gap is comfortably inside one standard deviation, so the models are not distinguishable.

2. Is it the same test set, the same folds, the same preprocessing? If AutoML did its own feature engineering, you are comparing two pipelines, not two models.

3. What does the AutoML model cost to serve, and can you explain it? A stacked ensemble of forty models scoring 0.02 higher is often a worse choice than one gradient booster.

And if the gap is real and large, the useful question is what did it find that I missed? — read the candidate-definition notebook. The feature engineering is usually the answer, and you can adopt it in your own pipeline.

4. Explain the difference between AIaaS, GPUaaS and a managed ML platform, with an example of when each is right.

The axis is the same as Unit 1's: how much of the stack the provider manages. AIaaS manages everything including the model; GPUaaS manages only the hardware; managed ML sits between, handling the lifecycle while you supply the algorithm and the data.

5. Your training job runs on a p3.2xlarge and the GPU shows 12% utilisation. What is wrong and what does it cost you?

The GPU is not the bottleneck — the data pipeline is. The GPU is idle 88% of the time waiting for batches.

Usual causes: single-threaded data loading, decoding images on the CPU, reading small files one at a time from object storage, or a batch size too small to fill the device.

What it costs: you are paying $3.06/hour for a machine doing $0.19/hour of work — and the fix is almost always CPU-side: more data-loader workers, prefetching, larger batches, pre-decoded data in a format like TFRecord or WebDataset, or reading larger files.

The general point: rent the resource that is the bottleneck. Paying GPU rates to run a CPU-bound pipeline is the most common way to waste money on a managed ML platform.

Exam questions from this unit

Two marks

1. What does the cloud change about a machine-learning algorithm?
2. Expand AIaaS and GPUaaS, and say what each really is.
3. Name three managed ML platforms.
4. What is a model registry for?
5. What is a feature store for?
6. What does AutoML actually do?
7. Name two things AutoML cannot do.
8. Why does a training job stop billing and an endpoint not?

Five marks

1. Explain the benefits of ML in the cloud, and the catch attached to each.
2. Explain the limitations of cloud ML platforms.
3. Explain AIaaS with examples, and when it is the wrong choice.
4. Compare SageMaker, Azure ML and Vertex AI by capability.
5. Explain what AutoML does, what it costs, and what it leaves undone.

Ten marks

1. Explain cloud-based ML services in full — AIaaS, GPUaaS and managed platforms — with the benefits, the limitations, and worked cost figures.

2. Evaluate AutoML for a business that wants to build a churn model. What will it do, what will it not do, and what would you recommend?

Mistakes that cost marks