Unit 5 — Training and Deployment of ML on the Cloud

Topics Covered

Syllabus topics: Factors for selecting cloud ML platforms: ETL/ELT pipeline support, scale-up/scale-out training, ML frameworks, pre-tuned services. Steps for training ML models in the cloud: data source identification, feature engineering, training, validation, deployment, monitoring. Monitoring and improving cloud-deployed ML models. Case studies and industry applications.

5.1 Factors for selecting cloud ML platforms

The syllabus names four factors. Each is a real question with a real answer.

1. ETL/ELT pipeline support

Can the platform get the data to the model without you writing a pipeline?

The deciding question is where your data already lives. A team on BigQuery has a real reason to prefer Vertex; a team on S3 has a real reason to prefer SageMaker. That is a better answer than a feature comparison.

2. Scale-up against scale-out

3. ML frameworks — version and container support

"Bring your own container" is the row that matters, because it is the answer to every future incompatibility — at the cost of maintaining the container.

4. Pre-tuned services

Does a good-enough model already exist? Unit 4's AIaaS discussion applies: generic tasks are already solved better than you will solve them, and specific ones are not.

⚠️ And the factor the syllabus omits

Lock-in, and the cost of leaving. A SageMaker Pipeline does not run on Vertex. The mitigations are ordinary engineering: keep the training code in a plain container, keep the data in an open format (Parquet), keep the artefact in an open format (ONNX or joblib), and treat the platform as a scheduler.

5.2 The six steps

The syllabus lists them. Here is what actually goes wrong at each.

  1. IDENTIFY      2. ENGINEER     3. TRAIN      4. VALIDATE
     the data         features        the model     it
        │                │               │             │
        └────────────────┴───────────────┴─────────────┘
                              │
                    5. DEPLOY  →  6. MONITOR
                              ↑          │
                              └──────────┘
                             (and retrain)

Step 1 — Data source identification

And the one that ends projects: does a label exist? An enormous number of "machine learning projects" are really "we would need to label 50,000 examples first" projects, and discovering that in week six is expensive.

Step 2 — Feature engineering

This is where the time goes, and it is Machine Learning's material unchanged.

The cloud-specific additions:

• Compute features where the data is. A GROUP BY in the warehouse beats exporting a terabyte and doing it in pandas.

• Use a feature store if one exists, for the training/serving skew reason.

• Version the feature definitions with the model. A model is only reproducible if its inputs are.

⚠️ Step 2 is also where leakage happens

Any feature computed using information not available at prediction time is leakage — and in a cloud pipeline it is easy to introduce, because the training job reads a warehouse table that has been updated since.

The classic form: a feature like total_orders_this_month computed from a table that already includes the outcome month. The model scores brilliantly and fails in production, and no infrastructure metric will tell you.

Step 3 — Training

The job is declarative:

estimator = SKLearn(entry_point="train.py", role=role,
                    instance_type="ml.m5.xlarge",
                    hyperparameters={"n_estimators": 100},
                    output_path=f"s3://{bucket}/models/")
estimator.fit({"train": f"s3://{bucket}/train/"})

Three conventions are the whole interface: hyperparameters arrive as command-line arguments, channels arrive as environment variables (SM_CHANNEL_TRAIN), and the model must be written to SM_MODEL_DIR. Getting the last one wrong is why a job "succeeds" and produces no artefact.

Spot training is up to 70% cheaper and interruptible. max_wait must exceed max_run, and without checkpoint_s3_uri an interrupted job restarts from zero — free money for a 20-minute job, a trap for a 20-hour one.

Step 4 — Validation

Everything from Machine Learning applies, and the cloud adds nothing except the temptation to skip it because a leaderboard exists.

The measured baseline, again:

Quote the dummy. Hold out a test set before preprocessing. Report cross-validated mean and standard deviation. In the AutoML run, the top two models differed by 0.0047 with standard deviations around 0.02 — inside the noise, and not a basis for a decision.

Step 5 — Deployment

The artefact is the deliverable. In 15_deploy_endpoint.py a 138,945-byte model file is written, reloaded and served — and the endpoint's answers are identical to calling the model in-process.

The container contract is two routes:

A health check that does real inference marks the container unhealthy whenever the model is merely slow — and the platform then kills a container that was working. A self-inflicted outage, and a classic one.

one request of 100 rows :    4.90 ms
100 requests of one row : 181.85 ms   (37x)

⚠️ And the error codes matter operationally

Every malformed request in the lab returns a 4xx, not a 5xx:

5xx means your service is broken and should page someone. If malformed client input returns 500, your error alarm fires for other people's bugs and you stop trusting it.

Step 6 — Monitoring

Covered in §5.3.

5.3 Monitoring, alarms and autoscaling

mean 85.0 ms   p50 42.0 ms   p95 87.8 ms   p99 737.5 ms

The six metrics worth alarming on

5.4 Improving a deployed model

⚠️ The failure that has no error message

A model does not break. It decays. The endpoint keeps returning 200 and the predictions quietly stop being right.

Model Monitor, in outline

monitor.suggest_baseline(baseline_dataset="s3://bucket/train/train.csv")
monitor.create_monitoring_schedule(endpoint_input=predictor.endpoint_name,
                                   schedule_cron_expression="cron(0 * ? * * *)")

Baseline the training distribution, then compare production inputs against it hourly. A drift alarm is the only thing that catches a model that has stopped working while every infrastructure metric stays green.

Retraining

And retraining must be a pipeline, not a person. If retraining is a notebook someone runs, it will stop happening when that person changes team.

Deploy the retrained model with a canary, and Unit 1's requirement applies: a canary needs a metric you trust, which is why monitoring comes first.

5.5 Case studies and industry applications

Practice problems

1. Your endpoint's mean latency is 60 ms and users complain it is slow. Explain and diagnose.

The mean is hiding the tail. In the measured example, twenty requests with a mean of 85 ms had a p99 of 737.5 ms — one request in twenty took 900 ms. If 5% of your users wait most of a second, they will complain and the dashboard will look fine.

Diagnose:

1. Look at p50, p95 and p99, not the mean.

2. Separate ModelLatency from OverheadLatency. The first is inference; the second is the platform's serialisation and routing.

3. Check for cold starts if the endpoint is serverless or recently scaled.

4. Check the payload size. Large JSON bodies parse slowly, and the measured batching result showed per-request overhead dominating the model.

5. Check garbage collection or model reloading in the container.

And fix the alarm: threshold on p99, and remember ModelLatency is in microseconds.

2. Autoscaling is enabled and your bill went up. How is that possible?

Measured, exactly: the most aggressive configuration (scale out at 50%, no cooldown) used 188 instance-hours against fixed capacity's 168 — 12% more — while dropping nothing.

The mechanism: a scaling decision takes effect one observation late, so the group overshoots on the way up and, with a short scale-in cooldown, oscillates rather than settling. You end up buying more than the peak, plus the boot time of every extra instance.

The fixes: raise the scale-out threshold, lengthen the scale-in cooldown, cap max-capacity, and — most importantly — measure, because the trade-off curve is specific to your traffic shape.

3. Design monitoring for a fraud-detection endpoint. What do you alarm on and why?

Infrastructure:

• Invocation5XXErrors > 0 for 1 min — a fraud endpoint failing means transactions go unchecked

• ModelLatency p99 > the payment timeout — a late answer is a missed decision

• Invocations == 0 for 10 min — silence means the caller is broken

Model:

• Input drift against the training baseline, hourly

• Score distribution shift — if the fraction flagged moves sharply, something changed upstream

• Precision on confirmed cases, weekly — labels arrive late here, so this is the lagging indicator

Business:

• Flagged rate against the historical band — a model flagging 30% instead of 0.3% is an incident regardless of what the metrics say

And the fraud-specific problem: labels arrive weeks late (chargebacks), so you cannot monitor accuracy in real time. Drift monitoring is not a convenience here — it is the only fast signal you have.

4. A model is retrained monthly on the last 12 months of data. What could go wrong?

1. Feedback loops. If the model's decisions changed behaviour, the last 12 months contain its own effects. A holdout control group is the only clean fix.

2. A regime change is diluted. A sharp shift three months ago is a quarter of the window; the model averages across a break rather than adapting to it.

3. The retraining pipeline drifts from the serving pipeline. If features are computed differently in the two places, every retrain is trained on a distribution it will not see.

4. Nothing validates the retrained model before it ships. A monthly automatic retrain that deploys without a gate will eventually deploy a worse model.

5. Label latency. If labels take six weeks, the last six weeks of the window are unlabelled or wrongly labelled.

The mitigations: a champion/challenger gate — the new model must beat the current one on a held-out set before promotion — a canary rollout, and a holdout group.

5. Choose a deployment shape for each: a fraud check during checkout; a nightly credit score for 20 million customers; a demo used a few times a week.

The deciding questions are: what is the latency budget, and what is the duty cycle? Steady and urgent → provisioned. Occasional → serverless. No latency requirement → batch. An ml.m5.large endpoint at $70/month for a weekly demo is the case this question exists to catch.

Exam questions from this unit

Two marks

1. Name the four factors for selecting a cloud ML platform.
2. Distinguish scale-up from scale-out.
3. Name the six steps of training a model in the cloud.
4. What must a container's /ping route not do?
5. Why alarm on p99 rather than the mean?
6. Give one metric that catches a dead endpoint.
7. Distinguish data drift from concept drift.
8. Name the three deployment shapes.

Five marks

1. Explain the four platform-selection factors with a question for each.
2. Explain the six steps, and one failure mode at each.
3. Compare real-time, serverless and batch inference with cost figures.
4. Explain autoscaling and why it can increase cost.
5. Explain model drift and how it is detected.

Ten marks

1. Describe the full lifecycle of training and deploying a model on the cloud, from data source to monitoring, with the failure modes at each step.

2. Design and justify a monitoring and retraining strategy for a cloud-deployed churn model, covering infrastructure, model and business metrics.

Mistakes that cost marks