Data, Model & Risk: Build and Maintain Responsibly

Key question: Can we build and maintain this responsibly?

This guide covers two connected Canvas elements:

• Data & Model Considerations — validates the data, model approach,

infrastructure, evaluation, and operating plan needed for the task.

• Risks & Guardrails — identifies the harms, failure modes, owners,

and tested controls that keep the system within acceptable boundaries.

Why it matters: Data quality problems and ethical risks tend to surface late in AI projects — often after real money has been spent — because they weren't examined early. This domain exists to surface them while they're still cheap to fix, and to make responsibility part of the design, not a late-stage audit.

---

Element 6: Data & Model Considerations

Purpose: Work out what data you need, whether it's actually usable, and what kind of model approach fits the task — from a simple lookup table to a multi-step autonomous agent.

Questions worth answering honestly

• Availability — does the data you need actually exist in usable

form, or are you assuming it does?

• Quality and representativeness — is it accurate, complete, current,

and representative of the people, languages, and situations on which it will be used?

• Rights, privacy, and data-subject requests — can this data legally

and ethically be used for this specific purpose? Can the system locate, correct, remove, or stop using a person's data when applicable law, consent withdrawal, or an approved request requires it?

• Pipeline readiness — can raw data be reliably cleaned, labelled,

transformed, versioned, and made available to the model? For retrieval-augmented generation (RAG), this includes document preparation, chunking, embedding, indexing, and removing superseded content.

• Model fit — what type of model or approach actually suits this

task? Not every problem needs a large language model (LLM), and not every LLM use case needs an agent.

• Performance constraints — what latency, accuracy, explainability,

fairness, and cost thresholds actually matter here?

• Lifecycle needs — how often will the model, prompts, or underlying

knowledge base need to be updated, monitored, or rolled back?

"The data exists" is assumed far too easily in most AI proposals. Fragmented, stale, or poorly governed data can quietly kill an otherwise strong opportunity — check directly rather than assuming.

Weak availability answer: "The data is in our customer relationship management (CRM) system."

Stronger availability answer: "In this illustrative example, customer interaction data is in Salesforce and updates in real time. We sampled 10,000 records from the last 12 months: 87% have the fields this task needs. The remaining 13% are missing interaction type; we may infer it from the email thread, but the support team must validate that approach before a build decision."

Weak quality answer: "The data is mostly clean."

Stronger quality answer: "We checked a representative sample for completeness, duplicate records, stale entries, language coverage, and labelling consistency. We found that Bahasa Malaysia records are less complete than English records. The data owner will resolve the gap or we will constrain the pilot so the system does not make poorer-quality recommendations for that group."

Model approaches: a spectrum, not a menu

The model landscape is often presented as though the only choice is "which large language model to use." That's a false frame. The right approach depends on the task's complexity, variability, explainability requirements, and tolerance for failure. In regulated or consequential decisions, a simpler and more interpretable approach may be the right choice even when a more complex model appears marginally more accurate. Here is the spectrum, ordered from simplest to most complex:

1. Rules and heuristics (no model at all). If the task can be expressed as "if X, then Y" logic — routing a support ticket by keyword, flagging an invoice above a threshold, applying a discount rule — a deterministic system is faster, cheaper, more auditable, and easier to fix when it breaks. Start here. For example, if rules cover the useful cases, the remaining exceptions may not justify a model.

2. Classic machine learning. Classification, regression, clustering, anomaly detection. Classification and regression commonly use labelled data; clustering and some anomaly detection are unsupervised. Assess suitability against the task, data, error tolerance, and operating constraints. Think churn prediction, demand forecasting, fraud scoring. These models can be cheap to run and fast to serve, with well-understood failure modes when their assumptions hold.

Data sufficiency is task-specific. Demonstrate it with representative, leakage-controlled evaluation data rather than a universal record count. For supervised approaches, do not count records as training data until you have checked label quality and coverage. If labels do not exist, decide whether manual labelling, weak supervision, or carefully validated synthetic data can create them within the Horizon's time and budget. The data pipeline and feature engineering required to turn raw data into usable inputs are often a larger effort than choosing the algorithm.

3. Bounded LLM call. One prompt in, one response out. Summarization, drafting, extraction, translation, simple Q&A. The task is completed in a single step. Cost is predictable (one application programming interface (API) call per request), latency is bounded (one round trip), and failures are contained (one bad output, not a cascade). For many tasks, this is the appropriate starting point.

4. Retrieval-augmented generation (RAG). Still a single model call, but the system first fetches relevant documents or data from a knowledge base and includes them in the prompt. The model grounds its response in your data rather than relying on its training memory. Good for: answering questions about internal policies, product documentation, legal contracts, or any domain where accuracy matters and the source material changes. The quality ceiling is set by your retrieval quality — if the wrong documents are fetched, the model's answer will be confidently wrong rather than obviously wrong.

RAG is a data product, not just a prompt. Define who owns source approval, document chunking and metadata, access controls, refresh and removal of superseded material, retrieval evaluation, and the way personal data can be found and removed from indexes or vector stores.

5. Model-directed tool use (bounded agents). The model can call external tools — a database query, a calculator, an API, a code interpreter — as part of generating its response. A single user request might trigger several tool calls, but the orchestration is still relatively contained: the model decides what to call, calls it, and synthesizes the result. Good for: tasks where the model needs real-time data or computation it cannot do internally (looking up a customer's order status, running a financial calculation, fetching live inventory levels). The key risk is that each tool is a new surface for errors and a new permission boundary to manage.

This is different from a deterministic tool workflow, where code calls known tools in a fixed order with validated inputs. Use the deterministic workflow when the sequence is predictable. Once the model can choose a tool, its parameters, or the next step, it is agentic enough to need the relevant evaluation, authorization, and containment safeguards.

6. Multi-step autonomous agents. The model plans a sequence of actions, executes them across multiple steps, observes the results, adjusts its plan, and continues until the task is done or a limit is reached. This is the most flexible and the most expensive approach. It's justified when: the task has variable steps that can't be predicted in advance, the steps depend on intermediate results, and the cost of an incomplete or incorrect execution is tolerable. Examples: researching a complex regulatory question across multiple sources, generating and then testing code against a suite of edge cases, orchestrating a multi-system data migration.

The default stance: most needs are a single call plus retrieval. Reach for agents only when the steps are genuinely variable and the stakes tolerate failure. A multi-step agent can complete all 10 steps, then reveal that an earlier error invalidated the result—consuming the full chain cost without an accepted outcome. A single call that fails has cost you one API call.

Choose the training path before you choose a model

Start with a capable pre-trained model, clear instructions, and retrieval where current domain knowledge is needed. Measure its performance on representative tasks before adding training work.

Fine-tuning may be justified when the task is stable, you have rights to use suitable training data, and representative, leakage-controlled evaluation shows a measurable gap that prompting, workflow design, or retrieval cannot close. Data sufficiency depends on the task and must be demonstrated, not inferred from a universal example count. Plan for versioned training data, repeatable evaluation, monitoring, and retraining; fine-tuning is an operating commitment, not a one-time configuration change.

Training from scratch is a multi-month, multi-person product and infrastructure programme. Consider it only when no suitable pre-trained model can meet a genuinely specialised need and you have the data, compute, evaluation capability, and operating team to sustain it. Treat "we want our own model" as a hypothesis to test, not a reason to begin.

The "Why an agent?" test. Apply this before permitting a model to choose a tool, tool parameter, or next step—not only before building a long autonomous chain. Answer these three questions: 1. Can this be done in a single model call with good retrieval? If yes, stop here. 2. Are the tools and steps predictable enough to hard-code the workflow? If yes, use a deterministic pipeline with validated model calls at specific nodes, not model-directed tool selection. 3. When the agent fails partway through (and it will), what is the blast radius? If the answer is "a human has to redo 30 minutes of work" or "a customer gets a wrong answer and we won't know which one," the agent is not ready for this task without stronger guardrails.

Size the infrastructure for the operating reality

Before a build decision, answer: where will this run; what latency and availability does the workflow need; what data, model artefact, index, and log storage is required; and what happens at ten times the expected request volume? A small pilot may be able to use a managed API and ordinary central processing unit (CPU)-based services. A real-time or self-hosted workload may need different capacity, network, resilience, and cost planning. Record the assumptions rather than treating them as implementation detail.

For agents that execute code, call tools, or access files, use isolated sandboxes such as short-lived containers or virtual machines. Restrict network egress to known endpoints and keep production credentials and databases outside the execution environment. Permission scoping limits what an agent is allowed to do; sandboxing limits the damage if it still behaves unexpectedly.

Evaluating agentic systems

For supervised classification tasks, accuracy, precision, recall, and F1 score (which balances precision and recall) summarize performance across a labelled evaluation set. For supervised regression tasks, select an appropriate error metric—such as mean absolute error or root mean squared error—based on the decision's cost of error; see the scikit-learn model-evaluation guidance. Agentic systems need a broader evaluation toolkit because the question isn't just "was the final answer right?" but "did the system complete the task, using the right tools, without wasting resources or causing side effects?"

Task-completion rate. A core metric for agentic workflows. Of 100 attempts at the task, how many produced an acceptable end result? This sounds straightforward but requires you to define "acceptable" precisely — a binary pass/fail, a graded quality score, or a set of acceptance criteria that can be checked automatically. Task-completion rate captures the compounding failure problem that single-output metrics miss. Assuming independent 95% step success and that all 10 steps are required, full-task success is 0.95^10 ≈ 60%.

Real-world step failures are often correlated rather than independent, so the actual completion rate may be higher or lower depending on the task structure.

Eval harnesses. Automated test suites that run the agent against a fixed set of tasks (a "golden dataset" of inputs with known-good outputs or acceptance criteria) and score the results. The harness runs tasks in parallel, records every intermediate step, and produces a dashboard of completion rates, step counts, tool-call accuracy, token usage, and latency. Think of it as unit testing for agents: you define the test cases, the harness runs them on every change to the agent's prompts or tools, and you catch regressions before they reach production.

A suggested starting policy is 20–50 test cases covering common requests, known edge cases, and five adversarial inputs designed to make the agent do something it should not. These are starting settings, not universal thresholds; scale the suite to the task, consequence, and change risk. Include inputs that represent your actual user population: relevant languages, dialects, name formats, access needs, and query styles. Report task-completion and error rates by relevant group as well as in aggregate; a high overall score can conceal materially worse performance for one group. Run the harness on every prompt or tool change. If performance drops below an agreed threshold, do not ship the change.

Red-teaming. Structured adversarial testing: give the agent unusual, ambiguous, or deliberately misleading inputs and see what it does. For agentic systems, red-teaming focuses on:

• Prompt injection — can a crafted input trick the agent into

calling a tool it shouldn't, accessing data it shouldn't, or producing harmful output?

• Goal hijacking — can the agent be redirected from its assigned

task to a different one?

• Resource abuse — can a single request trigger an expensive loop

that runs up token costs without producing value?

• Differential treatment — do equivalent requests from relevant

demographic, language, disability, or name-format groups receive materially different quality, safety, or access outcomes?

Red-teaming is not a one-time exercise. Run it before launch, after every significant change to the agent's configuration, and on a schedule. Quarterly is a suggested starting cadence, not a universal requirement; set the cadence according to consequence and change risk.

LLM-as-judge. Using one language model to evaluate the outputs of another. Useful for tasks where the "correct" answer isn't binary — the quality of a drafted email, the relevance of a research summary, the coherence of a multi-step plan. LLM-as-judge can make first-pass evaluation more scalable, but whether it is cheaper than human review depends on the model, volume, human-review design, and required calibration. It also has real limitations:

• Depending on the model, prompt, and sampling settings, the same input

can receive different scores across runs.

• Research has documented position, verbosity, self-enhancement, and

self-preference biases in LLM judges; see Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023), and Wataoka et al., Self-Preference Bias in LLM-as-a-Judge (2024).

Calibrate an LLM judge against a blinded human-labelled set, and do not use it as the sole evaluator for high-consequence decisions. Define a risk-based human-review sample before launch, then increase it when a task has higher consequence, a new model or workflow change, a quality incident, or uneven results across groups. For a high-stakes early deployment, 10% is a suggested starting policy, not a universal rule. Compare human scores to LLM-judge scores; if they diverge beyond the agreed tolerance, adjust the judge prompt, add human review, or decide that this task is not suitable for LLM-as-judge.

Tool-call accuracy. Did the agent call the right tool, with the right parameters, in the right order? This can be checked deterministically for tools with defined schemas. The worked task-completion example above illustrates how otherwise reliable steps can compound into a much lower end-to-end success rate. Track this metric per tool, not just in aggregate — one misbehaving tool can drag down the whole system.

Keep the system current and recoverable

A deployed model, prompt, knowledge base, or agent configuration will change in practice even when its code does not. Inputs shift, source material becomes stale, usage patterns change, and a provider may change model availability or terms. Set a named operating owner, review cadence, and thresholds for the current Horizon.

Monitor the measures that can make the system unsafe or uneconomic: quality and task completion, data and retrieval drift, latency, cost per completed task, incidents, and—where people can be affected—performance and error rates by relevant group. A fairness drift signal matters as much as an aggregate accuracy signal: a stable overall score can still hide deteriorating results for a group.

Version the model or provider configuration, prompts, tools, source data, indexes, and evaluation set. Run the evaluation harness before release, record the result, and retain a tested rollback path. Define whether an observed performance drop, source-data change, fairness gap, security incident, or scheduled review triggers retraining, re-indexing, prompt change, rollback, or escalation. Do not retrain automatically without an approval and evaluation gate for consequential use cases.

This is the operating meaning of the data infrastructure maturity readiness score in Value & Cost: a Scale-stage system needs reliable access, controls, monitoring, stewardship, and recovery—not merely a successful prototype.

Token-cost scaling for agentic workloads

A single LLM call has a predictable cost: input tokens plus output tokens, multiplied by the model's price per token. An agentic workload multiplies that cost in ways that are easy to underestimate.

The compounding cost problem. Consider a customer support agent that handles a complex query:

A single-call equivalent for a simple question might use 1,500 input tokens and 500 output tokens. The agentic version used roughly 7x more input tokens because each step carries the growing context of previous steps. That difference is the input to your current cost model, not a reason to reuse a historical price example.

Calculate the run cost for each task type using current provider rates. First calculate the total operating cost across the measurement period:

total task cost = model, tool, storage, network, review, support, retry, and escalation costs for successful and failed attempts

Then calculate:

cost per completed task = total task cost ÷ number of tasks meeting agreed acceptance criteria

Multiply that figure by the expected monthly completions. Model the expected case and higher-volume or higher-retry cases; 2x and 5x volume or demand scenarios are suggested starting cases, not universal thresholds. Record the provider, model, price date, token assumptions, tool and human costs, completion criteria and rate, and any cached-token or batch discount so the estimate can be checked and updated. At scale, retries, failed chains, growing context, and human review can be as material as the base model call.

These figures belong in the Cost Drivers run-cost model alongside infrastructure, monitoring, approval, and support costs. Do not bury an agent's per-completed-task cost in a technical appendix.

Cost controls for agentic workloads:

• Token budgets per task. Set a maximum token spend per agent

execution. For example, 50,000 tokens can be a suggested starting policy, but set the cap for the task's consequence, cost, and required work. If the agent exceeds the budget before completing the task, it stops and escalates to a human. This prevents runaway loops from becoming expensive runaway loops.

• Step limits. Cap the number of steps an agent can take. For

example, 15 steps can be a suggested starting policy, not a universal maximum. If the agent hasn't completed the task within the limit, it reports what it accomplished and what it couldn't, rather than looping indefinitely.

• Cost dashboards per task type. Track the average token cost,

average step count, and cost distribution per task type. If the average cost per completed task starts climbing, investigate whether the agent is retrying too often, carrying unnecessary context, or hitting edge cases that trigger long chains.

• Prompt caching. Some providers allow caching of repeated prompt

prefixes (system instructions, tool definitions), which can reduce the effective cost of the context that gets re-sent with every step. The savings can be substantial for agents with long system prompts.

• Model tiering. Use a smaller, cheaper model for routine steps

(data extraction, simple classification within the chain) and reserve the larger, more expensive model for steps that genuinely need it (complex reasoning, synthesis). Not every step in an agent chain needs a frontier model.

Score anchors: Data & Model Considerations

When scoring this element on the Canvas (1–5 scale), use these anchors to calibrate across teams and projects:

---

Element 7: Risks & Guardrails

Purpose: Name what could go wrong, and define concrete controls that prevent, detect, or mitigate it.

Treat this as a structured exercise, not a brainstorm. For each risk, work through: what could go wrong, who's harmed if it does, how likely is it, how severe is the consequence, and what control addresses it.

Standard risk types

• Output risk — hallucination, harmful content, fabricated citations,

poor recommendations.

• Data risk — leakage, privacy breach, poor provenance,

contamination, or inability to honour approved data-subject requests.

• Decision risk — bias, inconsistency, hidden discrimination,

over-reliance on the AI's judgment.

• Transparency risk — people are unaware that they are interacting

with AI or that a communication, recommendation, or media item was AI-generated.

• Operational risk — model drift, latency, downtime, vendor

dependence.

• Governance risk — unclear approvals, no audit trail, weak policy

alignment.

Fairness is a testable risk, not a one-word label

An AI system can be accurate overall while working substantially worse for particular people. Identify the characteristics relevant to the actual deployment—not a generic checklist. In a Malaysian context, that may include ethnicity and indigenous identity, religion, language (for example Bahasa Malaysia, Mandarin, Tamil, or English), age, disability, gender, name format, or immigration status. The relevant set depends on the user segments, the decision, and applicable law.

Guardrail: Test and monitor outcomes by relevant group. Before launch, build a sufficiently representative evaluation set, compare quality, error, refusal, escalation, and access outcomes by group, and set a documented tolerance for material differences. Investigate gaps; do not assume an aggregate score proves fair treatment. Assign an owner to monitor the same measures after launch and define what triggers a pause, remediation, or escalation. This implements the fairness and inclusiveness intent of Malaysia's National Guidelines on AI Governance and Ethics (AIGE) in system design.

Be clear when people are interacting with AI

If a customer, employee, or other affected person could reasonably think they are interacting with a human or receiving human-authored content, consider a clear, timely disclosure that the system or material output is AI-generated or AI-assisted. Pair it with a route to human help or challenge where the consequence warrants it; do not hide it in terms of service. For EU-relevant deployments, assess Article 50 of the EU AI Act, whose transparency provisions have applied since 2 August

1. Its obligations are scoped to particular systems and content and

include exceptions, so AI-assisted content is not automatically subject to a disclosure obligation.

Guardrail: Document disclosure and handoff rules. Define which channels, outputs, and decisions need a disclosure; test that it appears at the relevant moment; and make the human fallback or review path usable. Treat transparency as a design requirement, informed by Malaysia's AIGE guidance and applicable sector, jurisdictional, and contractual requirements.

Design for personal-data control

Personal data can spread through prompts, logs, training sets, document stores, tool calls, backups, and vector indexes. A generic privacy notice does not show that the system can comply when data must be accessed, corrected, removed, or its use stopped.

Guardrail: Maintain data lineage and an actionable request path. Map where personal data enters, is stored, and is sent; keep tool access traceable; apply retention and deletion controls; and test the procedure for approved data-subject and consent-related requests. For RAG, verify that personal data is discoverable and removable from its source, indexes, and any downstream cache or backup according to the approved retention process. Confirm the legal interpretation with the privacy owner or counsel; this guide is not legal advice.

Agent-specific risks and their guardrails

Agentic systems introduce failure modes that don't exist — or exist in milder form — with single-call models. Each risk below is paired with the guardrail that addresses it. Both are needed; naming a risk without a control is documentation, not management.

1. Compounding errors across multi-step chains. Each step in an agent chain has some error rate. The worked Task-completion rate example above shows how apparently reliable steps can compound into a much lower full-task success rate. Real-world step failures may be correlated, so measure completion on representative tasks rather than assuming the example predicts production results. Errors can also compound invisibly: step 4 might produce a result that's plausible but wrong, and steps 5–10 build on that foundation without detecting the problem.

Guardrail: Step-level validation checkpoints. At each step where the output feeds into subsequent steps, validate the result against known constraints before proceeding. For a database query, check that the returned record exists and matches the expected schema. For a calculation, verify the result is within a plausible range. For a drafted response, check that all cited facts trace back to a source in the context. When a validation fails, the agent retries the step (up to a limit) or escalates to a human.

2. Runaway loops. An agent enters a cycle where it repeats the same action, or alternates between two or three actions, without making progress toward the goal. This can happen when the agent misinterprets an error message, when a tool returns unexpected output that the agent doesn't know how to handle, or when the agent's plan has a logical cycle.

Guardrail: Iteration limits and duplicate-action detection. Set a hard cap on total iterations. For example, 15 steps can be a suggested starting policy, not a universal cap. Additionally, monitor for repeated tool calls with identical or near-identical parameters. For example, halting after the same tool is called with the same arguments three times in a row is a suggested starting trigger; tailor it to the task. Log every iteration so post-incident analysis can identify the loop trigger.

3. Tool misuse and excessive permissions. An agent with access to a database, file system, email system, or payment API can cause real damage if it uses a tool incorrectly or uses a tool it shouldn't have access to. The risk scales with the breadth of permissions granted.

Guardrail: Least-privilege permissions and sandbox isolation. Each tool the agent can call should have the minimum permissions needed for the task. A customer support agent's database tool should be read-only and scoped to the customer table, not write-enabled across the full database. An email-sending tool should require explicit confirmation before sending to external addresses. Permissions should be defined per task type, not globally — the same agent running a "research" task should have different tool permissions than when running a "send communications" task. Run code, file, and external-tool actions in an isolated, short-lived environment; restrict network egress to approved endpoints; and do not expose production credentials or broadly privileged systems inside that environment.

4. Prompt injection and goal hijacking. An adversary crafts an input that tricks the agent into deviating from its assigned task — calling a tool it shouldn't, accessing restricted data, producing harmful output, or exfiltrating information through tool calls (e.g., encoding stolen data in the parameters of an API call to an external service).

Guardrail: Treat external content as untrusted; enforce tool policy outside the model. User input, retrieved documents, web pages, email, attachments, and tool responses can all contain indirect prompt injection. Do not treat any of them as instructions. Restrict tools to a task-specific allowlist, then enforce user and data authorization, parameter schemas and value constraints, rate limits, and destination allowlists in a server-side tool broker independent of the model's instructions. Monitor anomalous parameters (for example, a customer-ID field that contains encoded data), restrict outbound network egress, and log every external request for investigation. Output filtering is a final check, not a substitute for authorization and containment.

5. Token-cost spirals. An agent encounters an edge case, retries repeatedly, accumulates context with each retry, and runs up a large token bill without producing a useful result. At scale, a small percentage of pathological requests can account for a disproportionate share of total spend.

Guardrail: Per-task token budgets with escalation. Set a maximum token spend per execution. For example, 50,000 tokens or USD 2.00 equivalent, whichever is reached first, can be a suggested starting policy—not a universal limit. When the budget is hit, the agent stops, reports what it accomplished, and escalates the remainder to a human. Track the distribution of token spend across requests. As a suggested starting trigger, investigate when the top 5% of requests account for more than 30% of total spend; tailor both the cohort and threshold to the deployment. Consider whether those request types need a different approach: a simpler model, a deterministic workflow, or a Recommendation or Approval interaction mode from the start.

6. Unobservable intermediate steps. The agent produces a plausible final output, but the reasoning and tool calls that produced it are invisible. You can't tell whether the agent actually checked the sources it cites, whether it considered and rejected alternatives, or whether it hallucinated an intermediate step that happened to produce a reasonable-looking answer by coincidence.

Guardrail: Structured, privacy-aware execution traces. Log the task ID, model and prompt version, tool calls, approved parameters, returned status, validation results, approvals, final output, and any failure or escalation. Redact or minimise personal and sensitive data; retain logs under a defined access and retention policy. This creates an investigable record without requiring storage or disclosure of private chain-of-thought reasoning. Make the traces queryable for incidents and sample production outputs and traces on a risk-based cadence. Weekly reviews in the first three months are a suggested launch cadence, not a universal requirement. For high-stakes outputs, give the reviewer or affected person a clear explanation of the relevant inputs, sources, decision rules, limitations, and human handoff—not a raw reasoning transcript.

A tested kill switch is different from a limit or escalation

Every agentic system needs a kill switch: a mechanism that immediately halts new agent execution and tool calls, cancels or safely drains in-flight work where the tool supports it, revokes or disables relevant credentials, and preserves the execution state and audit record for investigation. It cannot be overridden by the agent itself. Assign named human authority to activate and restore it, and test the activation, external-action reconciliation, and controlled restoration procedure before launch and after material architecture changes.

For an action that cannot be cancelled—such as an already sent email, payment instruction, or third-party API request—the procedure must identify what has happened, contain further effects, notify the appropriate owner, and reconcile or remediate it. This is distinct from an iteration limit, which ends a single task when it reaches a cap, and escalation, which hands an incomplete task to a person. A kill switch stops the service or defined capability because the system itself may be unsafe or untrustworthy.

Post-launch ownership

Awareness of ongoing risk isn't the missing piece — accountability is. For every risk you name, add an owner and a review cadence.

Weak answer: "Risk: model drift over time."

Stronger answer: "Risk: model and fairness drift. Owner: data team lead. Reviewed monthly against agreed task-completion, accuracy, and group-level error thresholds. A material performance or fairness gap pauses the affected workflow, triggers investigation and a controlled re-evaluation; incidents are logged and escalated to the AI steering group. The business owner reviews a risk-based sample of production outputs, with 10% as a suggested starting policy during a high-stakes early deployment, and adjusts the rate when risk or evidence changes."

If a risk on your list doesn't have a name attached to it, it isn't actually managed — it's just documented.

This applies with extra force to agentic systems, where the attack surface and failure modes evolve as the agent encounters new task types in production. Assign a specific person to review the agent's audit traces and sampled outputs on a risk-based launch cadence; weekly trace reviews, quarterly red-teaming, and monthly cost-distribution reviews are suggested starting cadences, not universal requirements. Reduce or increase the cadence only with a documented rationale based on evidence and consequence.

The pre-mortem exercise

Before launch, run this structured reflection with the project team (minimum: business owner, technical lead, and a risk or compliance voice):

"It's 12 months from now. This project has failed. What happened?"

This is not a brainstorm. Each person writes down their answer independently, then the group shares and clusters the responses. The value is in the specificity — "the model drifted" is too vague; "customer complaints about incorrect billing answers reached the regulator because no one was monitoring the agent's output quality after month 3" is actionable.

Structure the exercise around five prompts:

1. What assumption in our Canvas turned out to be wrong? (e.g.,

"We assumed users would review the agent's output before sending it, but they stopped checking after week two.")

1. What risk did we name but under-invest in? (e.g., "We identified

prompt injection as a risk but tested too few adversarial inputs before launch.")

1. What did we not think to name? (e.g., "Nobody considered that

the agent's training data included outdated regulatory guidance that contradicts the current rules.")

1. Where did the economics break down? (e.g., "Token costs at

production volume were 4x our pilot estimate because we didn't account for retry loops.")

1. What would we tell ourselves 12 months ago? This prompt tends

to produce the most honest and actionable items.

Capturing the top 5–7 failure scenarios is a suggested starting format, not a universal threshold. For each, ask: does our current Canvas already have a guardrail that addresses this? If not, add one. A 60–90-minute workshop is a suggested starting format, not a universal duration. The exercise tends to surface risks that the standard checklist misses because it forces people to think from the failure backward rather than from the plan forward.

Vendor lock-in and exit strategy

AI projects often depend on specific model providers, APIs, or platforms. That dependency carries risks that should be named and planned for, not discovered when the invoice changes.

Model availability changes. Providers may change model availability or withdraw services under their lifecycle policies and contract terms. A change affecting a model your system depends on can require prompt retesting, output revalidation, and possibly rebuilt tool integrations that rely on model-specific behaviour.

Guardrail: Track your model dependencies and lifecycle terms. Maintain a register of which models your system uses, then consult the selected provider's lifecycle policy and contract terms for applicable notice, support, and end-of-life conditions. Record the lead time needed to migrate. Build your system with an abstraction layer between your application logic and the model API so that switching models does not require rewriting the entire codebase. Consider a migration contingency proportionate to the dependency risk.

Pricing changes. Providers may change pricing subject to product and contract terms. A system that is economical at current pricing may not remain so after a price revision.

Guardrail: Model your unit economics with a pricing sensitivity analysis. As a suggested stress test, examine what happens to cost per interaction if the provider doubles its token price; use scenarios that fit the dependency risk rather than treating this as a universal case. If the project becomes uneconomical, you need either a migration path or a contractual arrangement that limits exposure. Where possible, negotiate committed-use pricing that provides cost certainty for the relevant term.

Migration path. If you need to switch providers — because of availability, pricing, performance, contractual, or strategic reasons — how long would it take and what would it cost?

Guardrail: Maintain portability by design. Use standard APIs where possible, keep your prompt engineering and evaluation harness provider-agnostic enough to adapt, and test your golden dataset against at least one alternative model before you need it. The evaluation harness you built for task-completion testing (see "Evaluating agentic systems" above) doubles as a migration testing tool — run it against the new model and compare results.

Score anchors: Risks & Guardrails

---

Grounding this in real regulatory frameworks

You do not need to become a compliance expert to use this domain well, but a few reference points are worth knowing. Check the current official text and obtain advice for your actual context; frameworks and sector requirements change.

• [**ISO/IEC 42001:2023, Information technology — Artificial intelligence

— Management system**](https://www.iso.org/standard/42001) — an international AI management-system standard. Its core idea: defined responsibilities, lifecycle governance, documented controls, and continual improvement.

• [**NIST AI Risk Management Framework (AI RMF 1.0; National Institute

of Standards and Technology, 2023)**](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) — a voluntary US framework organized around four functions: Govern, Map, Measure, Manage. It is useful as a repeatable pattern beyond the US: establish governance context, map the use case and who it affects, measure the indicators that matter, and define how mitigations are managed over time.

• [**EU AI Act — Regulation (EU) 2024/1689 (European Parliament and

Council; current consolidated text, 27 July 2026)**](https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng) — a risk-based regulation. Its practical lesson is that use cases touching employment, access to services, safety, or other sensitive decisions warrant stronger oversight and documentation. It includes transparency obligations for certain AI systems and content; assess the relevant provisions and exceptions for the actual deployment.

• [**Malaysia's National Guidelines on AI Governance and Ethics (AIGE;

Ministry of Science, Technology and Innovation, 2024)**](https://mastic.mosti.gov.my/storage/2024/09/THE-NATIONAL-GUIDELINES-ON-AI-GOVERNANCE-ETHICS.pdf) — the local anchor for Malaysian organizations, built around fairness, reliability, safety and control, privacy and security, inclusiveness, transparency, accountability, and human benefit. Use it alongside, rather than instead of, applicable law and sector rules.

• Malaysia's Personal Data Protection Act 2010 (Act 709), as amended by the

Personal Data Protection (Amendment) Act 2024 (Act A1727), applies to personal-data processing in commercial transactions, subject to statutory exclusions; confirm scope and applicable requirements. See the official application and non-application guidance and confirm obligations with the privacy owner or counsel.

Malaysian sector escalation points

The following is a starting map, not a complete legal register. If a system operates in a regulated sector, identify the licensed entity, affected decision, and applicable current requirements before it reaches a real customer or operational decision.

When to escalate beyond this guide: if your use case touches employment decisions, healthcare, financial access, education, safety-critical processes, identity verification, or anything that meaningfully affects a person's rights or opportunities, treat this Canvas as a starting point, not a substitute for legal, privacy, sector, or risk review. Stronger human oversight and documentation should be the default in those cases, not an afterthought.

Agentic systems that can take autonomous actions affecting people's finances, access to services, or legal standing should be treated as high-risk by default, regardless of whether a specific regulation uses the word "agent." The principle is the same: the more autonomy the system has, the stronger the oversight, approval, logging, rollback, and human intervention need to be.

---

Before moving on

You should have:

• A named data source, representative sample, documented provenance, and

a real answer about whether you may use, retain, and remove the data.

• A model approach justified against simpler alternatives and an explicit

pre-trained, fine-tuning, or training-from-scratch decision—with a documented "Why an agent?" answer for multi-step or tool-use work.

• A current-Horizon infrastructure, pipeline, and lifecycle plan:

access controls, latency and scale assumptions, versioning, monitoring, rollback, and a named operating owner.

• An evaluation plan beyond accuracy: task-completion rate, a golden

dataset, relevant group-level results, and—for agentic systems— red-teaming and tool-call evaluation.

• Current per-completed-task cost estimates at expected production volume

with sensitivity analysis for growth, retries, and price change, carried into the Cost Drivers model.

• A concrete risk register; five risks is a suggested starting level,

with named owners, review cadences, escalation paths, and tested guardrails—including data lineage, applicable AI disclosure, fairness monitoring, and human handoff.

• For agentic systems: least-privilege permissions, sandbox isolation,

iteration and token limits, structured execution traces, and a tested kill switch.

• A completed pre-mortem exercise with its findings reflected in the

risk register.

• A vendor dependency register with a migration path and pricing

sensitivity analysis.

• A clear escalation decision for high-impact or regulated use cases,

including the relevant sector owner, privacy, legal, risk, or clinical review where appropriate.