AI Center of Excellence Best Practices: A Guide to Scaling GenAI

A Center of Excellence (CoE) is usually described as a central team that owns AI. For most agencies and GLCs, that framing is the reason stand-up stalls. The defensible position here: your CoE should be a small standards-and-enablement function — 4 to 6 people — that ships production-grade AI in 12 weeks, not a large central build team that spends its first year writing policy. Ownership of models and budgets stays with business units; the CoE owns standards, reuse and evidence.

That matters because the failure mode is rarely technology. It is duplicated pilots, no audit trail, and no repeatable way to move from a working demo to a system in operations.

Why GenAI pilots stall after the demo

The pattern is consistent across corporate and public sector contexts:

• Pilot sprawl. Five departments run five chatbot experiments. None share a prompt library, evaluation method or data handling rule.
• No production handover. The pilot has a demo owner, not an operations owner.
• Governance arrives late. Data classification and human review questions surface after a tool is already in use.
• Funding is event-based. One-off training budget, no multi-year capability line.

A CoE exists to remove those four blockers, not to run every AI project itself.

What the CoE owns — and what it should not

CoE owns	Business unit owns
Reference architecture and approved tool list	The specific use case and its outcome metric
Data classification and usage guardrails	Day-to-day data handling within those guardrails
Evaluation and acceptance testing standards	User acceptance in their workflow
Reusable assets: prompt libraries, connectors, templates	Adoption and process change
Incident register and audit trail	Remediation of their own workflow
Capability curriculum and role-based training	Nomination and release of staff

Draw the boundary explicitly in a one-page charter. Ambiguity here is what turns a CoE into a bottleneck.

Team structure for a 4 to 6 person CoE

• CoE lead (1) — reports to a permanent secretary, CIO or transformation lead. Owns the charter and the portfolio view.
• AI product / delivery lead (1) — runs the 12-week cycle, manages vendor and internal interfaces.
• Data and platform engineer (1–2) — integration, access control, logging, evaluation harness.
• Responsible AI / governance analyst (1) — data classification, human-review rules, risk register, alignment with internal audit.
• Capability and change lead (1) — role-based training design and adoption measurement.

In smaller agencies, roles 1 and 2 can be combined. What should not be combined is build and assurance — the person shipping a solution should not be the only person signing it off.

Governance that survives audit

Minimum viable governance, drafted in weeks 1–3:

1. Data classification rule. Which categories (public, internal, restricted) may enter which tools. Reference your existing data policy and PDPA obligations rather than inventing a parallel one.
2. Human review requirement. Define which outputs require a named reviewer before use — typically anything affecting citizens, payments, eligibility or legal position.
3. Tool approval path. A short list of approved tools with settings reviewed, plus a route for exceptions.
4. Incident and error register. Log errors, near-misses and corrections. This is your evidence file when audit asks how the system behaves.
5. Decision rights. Who approves a new use case, who approves go-live, who can pause.

Governance that is written but not translated into employee behaviour does not reduce risk. Pair the policy with role-based briefing so staff understand what it means in their actual workflow.

A 12-week rollout timeline with named owners

The 12-week sequence: mandate, guardrails, build, pilot, assurance, go-live — each with a named owner.

Weeks 1–2 — Mandate. Owner: CoE lead. Written charter, sponsor confirmed at executive level, two priority use cases selected against a stated outcome (cycle time, error rate, enquiry handling time).

Weeks 3–4 — Baseline and guardrails. Owner: governance analyst. Data classification rules signed off, approved tool list published, current-state inventory of existing AI experiments recorded.

Weeks 5–7 — Build and evaluate. Owner: delivery lead and platform engineer. Priority use case built against the reference architecture, evaluation set defined before build, human-review path instrumented.

Weeks 8–9 — Controlled pilot. Owner: capability lead. 20–50 users drawn from real roles, not volunteers only. Measure usage, error handling and time saved against the week 1–2 baseline.

Weeks 10–11 — Assurance. Owner: governance analyst. Review the incident register, confirm logging, document what changed. Escalate anything unresolved rather than closing it silently.

Weeks 12 — Go-live decision. Owner: CoE lead and business unit owner. Named operations owner, support model and training completed before rollout widens.

Do not skip weeks 3–4. Almost every stalled programme we see traded guardrails for speed and paid for it later in remediation.

What the numbers mean for a Malaysian team

Indicative planning ranges, exclusive of vendor licensing, based on programme structures AIHQ has delivered across corporate, public sector and regulated environments:

• CoE stand-up facilitation and charter development: MYR 25,000–60,000 for a structured engagement covering charter, governance drafting and use-case prioritisation.
• Role-based AI training: commonly planned in the MYR 1,200–3,000 per participant range for multi-day programmes, with volume considerations for cohorts above 30.
• Use-case discovery and pilot planning workshop: MYR 15,000–40,000 depending on scope and number of departments involved.
• Custom solution build (for example, an internal SOP copilot or enquiry chatbot): MYR 40,000–150,000 for a scoped first release, with ongoing support separately costed.

Two planning notes for the public sector:

• Training can often be structured to be HRDC claimable, subject to client eligibility, grant approval and HRD Corp submission requirements. Budget-holders should confirm eligibility with HR before committing a line item.
• Budget for the second year, not just the pilot. The maintenance, evaluation and retraining line is what keeps a CoE credible after launch.

AIHQ has trained and engaged over 9,000 professionals and has worked with government agencies, GLCs, professional institutions and regulated organisations, including Selangor State Government and Local Authorities, SME Corp Malaysia, MDEC and Media Prima Group.

Five CoE best practices worth adopting

1. Treat the CoE as a standards body, not a service desk. If every request routes through the centre, the centre becomes the queue.
2. Ship one use case to production before scaling. One system in operations teaches more than five demos.
3. Measure adoption, not enthusiasm. Track weekly active use by role, not headcount trained.
4. Reuse assets deliberately. A prompt library, connector pattern and evaluation set should be reused by the second use case.
5. Report to the executive sponsor in outcome language. Cycle time, error rate, enquiry volume handled — not model names.

Frequently asked questions

The FAQ section below covers structure, cost, governance and scope questions raised by agencies and GLCs.

Next step

If your agency or GLC is deciding how to structure AI capability for the coming financial year, the most useful first conversation is a scoping one — not a procurement one.