AI Readiness Assessment for Engineering Teams

An engineering team is ready for an AI use case when it can define the task, control the data and actions, measure failures, operate the serving path, and assign accountable owners. Model access and a working demo are not readiness evidence.

Assess one use case at a time. A company can be ready to deploy internal document search and unready to automate customer refunds. The score below is a delivery gate, not a generic maturity label.

Use evidence, not confidence

Score each dimension from 0 to 2:

• 0 — missing: no owner or artifact exists;
• 1 — partial: a draft, informal practice, or untested control exists;
• 2 — evidenced: an owner can show the artifact and a test, decision, or operating record.

The maximum score is 16, but the total is not the whole decision:

• 0–7: discovery only. Do not promise a production date. Bound the use case and resolve data or ownership gaps.
• 8–12: controlled pilot. Limit users, data, permissions, and consequences. Build the missing evidence during the pilot.
• 13–16: production candidate. Proceed only if no red-line condition applies and the release gate passes.

This scoring model is a JoinAI delivery artifact, not a certification. NIST's AI Risk Management Framework is deliberately broader and organizes risk work through Govern, Map, Measure, and Manage. NIST also cautions that its playbook is neither a checklist nor a mandatory sequence; organizations tailor actions to context. Use the NIST AI RMF Core and NIST AI RMF Playbook for the underlying risk outcomes.

Apply five red-line conditions

A high total score must not hide one unsafe boundary. Stop production promotion when any of these is true:

1. No accountable use-case owner. Engineering cannot decide the acceptable business failure alone.
2. Unresolved data authority. The team cannot show that the data may be used for this task, user, and retention period.
3. No evaluation gate. There is no reviewed test set or threshold tied to the user-visible risk.
4. Unbounded consequential action. The model can spend, delete, disclose, approve, or communicate externally without an enforced policy boundary.
5. No containment path. The team cannot disable the feature, revoke credentials, roll back a version, or route users safely when quality or a dependency fails.

The NIST Generative AI Profile states that suggested actions must be selected according to the organization, AI actor, and use case. It includes governance, data privacy, intellectual property, measurement, and management actions rather than treating model accuracy as the only risk. See the NIST Generative AI Profile.

Assess the use case before the platform

Start with a one-page contract:

User:
Decision or task:
Current workflow and baseline:
AI system output:
Allowed data and actions:
Unacceptable outcomes:
Human fallback:
Success metric and review window:
Accountable owner:

“Customer support copilot” is not specific enough. A usable contract might be: “Draft a reply for an authenticated support agent using the current policy corpus; never send the reply; cite the policy passages used; abstain when no allowed source supports the answer.”

That boundary changes the design. It identifies the user, prevents autonomous communication, defines allowed evidence, and creates a testable abstention rule.

If the team cannot complete this page, pause platform selection. The first production LLM application guide shows how to turn the contract into a narrow vertical slice.

Once the contract is complete, use the build-versus-buy agent platform matrix to score control, integration, evaluation portability, operations, and exit cost from evidence rather than feature lists.

Demand artifacts for every claim

During the assessment, replace adjectives with inspectable evidence.

Do not assign two points because a vendor offers a feature. The team must show that the feature is configured, tested, and owned in this system.

For security, OWASP's Artificial Intelligence Security Verification Standard describes testable requirements across the AI system lifecycle. It is useful for turning “secure” into design and verification work, but the applicable requirements still depend on the use case. See OWASP AISVS.

Run a two-hour assessment session

Invite the product owner, technical lead, data owner, security or privacy representative, and the person expected to operate the system. Send the use-case contract and rubric in advance.

Use this agenda:

00–15  agree on user, task, boundary, and baseline
15–35  map data sources, provenance, retention, and access
35–55  define evaluation cases, metrics, and release threshold
55–75  draw architecture, authority, and failure boundaries
75–95  define traces, containment, support, and rollback
95–110 estimate volume, latency, cost, and dependencies
110–120 assign evidence gaps, owners, and decision date

Record “unknown” instead of negotiating a higher score in the room. The output is a gap register:

gap | consequence | artifact needed | owner | due date | decision blocked

The assessment ends with one of four decisions: reject the use case, continue discovery, run a constrained pilot, or prepare a production release. It should not end with a vague recommendation to “improve AI maturity.”

Define the pilot as an evidence purchase

A pilot is justified when it cheaply resolves a decision. Examples:

• Can reviewers agree on acceptable answers for this task?
• Does the allowed corpus contain enough evidence to answer the target questions?
• Can the workflow meet its latency budget with the required retrieval and policy checks?
• What percentage of cases require human escalation?
• Can the system contain prompt injection without destroying utility?

Specify the hypothesis, sample, duration, threshold, and stop condition before building. A pilot that merely demonstrates the UI creates enthusiasm but does not buy readiness evidence.

Instrument one task hierarchy from the start. The production LLM observability schema explains the minimum trace needed to connect quality, latency, cost, and side effects without indiscriminately logging sensitive content.

Turn the result into a 30-day decision plan

Prioritize gaps that block the next decision, not every possible improvement:

1. resolve red-line conditions;
2. produce the use-case contract and reviewed evaluation set;
3. implement one secure, observable vertical slice;
4. test failure, rollback, and human fallback;
5. measure the pilot against the baseline;
6. decide whether to stop, revise, or promote.

For a production candidate, expand these decisions into the gated 90-day AI prototype-to-production plan, with explicit stop conditions at discovery, vertical slice, failure testing, pilot, and promotion.

If the assessment exposes unresolved ownership, data, evaluation, or delivery gaps, book a JoinAI consulting meeting with the completed rubric. The conversation can start from evidence and blocked decisions instead of a generic AI strategy pitch.

Sources and further reading

• NIST AI Risk Management Framework Core
• NIST AI RMF Playbook
• NIST Generative AI Profile
• OWASP Artificial Intelligence Security Verification Standard