Kishore Kanthan

KPI metadata drafts

A summer intern project. An AI pipeline drafts the metadata for each KPI. People review and approve every draft before anyone uses it.

A search finds a KPI with empty fields. An AI drafts each field from the KPI's sources, and a person approves the drafts.

The problem

Many KPIs had no metadata, so people could not find the metric they needed and often built a new one. Writing the metadata by hand for every KPI was slow work that nobody had time for.

Constraints

  • A person approves every draft before it is used. No draft skips this gate.
  • The drafts come from the team’s own sources, not from what the model remembers.
  • Every draft must read like the team’s best metadata, in the same structure and style.

What was built

Every draft stops at the human gate. Only a draft that a person approves reaches the list of metrics.

Reference KPIs. The team chose a set of KPIs whose metadata was already good. These references set the standard: what each field says, how long it is and how it reads.

An embedding pipeline. An embedding turns text into a list of numbers that captures its meaning. KPIs with similar meanings get similar numbers. The pipeline embeds each reference KPI. For a new KPI, it finds the closest references.

A retrieval pipeline (RAG). For each KPI, the pipeline also finds the related sources. It gives the sources and the closest references to a large language model (LLM). The sources supply the facts. The references show the structure and the style. The model then drafts the empty metadata fields.

The human gate. A person reads each draft and checks it against its sources. The reviewer approves the draft as it is, or updates it and then approves it. A fix needs no new run of the pipeline. Only approved metadata reaches the list of metrics. Nothing the model writes goes straight to users.

My part

The summer interns built it, and I mentored them.

Key decisions

  • AI drafts, people decide. The pipeline takes over the slow first draft. People keep the final word on what each KPI means.
  • Retrieval, not memory. Each draft is written from the KPI’s own sources, so a reviewer can check it against them.
  • Show, do not tell. Close reference KPIs teach the model the house style better than a long list of style rules.
  • The gate is part of the design. The review step is not an extra. No metadata reaches users without it.

What changed

Filling a metadata gap became a review task, not a writing task.