I first built this workflow inside Zue AI Receptionist because I wanted a smaller model to learn the receptionist behavior I had already designed. The training data came from self-play. For each synthetic business profile, one model acted as a caller, guided by a caller persona and a goal. A stronger frontier model acted as the receptionist, using the same system prompt and tool definitions I intended the fine-tuned model to run with. I let the two talk, collected those conversations as OpenAI-format training rows, then fine-tuned an OpenAI model on them. That was the loop: describe the business and caller, let the teacher demonstrate the behavior, train a smaller model on those demonstrations, then put it back into the product and see where it failed.
The loop was useful, but its implementation was much too specific. It lived inside the receptionist backend and assumed its business schema, prompts, and tools. Its tool examples also stopped at the assistant’s tool call, which could teach a model what to call but not how to consume the result and finish the conversation. When I extracted the code, I did not want to publish an old product folder. I wanted to keep the self-play and distillation machinery while removing the receptionist assumptions.
That extraction became finetuning: a TypeScript SDK and CLI for building the data and running the workflow, plus a separately versioned Python trainer for the model work. On the chat side, I can start with existing records or describe both sides of a domain through scenarios, generate self-play conversations and complete tool trajectories, use a frontier model as the teacher, judge its candidate responses separately, then freeze what survived and train a smaller open model on it. The embedding side follows the same broad loop without pretending the data is chat. It handles query-document pairs and data for similarity, classification, and clustering, and can distill vectors, scores, or rankings before training and export.
There is also a practical reason I wanted the workflow to stand on its own. OpenAI has announced that even active customers will no longer be able to create new self-serve fine-tuning jobs starting January 6, 2027. I used that hosted service to train the receptionist models, but I do not want this workflow to depend on the lifecycle of one provider’s fine-tuning product. The target now is any open chat or embedding model whose tokenizer, template, pooling, and training recipe can be described explicitly. I have already prepared the first couple of open-model targets for that path.
I was not interested in publishing one more wrapper around a training library. The useful part was everything between the steps. Without a system around them, a fine-tuning project quickly becomes a training script surrounded by invisible state: the notebook knows which dataset was loaded, somebody remembers which prompt template was current, a folder called checkpoint-final-2 is probably the right one, and the exported model looks official until you ask whether its data split leaked or its files still match the run that produced them.
I wanted to be able to come back months later and answer the boring questions without reconstructing the run from a notebook: which records went in, which prompt and template shaped them, which provider work had already completed, whether the split leaked, and whether the exported files still matched the run. That is why dataset construction, distillation, training, resume, evaluation, and export live on one path. Each stage hands the next one an identity and a receipt instead of another unlabeled folder.
The training row does not stop at the tool call
The first thing that had to change was where a tool-calling example ended. The original two-agent loop now lives in a general scenario: a business, the assistant’s role and system prompt, realistic caller personas and their goals, its tools, and the rules for when a conversation should stop or escalate. During generation, one agent follows the caller persona while the teacher follows the target assistant prompt. From that exchange, finetuning can produce ordinary conversations, tool-decision examples, or the default I care about: full tool trajectories containing the assistant’s tool call, the matching tool result, and the final assistant response that uses it.
That last response is where much of the behavior actually lives. Choosing check_availability is only half the job. The model also has to read the returned slots, explain them naturally, preserve the user’s constraints, and close the loop. finetuning validates that tool-call IDs are unique, arguments match the declared schema, results refer to the right call and tool name, and the events appear in the right order before a row can enter a dataset. A malformed tool exchange fails as data, not three hours into training.
I wanted this path to be testable before anybody made a provider call, so I keep deterministic receptionist and retail scenarios in the repository. They exercise complete trajectories locally. Real generation is a separate, explicit choice through OpenAI or Anthropic. The scenario remains the product model either way, so changing domains means changing public configuration instead of forking the framework. The chat workflow runs the whole path offline first, including freeze, fake-teacher distillation, CPU training, resume, evaluation, and export.
I still have to decide which behavior is worth teaching. finetuning cannot make that product judgment for me. What it can do is remember exactly how I expressed it and which version reached training. I choose the behavior, and the system preserves the evidence.
The agent is the one at the terminal
A person can use the CLI, but most of the time I am not the one typing its commands. I tell an agent what I want done, it opens finetuning --help, follows the command tree, and runs the workflow. Agents built much of finetuning too, and I noticed the same thing while we were working on it: whenever the CLI explained itself well, I could stop teaching each new agent how to use it. I do not want a separate skill that has to memorize every command, or a tool schema taking up context just to mirror a binary that can already describe itself. I keep arriving at this pattern in my other agent-native tools, which is why I wrote that the answer is almost always a CLI.
What began as a documentation preference ended up shaping the API. Commands follow predictable noun-and-verb paths, so once an agent has found distill plan, training resume and embed evaluate compare are where it expects them to be. For commands with JSON output, the result goes to stdout while diagnostics go to stderr. Failures say whether the problem is bad input, an exhausted limit, a provider error, an incompatible checkpoint, or a corrupted artifact. An agent can use --dry-run to see what will happen before anything is written. It never gets --force for free, and finding a credential in the environment is not permission to make a provider call. These are small choices, but together they mean the agent can work out what happened and whether it is safe to keep going without pulling me back in to translate the tool.
That interface is what makes the next experiment possible. I am slowly starting to explore giving agents more of the loop: inspecting the current dataset, noticing missing behavior, distilling a stronger model’s behavior into candidates, training an open model, running the evaluations, and turning the failures into the next dataset. The agent can run the loop, but it still cannot grade its own work. Every step has to leave behind evidence I can inspect before anything moves forward.
A run is a chain of identities, not a folder of outputs
I kept chat and embedding records separate because one generic shape would hide what matters in each. A chat example needs roles, typed tool calls and results, review decisions, leakage groups, and content hashes. An embedding example needs queries, documents, positives and negatives, labels, scores, vectors, candidate pools, and teacher attribution. A retrieval dataset can preserve its source families, mine allowed hard negatives, distill rankings, and evaluate recall or MRR without pretending it is a conversation. Similarity, classification, and clustering tasks keep their own mappings for the same reason.
They still share one rule: keep the source data canonical and apply model-specific formatting as late as possible. Chat templates and embedding prompts, pooling, padding, normalization, and dimensions belong at the training boundary, not pasted permanently into the records.
That gives me room to compare models without lying to myself about what changed. If a query prefix was already pasted into every row, there is no clean way to try a model with a different prompt convention. If an assistant template changed without changing the run identity, two supposedly comparable checkpoints actually trained on different token sequences. The canonical record and control-plane model keeps those differences named.
When I freeze a dataset, I want it to stop meaning “whatever happens to be in this folder” and become an exact thing I can name. Canonical JSON serialization gives it a SHA-256 hash, and the manifest ties that hash to its record count, lineage, split and contamination evidence, rights metadata, and everything produced later. Change one row and it is a different dataset. The split logic also keeps related records together when they share a source, user, document, template, translation, or synthetic family. Deduplication clusters are checked across splits before an embedding dataset can freeze.
The result is deliberately boring to inspect: append-only JSONL for the records, canonical JSON manifests for identity, and content-addressed artifacts for everything produced. Boring files with stable hashes are a better source of truth than a notebook’s process memory.
The teacher produces candidates, not truth
The receptionist self-play loop was where this first clicked for me. A stronger model ran under the target receptionist prompt and demonstrated the behavior I wanted, and those conversations became supervised examples for a smaller student. What moved from one model to the other was the demonstrated response and tool use, not the teacher’s weights. finetuning generalizes that into response distillation, but it does not assume a capable model’s first answer belongs in the dataset. The teacher generates candidates. A separate judge checks whether they actually follow the behavior I asked for. The verifier is a separate role because the system producing an answer should not also be the one that certifies it.
Generating a response and judging it are two separate jobs, so I track them separately. Before a provider-backed run can begin, I have to name the exact provider and model for each role, refer to credentials through environment-variable names, opt into network access, and declare a separate limit for each one. OpenAI and Anthropic can fill either role, but a credential being present is not authorization to use it. The run therefore records which model produced a candidate and which model judged it instead of leaving that attribution in a config file somebody has to remember later.
The harder failure happens after the provider has answered. If the local process dies before recording the result, a naive resume sends the same request again. finetuning gives every request a stable identity and writes the provider response, usage, retry history, and model snapshot into the run state before advancing. Resume reuses that completed work. If the result of a request is ambiguous, the toolkit retries only when the provider supports idempotency. Otherwise it stops instead of risking a duplicate call. The run checks its declared limit before each request and records observed usage after it returns, while treating a local estimate as an estimate rather than a provider-enforced ceiling.
Once the candidates come back, I want to see how each one earned its place in the dataset. Structural validation, sensitive-data scanning, policy review, executable checks, rubric judging, filtering, deduplication, group-aware splitting, and contamination checks stay separate because each one answers a different question. “Rejected” is not useful unless I can see why. Every decision annotates the candidate instead of erasing its lineage, and validation and test records are locked before teacher generation so held-out answers never become convenient hints for the model being evaluated.
The same discipline applies to embeddings even though the teacher signal changes. An embedding teacher might provide vectors, graded relevance scores, or rankings, depending on the task. Those are not interchangeable. Response text is not an embedding target, and a score does not become a Boolean label unless its threshold and calibration are part of the data contract.
Resume is a compatibility decision
I did not try to make TypeScript train models. The TypeScript side owns the data, schemas, providers, orchestration, evaluation, and CLI. Python owns the work its ecosystem is good at: tokenizer templates, tokenization and assistant masks, embedding pooling, Transformers and TRL execution, checkpoints, and clean reload verification. The two packages can be released independently, but both validate the same JSON contracts and reject incompatible major versions.
The boundary between them is narrow on purpose. TypeScript starts the Python process without a shell, passes it one absolute, versioned job specification, and accepts only ordered JSONL events back. A malformed event or skipped sequence fails the run. If I cancel, the child process is terminated and the cancellation becomes an explicit terminal event instead of an ambiguous half-finished directory.
Resume has to be stricter than “point at the newest checkpoint.” For chat, the run binds the dataset hash and recipe identity, including pinned model, tokenizer, and template information. Embedding identity goes further because it has to. It binds model, tokenizer, and config revisions, data and split hashes, task mapping, query and document prompts, pooling, padding, normalization, dimensions, objective, and seed. Change an immutable field and the checkpoint is incompatible.
A checkpoint counts as a full resume only if it contains the model plus trainer, optimizer, scheduler, scaler, RNG, sampler, and step state. If it only has weights, finetuning calls it a warm start instead of pretending it can continue the same run. Export follows the same rule: name the artifact honestly and prove what is inside it. The manifest lists relative paths, byte sizes, and SHA-256 hashes, and verification rejects duplicate paths, absolute paths, .. traversal, symlinks, files outside the artifact root, or content that no longer matches. The model is portable because its receipt travels with it.
The receptionist pipeline gave me a way to train one model for one product. finetuning is the version I can keep carrying forward across products, providers, and whichever open models I want to train next. If I reopen an artifact six months later, I should not have to trust its filename or remember the run. I should be able to trace it back to the exact data, teacher, recipe, checkpoint, and evaluation that produced it. When an agent runs the next iteration, it should inherit that history instead of starting from another pile of scripts.