We tested whether character post-training transferred to Mistral 7B. It did — with trade-offs.

Every Vinci Prova release begins with a hypothesis.

Prova is our research and development class.

These models are not candidates for the main Vinci lineup, and we do not judge their success by whether they should become anyone's daily model.

We judge them by whether they answer the question we trained them to answer—and whether that answer teaches us something we can carry into Piccolo, Bozza, Tela, and the models that follow.

Today we are releasing Vinci Prova 7B 1.0, along with GGUF builds for local research and experimentation.

This release tested one hypothesis:

Does the Vinci character-training recipe transfer to a different model lineage?

On the lineage and evaluations we tested, the answer is yes.

Model-judged fabrication fell from 53.8% to 8.6% on our development prompts. On a held-out set written after the recipe and shipping checkpoint were frozen, it fell from 46.2% to 7.5%.

The behavioural transfer survived.

The experiment also produced a more important qualification: the model primarily learned to make fewer unsupported assertions.

We did not show that it became more accurate once it decided to answer specifically.

That distinction now goes directly into the training and evaluation of our mainline models.

The evidence is the deliverable.

The weights are the artifact.

The findings, before the model

For researchers who do not need another 7B checkpoint, these are the useful parts:

1. The character recipe transferred on one additional lineage. Four internal behavioural evaluations moved from fail to pass, and the fabrication result survived a post-freeze held-out test.
2. Lower fabrication did not mean greater knowledge. The evidence points to increased reticence: the model made fewer specific assertions, without demonstrated improvement in accuracy conditional on asserting.
3. The development set overstated part of the gain. A DPO beta change that looked worth 7.5 percentage points during development was worth an estimated 2.97 points on held-out prompts across matched seeds.
4. A cheap evaluator selected the wrong checkpoint. A regex-based gate preferred a model that fabricated more than twice as often when its claims were checked against sources.
5. Behavioural gains still need capability gates. Judged fabrication improved substantially, but GSM8K fell by 5.6 points against the upstream base.

The first finding answers the hypothesis.

The other four determine what we do with the answer.

Why transfer matters

A behavioural recipe that works on one model family may not be a recipe at all.

It may depend on the base model's existing personality, tokenizer, instruction tuning, refusal behaviour, or the particular weaknesses already present in that lineage. If the result disappears when the substrate changes, we have learned how to modify one checkpoint—not how to build a repeatable Vinci layer.

Our earlier character training was developed on different model families. It aimed to shape a model's disposition: honesty about uncertainty, resistance to sycophancy, restraint around unsupported claims, and the ability to hold a position under incorrect or adversarial pushback.

For Prova 7B 1.0, we froze that broad approach and applied it to mistralai/Mistral-7B-Instruct-v0.3 through supervised fine-tuning followed by DPO.

We chose that base for continuity with earlier internal experiments.

It is also retired.

Mistral deprecated Mistral 7B Instruct v0.3 in November 2024 and retired it in March 2025. The weights remain available under Apache-2.0, but this is not a base we would recommend for a new model today.

That makes it a poor product foundation.

It does not make it useless as an experimental substrate.

What transferred

We evaluated the upstream Mistral base and this release using the same prompts, harness, and greedy decoding.

All four gates moved from fail to pass.

The character result was not one broad style score. The model improved across avoiding unnecessary verbosity, resisting sycophancy, holding its position under incorrect pushback, and avoiding preachy refusals.

One axis—conventional wisdom—remained weak at 2/4. Four prompts cannot establish much in either direction, and we do not consider it solved.

More importantly, these are development-set findings. We used the sets repeatedly while comparing training runs. They show that the target behaviour transferred on the prompts we measured, but they are not unbiased estimates of general performance.

Only the fabrication result has a post-freeze held-out test.

The held-out result

The held-out fabrication suite was created after the training recipe and shipping checkpoint were frozen.

It contains 93 adversarial baits and 11 answerable controls, using different jurisdictions and subject matter from the development set. We screened it against 80,752 training prompt records and found no exact or near-duplicate matches.

The base-to-release comparison reproduced on prompts we had never developed against:

46.2% → 7.5%.

The development and held-out estimates are unusually close for the shipping checkpoint. That gives us more confidence that the overall transfer is real.

It does not establish universal portability. We transferred the recipe to one additional base. A general cross-lineage claim would require the same frozen recipe to succeed across several meaningfully different and current model families.

The result is narrower:

The Vinci character recipe transferred to Mistral 7B Instruct v0.3 on the behaviours measured, including a post-freeze reduction in model-judged fabrication.

That is the hypothesis this release answers.

What the model actually learned

A model can fabricate less for two different reasons.

It can become better at knowing which claims are correct.

Or it can become less willing to make claims at all.

Our evidence points toward the second explanation.

Across matched training seeds, lowering the DPO beta from 0.1 to 0.05 reduced held-out fabrication by an estimated 2.97 percentage points. Fourteen of 17 paired seeds improved.

But those lower-beta models also made fewer specific assertions: 17.2 per 93 baits, compared with 22.0 for the higher-beta models.

We found no evidence that accuracy conditional on making a specific assertion improved. The observed conditional error rates were 48.8% and 42.7%, and we did not test that difference for statistical significance.

Our supported interpretation is therefore narrower than "the model became more truthful":

This training makes the model more reticent when a prompt invites an unsupported answer. We have not shown that it makes the model more accurate once it chooses to answer specifically.

That distinction matters.

Abstaining when uncertain is useful. A small model that admits it does not know can be more trustworthy than one that confidently fills every gap.

But abstention is not intelligence.

A model that answers less can look more accurate without understanding more. It can also refuse ordinary work, ignore a requested format, or hedge before making the same unsupported claim anyway.

Prova 7B 1.0 sometimes does all three.

Its characteristic failure sounds careful:

"I cannot pull the exact figure from memory, but the relevant provision is probably Section 31 or 32."

The hedge is honest.

The specific that follows it can still be invented.

What it cost

This was character training, not capability training. We did not expect it to teach new mathematics, coding, or general knowledge.

We still required it to preserve as much of the base model's capability as possible.

It did not preserve all of it.

General knowledge remained roughly flat. TruthfulQA improved by approximately three points.

GSM8K fell by 5.6 points, with almost all of that loss appearing during supervised fine-tuning. DPO recovered a small part of it.

The honest summary of the trade is:

A 53.8% → 8.6% reduction in judged fabrication, bought with 5.6 points of GSM8K.

Whether that is a good trade depends on the work.

For a system deciding whether to assert an uncertain fact, reticence can help.

For arithmetic and multi-step reasoning, this is a worse model than the base.

It is also not competitive with current small open models. Models with fewer parameters substantially beat it on general capability. Vinci Bozza 1.0 remains our recommended small Vinci model.

Prova succeeded as an experiment.

That does not make it a successful general-purpose model.

The evaluator nearly made us ship the wrong checkpoint

The most useful failure in this project was not in the model.

It was in the evaluator.

Our cheap deterministic gate looked for hedging and refusal language. If the model said something like "I cannot verify the exact figure," the gate treated the answer as acceptable.

That works until the model hedges and then fabricates anyway.

When we compared the shipping checkpoint with the one it replaced, the gate ranked them backward:

The cheap gate preferred the checkpoint that fabricated more than twice as often when its specific claims were checked against sources.

Across 42 models with both scores, the gate's rank correlation with source-judged fabrication was ρ = +0.105—not distinguishable from zero.

That is not a calibration problem.

It is a ranking failure.

Changing the pass threshold cannot fix an evaluator that prefers the wrong model.

We now keep the deterministic scorer for cheap triage. It cannot select a shipping checkpoint by itself.

This lesson goes directly into the mainline: the evaluator is part of the training system. If it rewards the appearance of uncertainty instead of checking the claim beneath it, the model will learn the appearance.

What the evidence does not establish

The fabrication results are model-judged.

A GPT-4o judge reviewed candidate claims using web search. It received the prompt, answer, and retrieved evidence, but not the identity of the checkpoint that produced the answer.

Afterward, OpenAI Codex separately source-checked all 15 positive findings from the released model against public primary or authoritative sources. All 15 remained item-level fabrications, so the reported rates did not change.

That second pass saw the original verdicts. It was a source-confirmation pass, not a blinded and statistically independent adjudication.

No human adjudicated the complete set.

There are other limits:

• The judge ran through a floating GPT-4o alias rather than a pinned snapshot.
• Only fabrication has a post-freeze held-out evaluation.
• The character, adversarial, and honest-positive findings remain development-set results.
• A small, stratified re-check of 20 judge-negative answers found no false negatives, but it is too small to estimate screening recall tightly.
• We have not yet run public honesty evaluations including AbstentionBench, SimpleQA Verified, MASK, AA-Omniscience, or Vectara HHEM.
• The training corpus and intermediate SFT parent are not public, so the released weights are externally inspectable but the training run is not externally reproducible.

We publish the item-level positive findings and recovered sources in the repository so readers can inspect what was counted.

The evidence is externally auditable.

It is not human-validated, and we will not describe it as such.

What this changes in the mainline

Prova exists to make the next Piccolo, Bozza, and Tela better.

This experiment changes five things in their training and evaluation:

1. We will measure willingness to answer separately from accuracy when answering

A lower fabrication rate can come from greater knowledge, better calibration, or simple refusal.

Future evaluations will report at least three quantities together: assertion rate, accuracy conditional on assertion, and over-refusal on answerable controls.

No single honesty number gets to collapse those behaviours again.

2. Capability preservation becomes a hard gate

Character training does not get credit for producing a more cautious model if it materially damages the work the model must perform.

Mainline candidates must preserve capability across coding, mathematics, instruction following, tool use, and the tasks expected of their role.

The final checkpoint is not automatically the model. The checkpoint that clears the gates is.

3. Source-grounded evaluation selects checkpoints

Regex and classifier gates remain useful for screening hundreds of experimental runs cheaply.

They do not make the final decision.

When the claim is about fabrication, the decisive evaluation must inspect the specific claims against evidence.

4. Held-out evidence comes after the recipe freezes

Repeated development-set improvement is not enough.

The recipe and candidate must be frozen before the held-out suite is written or opened. The held-out result confirms the decision; it does not participate in making it.

5. Character and capability remain separate training jobs

Preference training can shape voice, restraint, honesty, and disposition.

It cannot create reasoning or knowledge that is absent from the training signal. Mainline development will continue to build capability first, then add the Vinci character layer without being allowed to destroy it.

Why publish the checkpoint

Researchers do not have to use Prova 7B 1.0 as a daily model for the release to be useful.

The checkpoint makes the behavioural result inspectable. It lets other people run their own prompts, test our interpretation, find counterexamples, compare the quantised builds, and decide whether the trade-off is acceptable for a narrow application.

It also preserves the evidence trail behind whatever comes next.

We are now applying the same frozen recipe to current, supported Apache-2.0 bases. That will test what this release cannot:

Does the effect continue across multiple lineages without beginning from a retired and comparatively weak model?

If it does, the learning moves into the mainline.

If it does not, that is also an answer Prova should publish.

This is the standard for the class:

A clear hypothesis. A controlled experiment. A result with its limits. A concrete change to what we build next.

Download

Vinci Prova 7B 1.0 is a 7.25-billion-parameter model released under Apache-2.0.

• Full-precision weights and complete model card
• GGUF builds
• GGUF formats: F16, Q8_0, Q5_K_M, and Q4_K_M

The full model card contains the training configuration, hashes, evaluation methodology, item-level fabrication findings, source audit, limitations, and known failure modes.

Read it before using the model.

In particular: do not use this release to produce legal, regulatory, or financial citations. Its remaining fabrications are concentrated in invented case names and statute sections, often wrapped in language that sounds appropriately cautious.

You can use the model.

But that is not primarily why we released it.

We released it because the experiment answered its hypothesis—and because the answer changes how we train the models intended for real work.

Character training transferred.

Reticence transferred with it.

Now the mainline has to keep the first, measure the second, and preserve the capability underneath both.