Teaching a Small Local Model to Think in Differentials

Most fine-tuning tutorials end when training loss decreases.

I wanted to answer a more interesting question: given a synthetic patient’s symptoms, can a small local model learn not only what is most likely—but what else it could be?

I fine-tuned Qwen2.5-1.5B-Instruct on English DDXPlus using native MLX LM and QLoRA-style adaptation. Then I compared the base and fine-tuned models on identical held-out cases, removed a finding to test the adapter’s sensitivity, fused the adapter, and deployed the result through oMLX and QwenPaw.

This is an educational machine-learning experiment on synthetic cases—not a clinical tool and not medical advice.

Foundation: the local serving stack

This experiment builds on the always-on local serving foundation described in my Complete Guide to an Always-On Mac Server with MLX, oMLX, and QwenPaw. If you still need to install or configure that stack and its background services, start there. This article assumes that foundation for the deployment stage and focuses on the DDXPlus data, QLoRA-style training, held-out evaluation, and reversible integration.

The mental model before we start

I did not train a language model from scratch. I started with a pretrained general-purpose instruct model: earlier training had already given it language and instruction-following capabilities. I then specialized its behavior using synthetic DDXPlus examples and kept the original base model unchanged for comparison.

The experiment covers four different questions:

• Data preparation: can I turn coded synthetic cases into faithful training examples without mixing the official splits?
• Adaptation: can a small adapter learn the constrained response behavior while the base weights remain frozen?
• Behavioral evaluation: do label agreement and response structure improve on withheld cases, rather than only improving the loss curve?
• Deployment: can I serve and use a standalone artifact, then restore the previous application configuration?

Pretrained instruct model + decoded DDXPlus examples
                         ↓
               Train a small adapter
                         ↓
Base vs base + adapter on identical held-out cases
                         ↓
      Verified fused model → oMLX → QwenPaw

“Teaching the model to think in differentials” is an experimental metaphor. The model is trained to produce a most-likely dataset label and ranked alternatives, not a clinician-style reasoning trace.

The command style

Each stage uses complete command blocks. Every block enters the experiment directory, activates the environment when needed, performs the operation, verifies its result, and stops on failure. There is no manual script editing between commands.

The implementation is too long to repeat usefully in the article, so the blocks use the reviewed companion source rather than hiding important corrections inside abbreviated examples. The first block checks for that source and stops rather than silently replacing it when unavailable:

night-thoughts/
└── experiments/
    └── ddxplus-mlx-lora/

Keep genuine stdout and stderr under results/logs/. Existing datasets, adapters, and full evaluation responses should be inspected and reused rather than silently overwritten.

What we will build

The requested response contract is deliberately small:

Most likely pathology
GERD

Differential diagnoses
1. GERD
2. Bronchitis
3. Possible NSTEMI / STEMI

Uncertainty
This is a synthetic DDXPlus case. The output is an educational prediction, not a clinical diagnosis.

This illustrates the format, not an experimental result. DDXPlus does not provide clinician-written rationales, recommended investigations, or treatment plans for these cases. I will not manufacture them.

Prerequisites

You need:

• an Apple Silicon Mac;
• uv and Git;
• Python 3.11, installable through uv;
• enough disk space for the official data, base model, adapter, and fused artifacts;
• oMLX and QwenPaw already configured for the deployment stage.

Memory pressure depends on macOS and other running applications. The final run described here used batch size 2. The documented fallback is batch size 1, but that fallback was not used and is not the configuration behind these results.

Part 1: Create the project and download DDXPlus

I used the official English release:

• DDXPlus Dataset English
• DOI: 10.6084/m9.figshare.22687585.v2
• License: CC BY 4.0

What is in a DDXPlus row?

Each row represents a synthetic patient case, not a real person’s record. The important fields are:

Field	What it means in this experiment
PATHOLOGY	The dataset’s reference label for the case.
DIFFERENTIAL_DIAGNOSIS	Source-derived alternative labels used for the reference differential and ranked targets.
EVIDENCES and INITIAL_EVIDENCE	Coded findings supplied with the synthetic case.
release_evidences.json	The dictionary that translates those codes and their values into readable English questions or findings.

When I use “ground truth” below, I mean the dataset’s PATHOLOGY reference label—not absolute medical truth or an independently verified diagnosis.

Symptoms describe the case’s current reported findings. Antecedents describe relevant history or background, such as earlier conditions or exposures. Both are supplied as evidence to the model; neither independently proves the reference label.

This block creates the environment, installs the pinned dependencies, downloads all five official files, verifies their MD5 hashes, and extracts the patient records. The extraction uses the real extensionless ZIP member names—not invented .csv filenames.

bash <<'BASH'
set -euo pipefail

command -v uv >/dev/null
command -v git >/dev/null

PROJECT="$HOME/night-thoughts/experiments/ddxplus-mlx-lora"

if [ ! -f "$PROJECT/experiment_config.py" ]; then
  if [ -e "$HOME/night-thoughts" ]; then
    echo "Companion source is missing; inspect the existing repository before replacing it."
    exit 1
  fi
  git clone --filter=blob:none --sparse \
    https://github.com/hslimi/night-thoughts.git \
    "$HOME/night-thoughts"
  git -C "$HOME/night-thoughts" sparse-checkout set \
    experiments/ddxplus-mlx-lora
fi

cd "$PROJECT"
uv venv --python 3.11 .venv
source .venv/bin/activate
uv pip install -r requirements.txt

mkdir -p raw data evaluation results/logs adapters fused
exec > >(tee -a results/logs/part1.log) 2>&1
cd raw

download() {
  local url="$1"
  local output="$2"
  if [ ! -f "$output" ]; then
    curl --fail --location --retry 3 "$url" --output "$output"
  else
    echo "Reusing downloaded file: $output"
  fi
}

verify_md5() {
  local actual
  actual=$(md5 -q "$1")
  if [ "$actual" != "$2" ]; then
    echo "MD5 mismatch for $1: expected $2, got $actual"
    exit 1
  fi
  echo "Verified: $1"
}

download https://ndownloader.figshare.com/files/40278013 release_evidences.json
download https://ndownloader.figshare.com/files/62561569 release_conditions.json
download https://ndownloader.figshare.com/files/40278019 release_train_patients.zip
download https://ndownloader.figshare.com/files/40278022 release_validate_patients.zip
download https://ndownloader.figshare.com/files/40278016 release_test_patients.zip

verify_md5 release_evidences.json 0660962abe0bbf99e1fb77188c253597
verify_md5 release_conditions.json a992eadcff3099dd72e1820fbd2115aa
verify_md5 release_train_patients.zip f75a78420ccb061951daa4db2d62139b
verify_md5 release_validate_patients.zip e4db88e9082f18410de6fd0cb7e74d18
verify_md5 release_test_patients.zip 6fd53d331cf19ed4439bf4f46b92ee94

for split in train validate test; do
  archive="release_${split}_patients.zip"
  member="release_${split}_patients"
  unzip -Z1 "$archive" | grep -Fx "$member" >/dev/null
  unzip -oq "$archive" "$member"
  test -s "$member"
done

cd ..
python - <<'PY'
import csv
import importlib.metadata as metadata
import json
import platform
from pathlib import Path

expected_rows = {"train": 1_025_602, "validate": 132_448, "test": 134_529}
for split, expected in expected_rows.items():
    with Path(f"raw/release_{split}_patients").open(newline="") as source:
        reader = csv.DictReader(source)
        assert set(reader.fieldnames) == {
            "AGE", "DIFFERENTIAL_DIAGNOSIS", "SEX", "PATHOLOGY",
            "EVIDENCES", "INITIAL_EVIDENCE",
        }
        rows = sum(1 for _ in reader)
    assert rows == expected, (split, rows)
    print(f"{split}: schema and {rows:,} official rows verified")

for filename in ("release_evidences.json", "release_conditions.json"):
    with Path("raw", filename).open() as source:
        value = json.load(source)
    print(f"{filename}: {len(value)} definitions")

print(f"Python: {platform.python_version()}")
print(f"MLX: {metadata.version('mlx')}")
print(f"MLX LM: {metadata.version('mlx-lm')}")
print("Stage 1 complete.")
PY
BASH

The verified release contains 223 evidence definitions and 49 pathologies. The patient members are CSV-formatted despite having no .csv extension.

Part 2: Prepare the datasets

From a raw row to a training example

The transformation looks like this:

Raw dataset row → decoded evidence → user prompt → target response

A system message defines the closed label set and required response sections. The user prompt describes the decoded synthetic case: age, sex, symptoms, and antecedents. The target response is the assistant example used during training. For training examples, it places PATHOLOGY in the most-likely section, builds three ranked differential entries from the supplied labels, and ends with the fixed uncertainty statement. I am formatting source-derived labels, not writing a clinician-style rationale.

During training, these prompt-and-target pairs teach the desired behavior. During evaluation, the model receives the system and user messages without the target response; its generated answer is what I score.

I did not train on the full training release. The experiment uses deterministic, stratified subsets sampled inside each official split:

Split	Cases per pathology	Total
Train	40	1,960
Validation	10	490
Test	10	490

Why the splits stay separate

Think of training as practice, validation as progress checks, and testing as a final problem set that must stay out of the practice material. In this experiment:

• Train examples update the adapter.
• Validation examples monitor training through validation loss; they do not update the adapter.
• Test cases are withheld for the final behavioral comparison.

Identical prompts across splits would be data leakage. A test prompt that had already appeared in training could reward memorization and inflate the results instead of measuring behavior on unseen cases.

Stratified sampling means sampling per pathology within each official split. Every one of the 49 labels is represented, rather than relying on an uncontrolled random subset that might miss or underrepresent labels. Equal per-label counts are an experimental design choice, not a claim about real-world prevalence.

The seed is 42. Official test patients are never moved into training. The final preprocessing also replaces duplicate decoded prompts with other eligible rows from the same official split and class, so all three prompt sets are unique and disjoint.

prepare_ddxplus.py uses ast.literal_eval for CSV list fields, never eval. Numeric evidence values retain their decoded values, while categorical N becomes No. Symptoms and antecedents stay separate.

One data edge case matters: the dataset spelling Larygospasm has only two usable reference differential labels in the selected validation and test cases. The preprocessing permits a two-item target there rather than inventing a third label. Training targets still contain exactly three source-derived labels. The evaluator still requires exactly three ranked predicted labels for a structured response, so a two-label prediction fails that strict format metric even in these cases.

The preprocessing command verifies and reuses a complete set of valid generated files. It stops if files are incomplete or their metadata, structure, split isolation, or pinned test hash differs. Replacing generated files requires inspecting them and explicitly running python prepare_ddxplus.py --overwrite-generated; that flag was not used for this run.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part2.log) 2>&1

python prepare_ddxplus.py
python - <<'PY'
import json
from collections import Counter
from pathlib import Path
from experiment_config import verify_test_cases

def read_jsonl(path):
    return [json.loads(line) for line in Path(path).read_text().splitlines()]

prompt_sets = {}
for split, expected in {"train": 1960, "valid": 490, "test": 490}.items():
    records = read_jsonl(f"data/{split}.jsonl")
    assert len(records) == expected
    prompt_sets[split] = {r["messages"][1]["content"] for r in records}
    assert len(prompt_sets[split]) == expected
    print(f"{split}: {expected} unique examples")

assert not prompt_sets["train"] & prompt_sets["valid"]
assert not prompt_sets["train"] & prompt_sets["test"]
assert not prompt_sets["valid"] & prompt_sets["test"]

cases = read_jsonl("evaluation/test_cases.jsonl")
counts = Counter(case["ground_truth"] for case in cases)
assert len(cases) == len({case["case_id"] for case in cases}) == 490
assert len(counts) == 49 and set(counts.values()) == {10}
print("Official split prompt overlap: 0")
print(f"test_cases_sha256: {verify_test_cases()}")
print("Stage 2 complete.")
PY
BASH

The genuine test-0000 example makes this transformation concrete. Its coded findings decode to symptoms including fever, sore throat, nasal congestion, and cough; its dataset reference label is URTI. The figure places those coded fields beside the actual decoded prompt used for evaluation. This illustrates what the dataset supplies, not a clinician’s explanation of the case.

Figure: a source-derived view of test-0000. The official row and generated prompt were checked against the saved held-out case; no evidence or label was added.

> What just happened? Coded dataset rows are now prompt-and-target examples. Training and validation examples guide and monitor adaptation, while the 490 test cases remain separate for the behavioral comparison.

Part 3: Smoke-test generation and evaluate the base model

Why measure the base first?

The base model already has general language and instruction-following capabilities from its earlier training. That does not tell us whether it can follow this particular closed-set differential format or agree with DDXPlus labels. I need to measure that starting point before adding the adapter. Without a baseline, I could not attribute a later improvement to the adapter rather than to the prompt or task setup alone.

The evaluator uses the pinned MLX conversion at revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677. It applies the same chat template, system prompt, greedy generation, and 220-token limit in every mode.

What the metrics mean

• Top-1 accuracy: was the first predicted label the dataset label?
• Top-3 accuracy: did the dataset label appear among the three ranked alternatives?
• Structured-response rate: did the model follow the requested sections and three-item ranked output contract?
• Reference overlap@3: how many of the predicted alternatives also appear in the source reference differential? The summary reports the mean across cases.

Formatting accuracy and label agreement are different. A perfectly formatted answer can still have the wrong top-1 label; a correct label can also appear in a response that fails the format check. I report those behaviors separately.

The first command is only a three-case generation smoke test. Its output is kept under a separate prefix and is not a benchmark result.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part3.log) 2>&1

python evaluate.py --mode base --limit 3 \
  --output-prefix results/base-smoke
python evaluate.py --mode base --limit 0
python - <<'PY'
import json
summary = json.load(open("results/base.summary.json"))
assert summary["cases"] == 490
assert summary["generation"] == {"max_tokens": 220, "temperature": 0.0}
print(json.dumps(summary, indent=2))
print("Stage 3 complete: full baseline saved; smoke results remain separate.")
PY
BASH

Figure: the complete base-model summary, generated from results/base.summary.json.

The evaluator initially missed labels when headings and content appeared on the same line. I corrected label extraction, retained a separate strict format check, and reapplied the final parser to the preserved responses. All reported modes use that same parser; this correction did not regenerate benchmark responses.

These scores measure agreement with a synthetic dataset—not accuracy on real patients.

> What just happened? The original model now has a measured baseline on the same 490 cases used later. This is the comparison point for attributing behavioral changes to the adapter.

Part 4: Smoke-test training and train the full adapter

LoRA and QLoRA in this experiment

A model’s weights are the numerical parameters it uses to generate responses. Full fine-tuning would update the base model’s weights directly. LoRA instead leaves those weights fixed and trains a much smaller set of additional parameters: the adapter. The adapter is not a standalone model; adapter evaluation loads the base model and adapter together.

Quantization reduces the numerical precision used to store the base weights. In this experiment, “QLoRA-style” means training a LoRA adapter on top of that quantized base. The original base remains available for comparison, while the learned adjustment lives in the separate adapter.

The training smoke test uses a separate adapter directory and only two iterations. It verifies that training and adapter saving work; its losses are not final training results. The full adapter and full training log are guarded against accidental replacement. For fresh reproduction, pinned_base_path() resolves the same pinned model revision to its local snapshot for training and fusion. The saved original training configuration names the upstream hub ID; I did not rerun training to change that record.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part4.log) 2>&1

MODEL="mlx-community/Qwen2.5-1.5B-Instruct-4bit"
BASE_MODEL="$(python -c 'from artifact_identity import pinned_base_path; print(pinned_base_path())')"
export MODEL

if [ -e adapters/smoke ] || [ -e adapters/ddxplus ]; then
  echo "A training adapter already exists; inspect it before replacing anything."
  exit 1
fi

if [ -e results/logs/part4-train-full.log ]; then
  echo "A full training log already exists; inspect it before replacing anything."
  exit 1
fi

mlx_lm.lora \
  --model "$BASE_MODEL" --train --data data \
  --fine-tune-type lora --adapter-path adapters/smoke \
  --batch-size 2 --grad-accumulation-steps 4 --num-layers 8 \
  --iters 2 --learning-rate 1e-5 --max-seq-length 2048 \
  --steps-per-report 1 --steps-per-eval 1 --save-every 2 \
  --mask-prompt --grad-checkpoint --seed 42

test -s adapters/smoke/adapters.safetensors

mlx_lm.lora \
  --model "$BASE_MODEL" --train --data data \
  --fine-tune-type lora --adapter-path adapters/ddxplus \
  --batch-size 2 --grad-accumulation-steps 4 --num-layers 8 \
  --iters 1200 --learning-rate 1e-5 --max-seq-length 2048 \
  --steps-per-report 10 --steps-per-eval 200 --save-every 200 \
  --mask-prompt --grad-checkpoint --seed 42 \
  2>&1 | tee results/logs/part4-train-full.log

test -s adapters/ddxplus/adapters.safetensors
python evaluate.py --mode adapter --limit 0
python analyze_results.py --compare base-adapter
BASH

The full run used a quantized base with LoRA adapters on eight layers, rank 8, scale 20.0, dropout 0.0, the Adam optimizer, batch size 2, gradient accumulation 4, learning rate 1e-5, and maximum sequence length 2048. Prompt masking and gradient checkpointing were enabled. The batch-size-one fallback was not used.

Reading the training settings

• Rank 8 controls the adapter’s capacity: how much trainable flexibility is available for the adaptation.
• Eight adapted layers limit which parts of the network receive adapter components. The base weights remain frozen throughout.
• Prompt masking leaves the patient prompt out of the training loss, so optimization focuses on producing the desired response rather than reproducing the input.
• Gradient accumulation combines gradients from several small batches before an update, simulating a larger effective batch without processing all of those examples simultaneously.
• Gradient checkpointing recomputes some intermediate values instead of retaining them all, trading additional computation for lower training-memory pressure.

The first reported training loss was 2.056 at iteration 10, and the final training loss was 0.057 at iteration 1,200. Validation loss was 2.257 at iteration 1 and 0.064 at iteration 1,200. Iteration 1 is a validation measurement, not an initial training-loss report.

Figure: loss curves generated only from results/logs/part4-train-full.log.

Why we did not stop at the loss curve

Loss measures token-prediction error under the training objective: how closely the model’s predicted tokens match the target responses. A falling loss says that optimization is fitting that objective better.

It does not directly tell us whether the first predicted pathology is correct, prove that the requested response format is followed, or establish behavior on unseen cases. That is why the held-out evaluation is the more meaningful behavioral result: it checks labels and structure on cases that did not update the adapter.

Figure: the complete adapter summary, generated from results/adapter.summary.json.

I did not measure peak memory and do not claim a total training time or compatibility with a particular memory size. The hardware observation is limited to: tested locally on Apple Silicon.

> What just happened? The base weights stayed fixed while a smaller adapter learned the response task. I now have training curves and, separately, adapter results on the same 490 held-out cases used for the base evaluation.

Part 5: Inspect disagreements and run an evidence-ablation probe

A better aggregate score does not explain every changed prediction. The comparison found 440 top-1 disagreements, 404 cases improved from incorrect to correct, and one regression from correct to incorrect. Fifty cases retained the same top-1 label.

What does evidence ablation test?

The question is simple: if we remove one relevant finding, does the prediction change? I compare the original prompt with the same prompt after one controlled edit, keeping the model and generation settings fixed.

This tests sensitivity to that prompt change. An unchanged response is a useful behavioral observation for this case. It does not create a new medical ground truth for the edited case, prove causality, or explain the model’s internal reasoning. It is not a second accuracy benchmark: there is no newly validated label against which to score the edited response.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part5.log) 2>&1

python analyze_results.py --compare base-adapter
python evidence_ablation.py --case-id test-0002 \
  --finding 'Do you have symptoms that are increased with physical exertion but alleviated with rest? Yes.'
BASH

The ablation code tracks sections and excludes the initial-evidence bullet. For test-0002, it removed this finding from Symptoms:

> Do you have symptoms that are increased with physical exertion but alleviated with rest? Yes.

The initial evidence remained present. The adapter returned exactly the same response before and after removal: top-1 Unstable angina, with Possible NSTEMI / STEMI and Stable angina in the differential. The original dataset label was Stable angina.

Figure: a genuine unchanged response from results/counterfactual.json.

The qualitative examples also include both improvement and failure:

• test-0001: base Chronic rhinosinusitis → adapter Anemia, matching the dataset label;
• test-0000: a shared top-1 failure—base Acute laryngitis, adapter Viral pharyngitis, dataset label URTI;
• test-0002: a well-formatted but incorrect adapter response—Unstable angina instead of Stable angina.

Figure: genuine base and adapter responses from the same held-out case.

Figure: remaining failures preserved in results/qualitative-examples.json.

> What just happened? The adapter improved many held-out predictions but did not eliminate failures. For the selected case, removing one finding left the response unchanged; that is a recorded sensitivity observation, not a new diagnostic label.

Part 6: Fuse and independently verify the standalone model

Why fuse, and why verify?

An adapter normally has to accompany its original base model: inference loads the base weights and applies the learned adapter adjustment. Fusion combines that adjustment into a standalone model artifact, which is simpler to copy and deploy because the serving layer no longer needs a separate adapter path.

That packaging change should not be trusted blindly. Numerical conversion can change generated responses, so the standalone artifact needs the same behavioral evaluation as the adapter. The first quantized fused artifact was rejected because its independent results diverged sharply from the adapter’s. I therefore evaluated a separate dequantized artifact instead of assuming that fusion had preserved behavior. Even after that correction, reviewed_with_differences means that known differences passed an explicit review—not that the two models are exactly equivalent.

The original mlx_lm.fuse command produced a quantized fused artifact that behaved very differently from the adapter. I rejected it rather than deploying it. Its full 490-case results were:

Metric	Rejected quantized fused
Top-1 accuracy	16.94%
Top-3 accuracy	23.88%
Structured-response rate	0.00%
Mean reference overlap@3	0.335 / 3

The correction was to create a separate dequantized artifact with --dequantize; I did not retrain or overwrite the full adapter. The following block creates and independently evaluates that artifact, then applies the strict response-identity check. A strict mismatch stops deployment.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part6.log) 2>&1

MODEL="mlx-community/Qwen2.5-1.5B-Instruct-4bit"
BASE_MODEL="$(python -c 'from artifact_identity import pinned_base_path; print(pinned_base_path())')"
ADAPTER="adapters/ddxplus"
FUSED="fused/ddxplus-qwen-dequantized"
export MODEL ADAPTER FUSED

if [ -e "$FUSED" ]; then
  echo "The fused destination exists; inspect it before replacing it."
  exit 1
fi

mlx_lm.fuse \
  --model "$BASE_MODEL" --adapter-path "$ADAPTER" \
  --save-path "$FUSED" --dequantize

test -f "$FUSED/config.json"
test -f "$FUSED/tokenizer.json"
python evaluate.py --mode fused --limit 0 \
  --output-prefix results/fused-dequantized-full
python analyze_results.py --compare adapter-fused \
  --fused-prefix fused-dequantized-full
python verify_fusion.py \
  --fused-prefix fused-dequantized-full \
  --model-path "$FUSED"
BASH

For this run, strict identity failed: 486 of 490 responses matched exactly, four differed, and 489 of 490 retained the same top-1 label. I inspected all four saved differences. Only then did I run the explicit reviewed-differences gate, limited to at most 2% differing responses and at most 0.01 absolute top-1 accuracy change:

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
python verify_fusion.py \
  --fused-prefix fused-dequantized-full \
  --model-path fused/ddxplus-qwen-dequantized \
  --accept-reviewed-differences
BASH

This is reviewed_with_differences, not exact equivalence. No threshold was changed to obtain a more attractive result.

Figure: rejected quantized output, corrected fused metrics, and the four reviewed response differences.

> What just happened? The first fused artifact failed behavioral verification and was rejected. A separate dequantized artifact was evaluated independently; only after reviewing its four response differences did I accept it for deployment.

Part 7: Deploy through oMLX and connect QwenPaw

Which component does what?

Component	Role in this experiment
MLX LM	Training and direct local evaluation
LoRA adapter	Learned task-specific adjustment
Fused model	Standalone deployable model artifact
oMLX	Local inference server and OpenAI-compatible API
QwenPaw	Agent/application layer that calls the configured model

In short, the adapter contains the learned adjustment, the fused model packages it for deployment, oMLX serves generation requests, and QwenPaw is the application using that service.

The deployment helper verifies the reviewed fused artifact and its file hashes, installs only the experiment model, and restarts the existing system LaunchDaemon directly. It does not use Homebrew services or modify unrelated model registrations. The forced restart briefly interrupts oMLX-served models and active requests. The helper discovers the exact model ID through /v1/models and uses that returned ID in the generation request.

The restart inside the helper is sudo launchctl kickstart -k system/com.omlx.server. If you have just performed that restart yourself, use python deploy_omlx.py --case-id test-0001 --skip-restart instead of restarting it again in the block below.

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part7.log) 2>&1

python deploy_omlx.py --case-id test-0001
python qwenpaw_model.py --sync-omlx-key
python qwenpaw_model.py --agent-id default

python - <<'PY'
import json
from pathlib import Path
metadata = json.loads(Path("evaluation/metadata.json").read_text())
case = next(
    json.loads(line)
    for line in Path("evaluation/test_cases.jsonl").read_text().splitlines()
    if json.loads(line)["case_id"] == "test-0001"
)
Path("results/qwenpaw-console-message.txt").write_text(
    metadata["system_prompt"] + "\n\n" + case["prompt"]
)
print("Synthetic held-out console prompt saved; no credential was printed.")
PY
BASH

The actual discovered ID was ddxplus-qwen. Direct oMLX returned Anemia for test-0001, matching the dataset label.

Figure: sanitized evidence from the real oMLX response; raw API payloads and credentials are not published.

The credential synchronization was a separate, explicitly permanent repair for an existing stale QwenPaw oMLX credential. The command used was:

cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
"$HOME/.qwenpaw/venv/bin/python" qwenpaw_provider_state.py --sync-current-key
"$HOME/.qwenpaw/venv/bin/python" qwenpaw_provider_state.py --verify-current-key

This reads the current key from oMLX’s local settings, updates the saved oMLX provider through QwenPaw’s installed API, verifies the persisted values match, and never prints the key. It preserves the provider URL and protocol. The request uses auto_discover: false; inspection of the installed QwenPaw 2.2.2 code shows that this request-scoped option suppresses a post-save discovery refresh and is not persisted provider configuration. This permanent credential repair is why rollback does not restore the old stale credential.

For the chat check, I opened http://localhost:8088/chat, started a clean chat, selected the fused model, and submitted the saved synthetic held-out prompt through the actual console. An HTTP health check alone would not satisfy this step.

A successful agent round trip means that QwenPaw accepted the prompt, called the configured backend, and displayed returned text. That verifies transport; it does not establish equivalence with direct oMLX generation or diagnostic correctness.

QwenPaw completed the round trip, but returned Acute dystonic reactions, not the dataset label Anemia. Its response also differed from direct oMLX. Agent context and existing generation settings may contribute; this check does not isolate the cause. This is a qualitative failure, not a benchmark or direct-equivalence test. I made zero prompt retries for a more attractive answer.

Figure: two unmodified crops from one real QwenPaw capture. Account identity, conversation history, workspace controls, credentials, and unrelated desktop content are excluded.

Part 8: Test rollback

Reversibility is part of reproducible engineering. An experiment should not require permanently sacrificing the previous working configuration just to test a new model. A tested rollback gives me a way back to that known state and makes the final system state explicit.

Before changing QwenPaw, the helper saved both the global configuration and the selected agent’s configuration, plus the original provider model registrations and removal list. Rollback restores those saved states, preserves unrelated settings, and removes the experiment’s QwenPaw model registration. It retains the trained adapter and fused model.

The following form uses the same real system daemon restart that I verified. The forced QwenPaw restart briefly interrupts sessions and active requests; the rollback verifier checks that the restarted process began after preparation:

bash <<'BASH'
set -euo pipefail
cd "$HOME/night-thoughts/experiments/ddxplus-mlx-lora"
source .venv/bin/activate
mkdir -p results/logs
exec > >(tee -a results/logs/part8.log) 2>&1

python qwenpaw_model.py --prepare-rollback
sudo launchctl kickstart -k system/com.qwenpaw.app
python qwenpaw_model.py --verify-rollback
BASH

I also used Restore default model in the experiment chat to clear its per-chat override and confirmed that the original response remained visible. No new response was generated. QwenPaw’s original global and per-agent selections were restored, and the configuration files matched their backups exactly.

QwenPaw may refresh only capability_provenance, discovered_at, and released_at during model discovery. Durable model settings and selections must still match. The explicitly requested permanent oMLX credential repair remains synchronized; rollback does not bring back the stale key. The deployed oMLX model was retained, and the optional model-removal block was not run.

> What just happened? oMLX served the fused model, and QwenPaw completed a real chat through the configured backend. Rollback then restored the previous model selections while retaining the trained artifacts and the intentional credential repair.

Results

Every final column uses all 490 held-out cases, the same system prompt, greedy generation, and max_tokens=220.

Metric	Base	Adapter	Dequantized fused
Held-out cases	490	490	490
Top-1 accuracy	10.41%	92.65%	92.86%
Top-3 accuracy	12.86%	97.14%	97.14%
Structured-response rate	0.00%	99.80%	99.80%
Mean reference overlap@3	0.206 / 3	2.043 / 3	2.049 / 3

How to interpret the held-out improvement

On the same 490 selected held-out cases and with the same generation settings, the adapter raises top-1 agreement from 10.41% to 92.65% and structured-response rate from 0.00% to 99.80%. These are two distinct changes: the first predicted label agrees with the dataset reference more often, and the response follows the requested contract more consistently. The comparison therefore shows more than a formatting change within this experiment.

I also inspected what changed case by case rather than reporting only the aggregate gain. Part 5 preserves the 440 top-1 disagreements, 404 incorrect-to-correct improvements, and one correct-to-incorrect regression.

The scope is deliberately narrow: a balanced, selected DDXPlus subset. These are not results on the complete official test release, real patients, external datasets, or evidence of clinical validation. They demonstrate improved behavior for this constrained setup, not general diagnostic ability.

All three summaries contain:

test_cases_sha256 = c5a5b94a3377c78b2ac1209899ca6e8d1df3a9aa25ce0a2e26ad33373f6c5838

The fused model’s top-1 accuracy was one case higher than the adapter’s; top-3 and structured-response rates were unchanged. Mean reference overlap increased slightly. These small differences do not establish exact behavioral equivalence.

Artifact and runtime identity metadata were backfilled later from the preserved artifacts, with that provenance recorded explicitly. It was not captured during the original generation. The final audit re-parsed every saved response, verified artifact and evaluator hashes, rebuilt all 490 test prompts and targets from official source rows, and recomputed every metric. No-generation resume checks then reused all 490 responses in each mode without loading a model.

Software observed in this run

Component	Version
Python	3.11.17
uv	0.13.0
MLX	0.32.4
MLX LM	0.32.2
NumPy	2.4.6
pandas	3.1.0
Transformers	5.20.2
Tokenizers	0.23.3
safetensors	0.8.0
huggingface-hub	2.3.0
oMLX	0.7.0
QwenPaw	2.2.2

Limitations

This experiment cannot establish clinical usefulness.

• DDXPlus patients are synthetic.
• The task is limited to the 49 dataset pathology labels.
• Real patients may have multiple conditions or incomplete histories.
• The stratified subset does not preserve the dataset’s original prevalence.
• Agreement with a reference differential is not clinical validation.
• Ten held-out Larygospasm cases have two reference labels; no third label was invented.
• Edited ablation cases do not automatically have valid labels.
• A confident and well-formatted answer can still be wrong.
• A successful oMLX or QwenPaw round trip is not a clinical evaluation.

The resulting model must not be used to diagnose, triage, or recommend care for a real person.

Final thoughts

The goal was not “a doctor on my Mac.” It was a complete local workflow with evidence at every stage, from authentic source data and split isolation through deployment and rollback. That is a much stronger result than an attractive training-loss curve—and a much more honest starting point for experimenting with diagnostic language models locally.

What this experiment demonstrated

• A compact local model can be adapted to a constrained synthetic differential-response task.
• Evaluation must measure behavior on held-out cases, not only loss.
• Deployment artifacts need independent verification; the rejected fusion shows why.
• Local deployment, application integration, and rollback can be tested end to end.

What it did not demonstrate

• Clinical validity.
• Real-patient performance.
• Clinician-style reasoning.
• Generalization beyond this selected DDXPlus setup.
• Equivalence between QwenPaw and direct oMLX responses.

Compact glossary

• Base model: the pretrained general-purpose instruct model that provides language and instruction-following capabilities before this experiment. It remains the comparison point.
• Quantization: representing model weights with lower numerical precision. This run used a 4-bit MLX conversion of the base model.
• LoRA: Low-Rank Adaptation, a method that trains a relatively small set of additional parameters while leaving the base weights frozen.
• QLoRA: in this experiment, QLoRA-style training means training a LoRA adapter on top of the quantized base.
• Adapter: the learned task-specific adjustment that must normally be loaded together with its base model. It is not a standalone model.
• Evidence: a supplied finding from a synthetic case, such as a symptom or antecedent, decoded from the dataset’s coded fields.
• Ground truth: the dataset’s reference PATHOLOGY label, not absolute medical truth.
• Held-out case: a case withheld from adapter training and used for behavioral evaluation; here, one of the selected 490 test cases.
• Fusion: merging the learned adapter adjustment into a standalone model artifact. The fused artifact still needs independent behavioral verification.
• Inference server: software that loads a model and answers generation requests through an API. oMLX fills that role here.

Dataset and model attribution

This tutorial adapts DDXPlus Dataset English by Tchango et al., released under CC BY 4.0:

• Official dataset
• DOI
• DDXPlus paper

Changes include decoding evidence into English, formatting records as chat examples, creating stratified subsets with duplicate-prompt replacement, and constructing ranked-differential targets from supplied labels without adding a missing third label.

The base model is Qwen2.5-1.5B-Instruct, licensed under Apache 2.0, using the pinned MLX Community 4-bit conversion. DDXPlus data and Qwen weights are not included in the companion repository.

Companion code

The implementation and reviewed non-sensitive summaries were developed alongside the Night Thoughts repository. At the time of publication, the DDXPlus experiment directory is not included in that public repository, so this article does not claim to offer a downloadable companion-source package.

Downloaded datasets, generated JSONL datasets, trained adapters, fused models, raw API payloads, credentials, local logs, and full response files are intentionally excluded from version control.

If you reproduce the experiment, I would especially like to see cases where training loss improves while diagnostic behavior does not.