llama-swap: enable nomic-embed-text embedding model (hermes-skills#26) #49

Merged

rpickler merged 1 commit from llama-swap-nomic-embed into main

2026-11-03 06:46:47 +00:00

hermes-bot commented

2026-11-02 21:44:33 +00:00

Member

Enables nomic-ai/nomic-embed-text-v2-moe-GGUF as a llama-swap slot on chicken for the weekly-postmortem normalization similarity scoring requested in richardpickler.com/hermes-skills#26 (comment from rpickler: "I also need a PR over in nixos to enable the nomic-embed-text model. Pay special attention to the launch commands for llama-server. I got a little lazy and hard coded much of the command to qwen38").

What changed

The llama-server command was hard-coded to qwen38 as you said: --ctx-size 262144, --parallel 2, --cache-type-k/v q8_0, --mmproj, chat sampling flags and --flash-attn on were interpolated into every model's cmd. The skeleton is now parameterized by launch kind — launchBase.chat / launchBase.embedding per-model bases, with launch.extraArgs, launch.mmproj and launch.kind layered on top — so each model fully defines its own command.

qwen38 (chat): the rendered command is token-identical to before (only comment placement and sampling-flag line breaks differ). No chat regression.
nomic (embedding): --embedding, --batch-size 512 --ubatch-size 512 --ctx-size 512, --flash-attn off, --parallel 1, no --mmproj, no chat sampling flags. Output normalization stays at llama.cpp's default --embd-normalize 2 (L2), which is what nomic recommends.
Also fixed on this branch: the chart models block now respects rpickler.llama-swap.models (it previously mapped the whole models attrset, so the option was silently ignored).

Launch command verification (no GPU here — eval/offline level)

Evaluated against the repo-pinned nixpkgs-unstable rev (02c087f3fe67, llama.cpp 0.6.0). The nomic slot renders as:

llama-server \
  --model .../nomic-embed-text-v2-moe.f16.gguf \
  --port ${PORT} --threads 8 --load-mode none --cpu-mask 0x0000FFFF \
  --split-mode layer --tensor-split 1,1 \
  --flash-attn off --fit off \
  --batch-size 512 --ubatch-size 512 --ctx-size 512 --parallel 1 \
  --n-gpu-layers 999 --perf --embedding

Two claims floating in the discussion were wrong for this file, both corrected here by decoding the GGUF metadata header of nomic-embed-text-v2-moe.f16.gguf directly:

nomic-bert-moe.pooling_type = 1 = MEAN pooling, not CLS. llama-server without an explicit --pooling uses this metadata default, so no flag is needed — but the reason the default is correct is MEAN, not CLS.
nomic-bert-moe.context_length = 512 (the model card agrees: max sequence length 512 tokens) — not 8k. batch=ubatch=ctx=512 matches the trained window; ubatch must never exceed ctx.

Caller contract (postmortem scripts): prepend the task-instruction prefixes (search_document: / search_query:) client-side and keep inputs within 512 tokens.

nix-instantiate --parse clean; module function-eval'd against the pinned nixpkgs to render both commands as shown. Deploy to chicken once merged (nixos-rebuild switch on chicken picks it up via the flake input bump).

Ref: richardpickler.com/hermes-skills#26

Enables `nomic-ai/nomic-embed-text-v2-moe-GGUF` as a llama-swap slot on chicken for the weekly-postmortem normalization similarity scoring requested in richardpickler.com/hermes-skills#26 (comment from rpickler: *"I also need a PR over in nixos to enable the nomic-embed-text model. Pay special attention to the launch commands for llama-server. I got a little lazy and hard coded much of the command to qwen38"*). ## What changed The `llama-server` command **was** hard-coded to qwen38 as you said: `--ctx-size 262144`, `--parallel 2`, `--cache-type-k/v q8_0`, `--mmproj`, chat sampling flags and `--flash-attn on` were interpolated into every model's `cmd`. The skeleton is now parameterized by launch kind — `launchBase.chat` / `launchBase.embedding` per-model bases, with `launch.extraArgs`, `launch.mmproj` and `launch.kind` layered on top — so each model fully defines its own command. - **qwen38 (chat)**: the rendered command is token-identical to before (only comment placement and sampling-flag line breaks differ). No chat regression. - **nomic (embedding)**: `--embedding`, `--batch-size 512 --ubatch-size 512 --ctx-size 512`, `--flash-attn off`, `--parallel 1`, no `--mmproj`, no chat sampling flags. Output normalization stays at llama.cpp's default `--embd-normalize 2` (L2), which is what nomic recommends. - Also fixed on this branch: the chart `models` block now respects `rpickler.llama-swap.models` (it previously mapped the whole `models` attrset, so the option was silently ignored). ## Launch command verification (no GPU here — eval/offline level) Evaluated against the repo-pinned nixpkgs-unstable rev (`02c087f3fe67`, llama.cpp 0.6.0). The nomic slot renders as: ```text llama-server \ --model .../nomic-embed-text-v2-moe.f16.gguf \ --port ${PORT} --threads 8 --load-mode none --cpu-mask 0x0000FFFF \ --split-mode layer --tensor-split 1,1 \ --flash-attn off --fit off \ --batch-size 512 --ubatch-size 512 --ctx-size 512 --parallel 1 \ --n-gpu-layers 999 --perf --embedding ``` Two claims floating in the discussion were **wrong for this file**, both corrected here by decoding the GGUF metadata header of `nomic-embed-text-v2-moe.f16.gguf` directly: 1. `nomic-bert-moe.pooling_type = 1` = **MEAN** pooling, not CLS. llama-server without an explicit `--pooling` uses this metadata default, so no flag is needed — but the *reason* the default is correct is MEAN, not CLS. 2. `nomic-bert-moe.context_length = 512` (the model card agrees: max sequence length 512 tokens) — not 8k. `batch=ubatch=ctx=512` matches the trained window; ubatch must never exceed ctx. Caller contract (postmortem scripts): prepend the task-instruction prefixes (`search_document:` / `search_query:`) client-side and keep inputs within 512 tokens. `nix-instantiate --parse` clean; module function-eval'd against the pinned nixpkgs to render both commands as shown. Deploy to chicken once merged (`nixos-rebuild switch` on chicken picks it up via the flake input bump). Ref: richardpickler.com/hermes-skills#26

hermes-bot added 1 commit

2026-11-02 21:44:34 +00:00

llama-swap: add nomic-embed-text embedding model e2c8e9e0ed

Refactor the hard-coded qwen38 llama-server launch command into a
skeleton parameterized by launch kind (chat vs embedding bases) plus
per-model extraArgs/mmproj/kind, so each model defines its own
command. The chat base renders token-identical to the previous
hard-coded command (no qwen38 regression). The nomic slot gets the
embedding shape: --embedding, batch=ubatch=ctx=512 (trained window,
verified against the GGUF header: context_length=512, pooling_type=1
=MEAN, read automatically by llama-server), flash-attn off, parallel 1,
no chat sampling, no mmproj. L2 output normalization is llama.cpp's
default (--embd-normalize=2) and is not passed.

Requested in richardpickler.com/hermes-skills#26.