~/blog/pi-dsh-hermes-local-model-ticket-benchmark

AI Workflow · part 31

[Benchmark] Pi vs dsh vs Hermes on one local model: dsh used 1.97× Pi's input tokens on small tickets, and the gap closed on long ones

❯ cat --toc

TL;DR

I compared Pi, dsh, and Hermes on my local DeepSeek V4.1 Flash service. All passed the small tickets; dsh used 1.97× Pi's median input tokens, including cached input. Longer tickets changed the token ranking and exposed deadline and report failures. I am keeping Pi for daily work, testing Hermes on more long tasks, and retesting dsh's generation limit. These results cover my tasks and measured installations.

Pi, dsh, and Hermes compared on local coding tickets, with dsh at 1.97 times Pi's small-ticket median input tokens

Introduction: I needed a reason to change my daily ticket runner

I use Pi with a local model for coding tickets every day. A report that deepseek-harness, or dsh, used roughly ten times the input tokens caught my attention. Switching the runner would affect my whole ticket-based workflow, so I wanted measurements from work I would actually assign.

I added Hermes and ran all three against the same DeepSeek V4.1 Flash service on my own machines, with two repetitions per task and harness. The results describe my November 22, 2026 installations, settings, and serving conditions. Other workloads, models, or settings may give different results. Small tickets gave me a clean starting point. Larger ones showed whether the differences survived sustained work. The result supports keeping Pi in place, with Hermes worth testing on more long tasks and dsh needing a targeted retest.

Measure completed work against one model service

A harness is the program around the model: it builds prompts, exposes tools, carries conversation history, and decides when to call the model again. Changing it can change token use even when the weights stay put. The useful question for my workflow was whether a runner could deliver correct files, finish within a deadline, and tell me what it had done.

I used DeepSeek V4.1 Flash, served as DeepSeek-V4.1-Flash-1017, across two DGX Spark machines with tensor parallelism set to two. That splits model execution across both machines. The serving setup is covered in my dual-Spark TensorFold article. The measured installations were Pi 1.0.2, dsh 0.2.2-rc.1, and Hermes v0.23.1 on November 22, 2026. Those version numbers identify this experiment; they are not a claim about the latest releases.

To reproduce the method on your own workload, prepare a model endpoint, a logging proxy, isolated task directories, and a grader that runs outside the agent. The proxy needs to capture actual request settings and server-reported usage. A terminal summary alone is insufficient because different clients account for cached input differently.

I would set up a comparison in this order:

  1. Write acceptance criteria before running agents. Choose representative tickets and build external checks, including hidden tests. Make a correct reference solution and an empty starting skeleton; verify that the grader gives them 100% and 0% respectively.
  2. Give each run a fresh workspace and configuration directory. Block access to sibling workspaces and the public network. Agents should solve their own copy rather than discover another run's answer.
  3. Point every runner at the same service through the proxy. Save request settings, usage, tool activity, and termination evidence. Preserve each harness's native behavior, and record the differences.
  4. Set a fixed deadline and repeat each cell. When serving capacity permits, start the runners on the same task together. Record interference from shared load. Keep sequential trials labeled as sequential.
  5. Freeze and grade the files when the run ends. Check that supplied test files are unchanged. For data-processing tickets, change the input and rerun the solution so a hardcoded answer cannot pass.
  6. Report score, termination, and final report separately. Sum usage across every model request, including cached input. Retain the cost of failed runs in the totals.

The final output should let you answer three independent questions. Are the files correct? Did the process finish on time? Is there a usable final report? An agent can satisfy the first condition while failing the other two.

Small tickets gave dsh a 1.97× median input ratio

I began with three short tasks: fix a bug and pass tests; read six Markdown documents and produce a factual report; and write a CSV-processing script that returned correct answers for both the original and perturbed data. Each harness ran each task twice, giving eighteen runs. All passed: Pi six of six, dsh six of six, and Hermes six of six.

Compare the median input column first. It includes cached input and supplies the ratio in the title.

HarnessMedian input tokensMedian output tokensMedian elapsed time
Pi19,352.51,701.526.9 s
dsh38,0712,534.540.2 s
Hermes58,365.51,81541.4 s

Dividing dsh's median input by Pi's gives 1.97×. Pi also had the lowest median elapsed time in this sample. Since all runs passed, the token differences describe how much model traffic each runner used to deliver accepted work.

A different calculation, total tokens divided by completed tasks, gave Pi 22,755, dsh 41,506.8, and Hermes 72,127 tokens per task. Those are aggregate averages. They should not be read as medians or substituted into the input-only ratio.

The motivation came from Justin3go's preliminary dsh review, which reported roughly 4.5K uncached input tokens for Pi and 47.6K for dsh in one person's trial. I did not have the original task and settings. My cached-inclusive totals measure something different, so they neither reproduce that comparison nor establish that it was wrong.

Longer tickets made deadlines part of the decision

Short scripts do not cover the work I need from a coding agent. I added two longer tickets and kept a 45-minute deadline for each run.

L1 built an inventory-ledger CLI from a specification about one and a half to three pages long. It had to handle three input formats, merging, deduplication, and first-in, first-out inventory accounting. Eight public tests gave the agent feedback; forty hidden tests determined its score.

L2 changed an existing parcel service with twenty-three files and 1,622 lines including tests. The ticket combined five bug fixes with a batch-shipping feature that crossed module boundaries. It had 185 public tests and forty hidden tests. Half the score came from the fraction of the five bugs repaired; the other half came from hidden-test performance.

Two tasks, two repeats, and three runners produced twelve runs, started in four concurrent batches. For the decision, read perfect scores and clean exits together. The dsh count of 2/4 leaves out the run that stopped at 16.78 minutes with nothing implemented:

HarnessMean scoreRuns with a perfect scoreClean exits within the deadline
Pi100%4/42/4
dsh75%3/42/4
Hermes100%4/43/4

Pi and Hermes produced fully correct file snapshots in every long run. Hermes terminated within the deadline more often in this small sample. I'll test Hermes on more long tickets, though two runs per task aren't enough to establish statistical significance. Both of Pi's timed-out runs had already written a STATUS: done report well before the limit, then kept running extra checks until the deadline stopped them.

The small-ticket token ranking also changed. Long runs accumulated millions of tokens across their requests. Their median total-token ratios were dsh/Pi 0.77–0.86 and Hermes/Pi 0.70–0.78. Those ranges account for missing final usage records, explained below. They do not make a low-token failed run a good result.

My migration decision remains concrete: keep Pi in the workflow I already use, test Hermes on more long tasks, and rerun dsh's failing inventory task with its generation limit addressed. This is a comparison of my workload and installations. I would use the same method before changing the runner behind something like my local AI news-video pipeline, where a missing report can leave downstream work waiting even when the files exist.

Deep dive: accounting, grading repairs, and termination evidence

You can skip this section and still use the procedure and results above. It records the controls and exceptions needed to interpret the numbers or build a comparable experiment.

Cached input explained the small-run accounting mismatch

The logging proxy provided a common accounting point. I summed server usage across all model requests in each run, including cached input, rather than trusting each harness's displayed total. Adding cached input back to the Pi and dsh summaries reconciled all eighteen small runs; Hermes already included it.

Here are the individual small runs. A is the bug fix, B is the document report, and C is the CSV script.

HarnessTask/runInput tokensOutput tokensSecondsRequests
dshA/125,73897119.26
dshA/225,08687818.76
dshB/131,1942,07234.97
dshB/244,9483,16747.89
dshC/157,9794,70669.49
dshC/249,3052,99745.49
hermesA/188,6691,45237.912
hermesA/256,15984431.08
hermesB/154,3201,87340.28
hermesB/244,6511,75742.67
hermesC/1115,5734,25268.713
hermesC/260,5722,64046.58
piA/117,43594518.26
piA/221,27094819.77
piB/113,5101,53123.65
piB/213,6571,87230.25
piC/125,9373,55552.16
piC/232,0173,85351.57

Total input across a run can exceed the model's context window because successive requests resend overlapping history. It is cumulative traffic, not the length of a single context. Token totals also do not directly express money: cached-token pricing, local capacity, and latency have separate consequences.

The small trial was sequential. It did not use the later concurrent-batch design, and its order was imperfect: after calibration, Pi stayed first in the first round. I did not clear the service cache. Timing and cache exposure therefore need that qualification.

Pi used medium reasoning in the small trial; dsh's small-trial setting was unspecified. Hermes started its own gateway. The first Hermes gateway was cleaned up afterward, outside that run’s timing; other runs included roughly a second of cleanup. These details limit fine-grained elapsed-time comparisons.

The CSV grader needed a repair, not another model run

A grader can reject valid work. Mine initially rejected a CSV answer because it contained an extra by_order field, even though the ticket allowed the output. I corrected the grader to match the acceptance criteria and regraded every existing output. I did not rerun the agents to obtain better answers.

The CSV checks also perturbed the input and reran the script. That tested whether the submitted program computed the answer rather than merely emitting the original expected values. Checking unchanged test files supplied another control against a solution that passed by weakening its tests.

Calibration runs were excluded from the comparison. One interrupted Pi calibration consumed 4,159 input and 278 output tokens. That was additional experiment setup usage, outside the measured ticket totals.

Matching reasoning labels left native behavior different

All long-run requests carried reasoning_effort=medium on the wire. Pi also had an 8,192-token thinking budget, while dsh had thinking enabled. These settings do not establish identical internal reasoning behavior.

For Hermes, I disabled the experiment's safe mode so its native provider would load. Operating-system and tool restrictions remained in place, including the workspace and network boundaries. Pi exposed read, shell, edit, and write tools. The isolated dsh configuration disabled search, skills, subagents, and title generation. Hermes selected terminal/file tools but still exposed tool discovery. Native prompts and context defaults also differed, and existing memories, extensions, and production skills were not loaded. I wanted the behavior of each harness as used here, so those differences belong in the experimental conditions.

The long-task graders passed their controls: correct references scored 100% and empty skeletons scored 0% on both tasks. The unmodified buggy L2 service scored 16.25%, showing why a passing subset of existing behavior was insufficient evidence of completing the ticket.

Read the long-run records across score, termination, and report availability before comparing token totals.

HarnessTask/runScoreMinutesRequests/toolsTotal tokens (K)Exit status
dshL1/1100%45.0225/281266.9–1561.8Timed out
dshL1/20%16.784/554.2Generation limit; no report
dshL2/1100%23.9724/331060.0Exited
dshL2/2100%19.7829/471248.2Exited
hermesL1/1100%19.2526/531045.9Exited
hermesL1/2100%45.0341/632687.7–3212.0Timed out
hermesL2/1100%21.0727/491019.5Exited
hermesL2/2100%15.5123/541044.6Exited
piL1/1100%45.0232/321797.0–2092.0Timed out
piL1/2100%45.0239/402106.8–2401.7Timed out
piL2/1100%22.3623/31891.0Exited
piL2/2100%14.3715/21450.2Exited

K in the table means one thousand tokens. Times slightly beyond 45 minutes include termination and cleanup. Four requests lacked final usage. For those requests, received usage supplies a lower bound and model and request limits supply an upper bound. I report intervals rather than replacing missing usage with an estimated midpoint. Shared service load and the absence of cache clearing remain part of the conditions.

Perfect snapshots sometimes came from timed-out runs

Four timeouts still scored 100% when I graded their files: both Pi L1 runs, dsh L1 repeat one, and Hermes L1 repeat two. Pi's timed-out runs had reports. The timed-out dsh and Hermes runs did not.

The Pi timeline explains how correct work became a timeout. L1 repeat one wrote a STATUS: done report at minute 27.5; repeat two did so at minute 38.1. After writing those reports, the runs continued with audits, random tests, or temporary-file cleanup until the 45-minute limit. Later individual requests took 70–290 seconds, with observed contexts around 90–120K tokens on the shared, concurrently serving endpoint. Dsh L1 repeat one and Hermes L1 repeat two were still running tests or modifying their own added tests at the deadline, without reports.

This affects how I would operate them. A correct snapshot is useful evidence of implementation ability, but a downstream dispatcher also needs a reliable end condition and handoff.

Dsh's other L1 run stopped after 16.78 minutes, scored 0%, and had no report. Across its four requests, one response reached the 32,768-output-token cap; its native log recorded max-tokens. That supports generation-budget exhaustion as the explanation for the stop. It does not establish a context-window overflow, which is a different limit and would require different evidence. The observed maximum contexts across the long runs were approximately 96–124K tokens. The 262.1K figure in the summary was an upper bound, not an observed context length.

The classifier recorded stopped_to_ask=3. On inspection, that category included missing reports; I detected no literal questions. I therefore do not describe those runs as agents asking the user for help.

I also observed no native compression events, even though Pi enables autocompaction by default.

Token efficiency depends on which outcome gets the denominator

For the long tasks, tokens per 1.0 of cumulative score were Pi 1,311.3–1,458.7K, dsh 1,209.8–1,308.1K, and Hermes 1,449.5–1,580.5K. This divides total token use by summed fractional score, including the cost of the zero-score run. It is not tokens per successful ticket delivered before the deadline.

The task split is particularly relevant for dsh. Its median total-token ratio to Pi was 0.29–0.41 on L1, including the failed run, and 1.72 on L2. Combining those into one figure conceals how differently it behaved on a new CLI and an existing service.

I would retain all three views in a repeat: raw usage, score-adjusted usage, and delivery within the deadline. The next dsh test needs to resolve the L1 generation limit while keeping the same acceptance criteria.

Also in this series: Delegating with ticket files · Changing the harness around a local news pipeline.

FAQ

Did dsh use ten times as many input tokens as Pi?
In my small-ticket sample, dsh used 1.97 times Pi's median input tokens, counting cached input. The earlier external report used a different metric and workload, so this comparison does not invalidate it.
Which harness would I use for local coding tickets?
I am keeping Pi for my existing workflow. Hermes looks promising enough to test on more long tasks. Before I consider dsh, I need to rerun L1 with its generation limit addressed.

Read next

Don't miss the next one

Subscribe, and you won't.

One-click unsubscribe anytime.