Overseeing software delivery with Jev

In the agentic workflow world of Coro, every job, every task is a series of workflow steps, each handled by specialized agents one after the other, mimicking a real world software developer process. Tasks are often broken down to subtasks, each with interdependencies to one another and crunched through by agents in a sequential flow. Some tasks are small while a complicated task could sometimes takes days or weeks to complete.

How do you ship software through automated agentic workflow that lasts for a week, confidently?

Confidently - that's the key word here. In Coro there is an evaluator agent that runs as the last step of each workflow and checks if the produced work is actually what was intended by the spec, acceptance criteria and ADR's. This usually guarantees that the job eventually finishes with a deliverable that matches the original intent. Evaluator agent is a capable LLM model with special instructions to guarantee just that.

While Evaluator guarantees the deliverables, it doesn't guarantee the time and it doesn't guarantee the cost.

Your agents may have spent the past 1 week running around in circles within the workflow steps, drifting from the spec until eventually the evaluator agent whipped them into place, they came to their senses and produced the right output. So then, how do we make sure the agents haven't drifted form the original spec? How do we make sure they are constantly "focused" on target task?

If only there was an overseer of the whole process...

Shift-it-left or distribute?

Evaluator is too late to optimize cost and analyze if the we are still on target. Then what do we do? We could shift it left. We could take part of that responsibility and move it to an earlier step in the flow. But which step? And more importantly, how? Having a traditional LLM runnƒcodeing constantly and judging everything would be far too costly. Plus we do need a bit more determinism and control here. We certainly don't want our overseer to hallucinate either!

This is when I met Jev. Built by TypeSafe, Jev a specialized model that is exceptionally helpful in decision making. Because of it's internals and how it works, it's also super cost saving too.

Jev is not for producing code, it doesn't generate documents and markdowns and plans. Instead, Jev is super good for making structured decisions over a given state. So let's build an overseer with Jev!

The Overseer

First thing to know - is that Jev's inputs and outputs are structured JSON. We supply it with:

• some state;

• a set of questions;

• and the permitted shape of each answer.

Jev returns typed values rather than verbal output. Coro currently uses all three answer types supported by its integration:

• Noul: the probability that a statement is true;

• Choice: one option from a caller-defined set, with confidence and option probabilities;

• Score: a position on a caller-defined scale, also with confidence and probabilities.

Conceptually, a request looks like this:

  "state": {
    "phase_obligations": ["..."],
    "trajectory": { "phase": "coding", "workItems": ["..."] }
  },
  "questions": {
    "breach": { "type": "choice", "criteria": { "none": "...", "obligation[0]": null } },
    "off_track": { "type": "noul", "instructions": "The work still serves the objective." },
    "severity": { "type": "score", "criteria": ["Cosmetic", "Worth a look", "Review", "Stop"] },
    "blocked": { "type": "noul", "instructions": "The agent is blocked." }
  }
}

So now we have the overseer mechanism, we should find the right place to call it. A really good place where we could invoke the overseer is on the workflow phase boundaries. We don't want to be constantly mingling with the agent while it's doing it's thing (ahem: micromanagement, anyone?), and the agent does usually correct itself and find the right path on it's own anyways, but a phase boundary after each step before moving onto the next step, is a good place for a checkpoint and review.

On each phase boundary, Coro sends two main pieces of state.

The first is the phase’s obligation list. Unless a workflow defines its own list, Coro checks whether:

1. the phase produced the output required by its workflow contract;

2. the work still serves the job’s objective;

3. the phase made progress instead of repeating earlier work;

4. failures were surfaced rather than silently worked around.

The second is a compact job trajectory. This contains:

• the original objective;

• the current phase and workflow lane;

• work-item status and loop counts;

• phase runs, turns, cost, and attributed rework;

• failed tool categories;

• recent lane switches;

• and open pull requests.

This is intentionally not a second full agent context. The phase transcript and git diff are not part of the Overseer payload because remember, we don't want Jev to perform a code review, we want it to evaluate the direction and progress of the run. We just want to know if we're on track. One request asks four questions in parallel:

• Which phase obligation, if any, appears unmet?

• What is the probability that the work still serves the objective?

• How serious is any problem?

• Does the agent appear blocked on something it cannot resolve?

These answers are recorded on the job together with the model version, latency, token count, phase, and a readable digest of the state that was evaluated.

Another architectural choice here was not to involve the agent that completed the phase with the overseer. If the previous phase was flagged by Jev's results, the next phase receives a short process note. The note describes the result as a process-confidence signal, not a code-quality score. This separation reduces the chance of turning the result into a self-reinforcing conversation where the same agent both performs the work and argues with its own evaluation.

Status Indicator

Once Jev comes back with answers, Coro dashboard takes the probability that the work still serves the objective and presents it as:

• On track: 70% or higher;

• Needs a look: 40–69%;

• Off track: below 40%.

Status indicator is displayed and constantly updated on each job:

Overseer Modes

Coro overseer is more than just a mere status indicator though. It currently has two operating modes:

• Observe records Jev’s answers and updates the dashboard, but never changes the job’s behaviour.

• Manage uses the same answers and may pause an interactive job when a stricter rule is met. Manage mode flags and pauses a phase when:

  • the probability that the work still serves the objective is 25% or lower; or

  • Jev selects a missed obligation with confidence of at least 0.5 and gives the problem a severity score of at least 1.5.

The manage mode also opens the possibility of such result: a run can display On track, 85% and still pause if Jev is sufficiently confident that the phase missed a serious obligation or broke a rule.

Depending on your choice you are able set different modes for different workflows too. You could choose to have only the Observe mode for short to standard workflows and pick Manage mode for long deep workflows that have the potential to run for longer periods of time.

So how does it work?

Well, this is still quite new. The numbers still need to be tested and tweaked with each new job. The basis of Jev's reasoning is the state information that we give it and the answers that we ask. I'm sure this part will require a good amount of testing and feedback too.

One thing that's for sure, is that with an overseer which is as robust and cost saving as it is right now with Jev, the confidence to ship software is way up than it's without.

Overseer asks questions:

Does this still serve the objective?
Which contract, if any, appears broken?
How serious is it?
Should a person look before the next phase?

Jev answers. Coro decides what happens next.