Finding the Right Role for the Smartest Model

Whenever a new model arrives, we come back to the same question: can we trust it with our customers' work?

GPT-6 Astra was no different. OpenAI describes it as particularly capable at using computers and browsers and carrying out multistep workflows. We tested it too. Then we asked another question: what would it cost for our customers' agents to work with this model for a month?

In a demo, it is easy to focus on the fact that the model finished the job. In everyday work, we need to entrust it with that job tomorrow and next week as well. The cost of one successful run and the cost we can afford to pay every day are different questions.

The workflows we have handled at UpServe—making short videos, processing documents, operating a browser, and working across databases—typically involve 30 to 50 rounds of the model assessing the situation and deciding what to do next. It reads material, runs a tool, checks the result, and moves on. If it takes a wrong turn, finding its way back takes work too.

The model is called repeatedly along the way. Each call can involve a different amount of information to read, reasoning to do, and output to produce. An agent's model cost is the total cost of those calls. Input and output have different rates, and caching previously used input also affects the cost. To understand the full operating cost of a task, we also need to account for tools and the environment in which the agent runs. (OpenAI model pricing)

A model with a higher price per token can therefore still cost less to complete a task if it needs fewer attempts. OpenAI also reports that, in some evaluations, Astra achieved better results with fewer output tokens, lowering its estimated API cost per task. Whether the same holds for our customers' work is something we need to verify ourselves.

Ultimately, the question is how much it costs to finish one task properly, including failed attempts and rework.

The cost of completing a task includes reading, execution, checks, and retries.

In one past internal analysis at UpServe, Ultra agents made up 18% of all agents and accounted for 63% of total model costs. Within that Ultra spending, 90% came from the three highest-spending agents. Looking at the number of tasks instead, Standard agents handled 82% of all tasks.

Tasks differ in size and difficulty. These figures alone cannot tell us which tier is more efficient. They do, however, show us where to look more closely. What work did the agents with the highest costs do, and how much did that spending contribute to the overall result?

That is also where our thinking about roles begins.

Setting the direction of a task, interpreting conflicting information, or deciding what to do in an exceptional situation may call for stronger judgment. A mistake at that point can send the work that follows in the wrong direction. Using a more capable model in those roles can be well worth the cost.

Collecting and organizing material according to established criteria does not always require the same level of judgment. We can identify the capabilities each task needs and assign it to a suitable agent.

Imagine a team gathering material and preparing a report. Collecting information against a defined set of criteria is a different job from reviewing contradictory evidence and reaching a conclusion. If the agent responsible for the difficult judgments establishes useful criteria, the other agents may spend less time discarding or repeating work they have already done.

A hypothetical team divides research, review, and report writing. Handing off results and coordinating also take work.

This is the value we hope to get from a more capable model: a contribution to the team's ability to finish the work properly, as well as a good result from the agent itself.

Dividing the work does not automatically reduce costs. Explaining things to one another and handing off results also cost something. Overlapping responsibilities can lead to the same work being done twice. For a straightforward task, one agent handling it from start to finish may be the better choice. To know whether a team helps, we need to look at both the result and the total cost.

These questions connect model selection to service design. What counts as one task, when we consider it complete, and which judgments deserve a larger share of the budget are all connected.

We welcome more capable models because they let us attempt work that was previously difficult to entrust to an agent. At the same time, we need to keep finding the right place for that intelligence in actual work.

Our customers need to be able to come back tomorrow with the work they entrusted to us today. They need the results they are looking for, at a cost they can keep paying. That is why testing a new model at UpServe also means thinking about a month of work.