Be the Architect: Intentional AI Engagement for Cybersecurity Engineering

TL;DR: How SpecterOps combined deliberate human engagement, model routing, and specialized tools to overhaul Ghostwriter’s user interface for $815.50 in model spend.

Disclosure: SpecterOps participates in OpenAI’s Trusted Access for Cyber and Daybreak programs. Codex is the primary example in this article because it fits our engineering workflow and because of that relationship. The operating principles described here are vendor-neutral and can be applied to other model families and coding agents.

AI access is not an AI strategy

When organizations talk about using AI for engineering, the conversation often starts with the model: Which one is most capable? Which one is fastest? Which one is cheapest? Those questions matter, but they come after a more important one: How should people engage with the agent?

Our answer is straightforward: be the architect.

An agent can take on a substantial amount of labor. You can task an agent to explore a codebase, propose alternative solutions, implement a design, update hundreds of related components, and perform repeatable checks. But you should not transfer ownership of the problem to an agent. The human still sets the direction, establishes what success means, challenges assumptions, and accepts responsibility for the result.

This is a leadership problem as much as it is a technical one. An AI agent is not a human team member: it does not share responsibility, understand organizational intent, or carry judgment from one situation into another unless that context is made available. Yet, some of the same principles that make delegation effective with people also matter here. Ambiguous direction produces ambiguous work. Missing constraints do not remain empty; the model fills them with a plausible interpretation. Incidental comments may receive more weight than we intended.

Consider a deliberately silly prompt: “Beans. Beans. Beans. Write me a haiku.” The model will almost certainly write a haiku about beans. It has no reason to know the repeated word was an aside, a test, or irrelevant context. The same behavior becomes consequential when the subject is an application architecture, an accessibility requirement, a research hypothesis, or a security control. This is also why long-running chats can start to go off the rails; a comment or correction you made an hour ago for a different task may influence the next task in ways you did not intend.

Prompt engineering, then, is not a search for magic phrasing. The human must begin with problem definition and proceed with deliberate and intentional expansion. Using an agent intentionally means deciding what context matters, what role the agent should play, what decisions remain open, and what evidence will be required before the work is accepted.

Intentional engagement comes first

At SpecterOps, we have been formalizing this with a framework called the AI Engagement Model, or AIEM, based on Mark Keith’s data science research at BYU. Its purpose is to make AI a force multiplier while keeping authorship, judgment, and understanding with the person responsible for the work. It begins with a simple premise:

> AI performance is a function of the human, the model, and the workflow.

AIEM follows a recurring loop: Frame, Challenge, Verify, Decide, Produce. We frame the problem and define success. We challenge assumptions and ask for alternatives. We verify claims and behavior. We decide which direction to take. Only then do we ask the agent to implement the work. The loop repeats as new information appears.

Our role changes as we move through that loop. At the beginning of a project, we may act as the Problem Setter. We can ask the agent to become a Creative Expander when we need options or a Critical Challenger when we want to stress test an idea. Once the direction is clear, the agent can shift into an Implementation Assistant. Later, it can help verify the result. We choose the role that fits the state of the work.

That choice also influences cost. Asking an expensive model to discover our goals, choose a direction, implement it, and judge its own success creates a long interaction shaped by assumptions we may never see. A mixed-model workflow can become just as expensive when every model is left to determine what the team wants.

Clear framing reduces that uncertainty. Concrete success criteria allow much of the implementation to move to a less expensive model while reserving the more capable model for ambiguity, difficult diagnosis, and consequential review.

Match the model to the work

Our engineering guidance starts with the least expensive model that can handle the task. We escalate when the work shows us that we need more capability, then return to the economical tier when that need has passed. Starting too low creates rework. Starting every task with the most expensive model pays a premium for routine questions and clear changes.

You must also consider reasoning effort, but don’t get caught up in these details. Reasoning effort is another routing choice that affects performance and cost. Higher effort can improve results when work requires more planning, analysis, or verification, although it also takes longer and consumes more tokens. Start with the model’s default effort and increase it when the task or the results justify the added expense. During the Ghostwriter project, Medium was a useful baseline and good for documentation, while High suited more involved engineering work. Extra High sometimes gave GPT-5.6 Luna enough reasoning time to complete tasks we might otherwise have routed to GPT-5.6 Terra. Treat reasoning effort as a tuning control alongside model selection, not as a substitute for clear direction. Effort levels like Max or Ultra should remain escalation choices. These unlock higher levels of reasoning or the ability to spawn subagents, and both can increase cost and how quickly you consume quotas.

Names may change and levels may shift, so keep the approach simple: start with the model’s default and establish a baseline that produces reliable results for your usual work. Increase effort for difficult problems or when results suggest deeper analysis would help. Consider lowering it for straightforward documentation or simple changes, and reserve the highest settings for work that justifies them. Revisit the setting when the work or results change; you don’t need to tune it on every turn.

For the Ghostwriter work described below, we used three GPT-5.6 models through OpenAI’s Codex agent with :

State of the work	Model choice	Reasoning Effort	Intended role
A bounded question or clear change in a single file	Luna	Extra High	Fast, economical execution
A larger change with known direction and concrete success criteria	Terra	High	Default engineering and implementation
Ambiguous design, cross-cutting diagnosis, or consequential review	Sol	High	Deeper exploration, challenge, and verification

Note: While we were writing this article, OpenAI introduced the GPT-6 family without a Terra tier. GPT-6 Sol and GPT-6.1 Sol matches GPT-5.6 Terra’s input and cached-input rates while carrying a lower output rate, and in our early use it has handled work we previously divided between GPT-5.6 Terra and GPT-5.6 Sol. OpenAI says the GPT-5.6 family will remain available during the rollout. The release made our exact pricing and cross-vendor comparisons age before we finished the article, reinforcing the need to track available models and route work deliberately as capabilities and economics change.

Anthropic’s Haiku, Sonnet, Opus, and Fable models offer similar practical tiers, although the lineups no longer map cleanly one-to-one. The model names will change; the routing approach still applies.

Specialist cyber models fit the same approach. At the time of this case study, the cyber models available to us cost roughly 2.5 times as much as GPT-5.6 Sol. We reserve that premium for authorized tasks that require specialist cyber capability—including cases where a general model declines a legitimate request because of its safeguards—and return to a general model afterward. Even in cybersecurity work, a specialist model should be an escalation choice instead of the default.

Tools and skills also shape how well an agent can gather context, challenge a plan, interact with an application, and verify its work. Three were particularly useful during the Ghostwriter project:

• Matt Pocock’s grill-me helped turn an incomplete project direction into a structured interrogation of goals, assumptions, and unresolved decisions.
• Anthropic’s frontend-design skill helped reinforce the design and accessibility direction and counter the generic “vibe-coded” appearance common in AI-generated interfaces. We cannot isolate its exact effect, although the agents consistently referenced and applied its guidance.
• Vercel Labs’ agent-browser gave the agents a smoother way to navigate the development application, exercise workflows, and capture screenshots or video for review. This proved much more reliable for our workflow than our initial browser automation setup with Chrome and Playwright MCP.

The model, effort, its tools, and the way we engage with both form one working system. Each part affects the outcome.

Maintain the context you provide

Skills, AGENTS.md files, and project documentation give agents durable context about how a team works. They can encode conventions, explain architecture, describe common workflows, and save the agent from rediscovering the same information during every session.

These materials require regular tuning. When an agent repeatedly misinterprets an instruction, takes an inefficient path, or produces an unexpected result, the surrounding context may need clarification. Changes to the codebase can also leave once-accurate guidance stale. Documentation carries more weight when an agent uses it to make decisions, so it needs to be clear, current, and specific. Keep these resources sharp.

The same applies to live session context. When corrections or completed tasks begin to dominate a chat, retire the session and start fresh so that stale context does not steer the next task.

Human review remains essential. An instruction like “Fix this test” carries unwritten expectations: resolve the defect in the code, preserve the purpose of the test, and avoid hard-coded output or shortcuts that merely make the check pass. An agent may satisfy the visible requirement while missing those expectations.

Well-maintained context reduces that ambiguity and makes agent behavior more predictable. Remember, you must still own the problem and the solution.

Have you ever tried to cut a tomato with a dull knife? It’s doable, but the results won’t be clean. Keeping your documentation, agent instructions, and context deliberate is like sharpening that blade.

Putting the approach to work

Ghostwriter is an open-source platform for managing offensive security engagements and crafting reports. Its interface had evolved over more than eight years, but the bulk of the design remained rooted in a decade-old design language built on Bootstrap v4. A meaningful refresh required changes across the whole application.

The refresh covered 99 distinct screens and required significant changes across 161 template files.

We upgraded the application to Bootstrap v5.3 and revisited every page. We tested new color schemes, modernized tabular interfaces, reduced accumulated UX friction, and established a more consistent and accessible design language. We also introduced several structural changes:

• an expandable dashboard for tracking active work;
• an “engagement bar” that replaced old breadcrumb-style navigation and made the relationships among reports, projects, clients, and activity logs easier to navigate; and
• a customizable sidebar that supports pinning clients, projects, and reports for quick access.

The core design work and UI conversion began on July 17 and were completed in less than a month. We previewed the new interface at Black Hat USA in early August. The branch then continued through testing, review, and release candidate refinement. Reviewer suggestions led to a few minor changes to the interface between then and the v7.3.0-rc1 release on September 14th, but one person completed the design work and migration between July 17 and August 14.

UI work made a useful test of our approach because it combines engineering with a healthy dose of human judgment. The feedback was often visual, incomplete, or difficult to express in the language of professional design. We needed to compare alternatives, inspect complex application states, preserve functional behavior, and decide when a local fix revealed a broader design problem. The models also lacked an understanding of the application that we, as its designers and users, had. It was exactly the kind of work in which asking a model to “figure out what we want” could become both costly and disappointing.

Changing roles as the work changed

The largest session came first. It cost $480.73—about 59% of the workstream’s total model spend—and accounted for 50 hours and 6 minutes of agent time—about 54% of the workstream’s total agent time. Much of it used Sol with the grill-me skill to interrogate our project plan, help establish the design direction, and complete an initial implementation.

The cost reflects how much ambiguity we had at the beginning. I am not a UI designer, and I did not always have the vocabulary or a concrete design ready to specify. I supplied screenshots, drawings, descriptions of friction, and reactions to the current interface. I instructed Sol to act as a Creative Expander: it proposed directions for consideration, challenged incomplete ideas, and helped translate preferences into decisions that could guide implementation.

The human role remained decisive. The model generated possibilities for me to consider; I decided what Ghostwriter should become. Once we agreed on the direction and could describe success more concretely, Terra took over most of the large implementation work. Luna handled basic questions and clear, localized changes where paying Terra or Sol rates would have added little value for the added cost.

Escalating when the cause was unclear

One escalation illustrates why routing by task state matters. Certain forms would flicker and scroll when opened. The behavior appeared related to JavaScript, although the cause could have been the browser, CSS, the tab implementation, or some combination. Terra attempted several fixes. None resolved the problem, even though its explanations sounded reasonable. After two lengthy turns, its third proposed solution was a delayed JavaScript function that forced the page back to the top after loading. That met the visible scroll-position requirement while leaving the flicker unresolved. It also made the problem more noticeable. I rejected the workaround and escalated the diagnosis to Sol.

Sol traced the behavior to the timing of TinyMCE editors initializing inside inactive Bootstrap tabs. The flicker appeared when those tabs became active. TinyMCE would initialize and cause the page to flash and scroll to the top of the new editor. After Sol resolved the immediate issue, I asked it to explore the broader codebase and evaluate our two editing systems. Ghostwriter already used Tiptap for collaborative editors. TinyMCE was our legacy WYSIWYG editor used for all other rich text fields. I decided to standardize on Tiptap throughout the application. I asked Sol to challenge that decision and propose alternatives. Sol did not identify any reason not to do it and or to not address this technical debt. Sol formed the plan and then Terra handled the migration.

In this case, Sol handled the ambiguity and cross-cutting diagnosis. Once we had made the architectural decision, the work became a clear implementation problem and returned to Terra. Meanwhile, Luna was an excellent, low-cost model for asking questions about a specific template and updating documentation.

Reviewing the whole interface

Verification followed the same pattern. As each piece of the application migrated, I asked Terra to use the agent-browser skill to verify that pages followed the new design language and continued to work properly without any loss of functionality. I followed the same process near the end of the migration to review the entire codebase for any missed templates or screens. Terra found additional pages on successive rounds. That was useful, but each new discovery reduced my confidence that the search had converged. I escalated the review to Sol. It found a few remaining templates and eventually returned without identifying additional missed pages.

This review also showed the practical tradeoffs. Luna worked well for individual files and simple questions. Terra was the workhorse for multi-file implementation, but its repeated passes over all 161 templates did not converge. Sol handled the broader review more reliably and found what Terra missed. Sol’s broader exploration can also pull in irrelevant context and expand the scope of a request, so we still reserve it for work that justifies the additional capability and cost.

The agents also exercised conditional application states and collected screenshots and videos. Ghostwriter has pages that appear differently depending on configuration. For example, a profile may have no MFA, a passkey, or a one-time-password device. I also had to be certain that the template changes did not break a user’s ability to manage MFA devices on these pages. There were dozens of these sorts of scenarios through the application, and automated browser interaction made checking that surface area much smoother and significantly faster.

I remained responsible for release acceptance. I manually browsed the application and tested critical workflows, including MFA behavior and login transitions, to ensure that the interface changes had not introduced unexpected loops or broken security-sensitive behavior. The agents gave us broader, more repeatable coverage. My review supplied the final judgment.

What the workstream cost—and what that number means

We parsed the underlying Codex session data with an internal prototype we call Agent Ledger. The final v7.3.0 release candidate workstream contains nine metered sessions, beginning with the design work on July 17 and ending with release candidate adjustments in September:

• $815.50 in measured model spend
• 1.29474 billion reported tokens
• 93 hours and 43 minutes of agent time

That spend covered design exploration, implementation across 99 screens and 161 template files, difficult debugging, browser-based QA, and release candidate refinement. The significant design and conversion effort took less than a month, followed by several weeks of testing and smaller adjustments. For that scope, keeping the final model spend below $1,000 was a compelling result.

These measurements still require context. Agent time measures execution inside Codex sessions; elapsed project duration and human labor are separate measures. Token volume tells us how much context and work passed through the models. We started new sessions when context approached roughly 75% of capacity or when a new day began after completing a significant body of work. The session boundaries reflect our context-management strategy rather than clean project phases.

Agent Ledger can also reprice the same recorded token volume under a single-model assumption. At GPT-5.6 Sol’s medium-effort rate, the recorded work would price at $904.56, or $89.06 more than the mixed-model total. This comparison measures price sensitivity for the tokens we actually used. An all-Sol project could have taken a different path, consumed a different number of tokens, and reached a different result.

The modest difference is informative. This project leaned heavily on Sol because the initial design problem was ambiguous and because we used it for difficult diagnosis and consequential review. The recorded model mix still priced $89.06 below the same token volume calculated at Sol’s medium-effort rate while keeping Sol available for the work where its additional capability mattered.

That calculation captures only one part of our approach: which models processed the recorded tokens. It cannot account for the tokens and iterations we avoided.

We established the design direction before scaling implementation. We translated visual preferences into concrete decisions, divided the migration into bounded tasks, defined success criteria, reviewed results continuously, and rejected or escalated poor approaches before they spread across the application. A directive such as “make Ghostwriter’s interface more modern” would have handed product direction to the agent along with the implementation. The agent might have produced competent work that was wrong for Ghostwriter, requiring repeated cycles of review, rejection, and reimplementation across the 161 templates.

We cannot replay the project with vague direction and assign a defensible price to that counterfactual. We therefore cannot claim that deliberate engagement saved a specific amount or cut the bill by a particular percentage. Based on the paths we rejected, the problems we caught early, and the amount of direction required to produce the final design, we believe deliberate project leadership materially constrained rework and model usage. Remaining the owner of the problem and the solution helped keep the model spend at $815.50 while producing an outcome we were willing to preview publicly.

The human labor involved was not free. During the core month, I spent roughly one to two hours per workday reviewing the agents’ work, testing critical workflows, and providing feedback—approximately 20 to 40 hours. Much of the agent work ran while I handled other responsibilities, which made it possible for one maintainer to direct the effort without devoting every workday to implementation.

Our estimate for a conventional implementation was two full-time engineers focused on the project for at least a month. Human engineers and agents would take different paths, so that estimate cannot produce a precise savings figure. It does establish the scale of the alternative we expected. Even acknowledging the human direction and verification involved, the combination of part-time oversight and $815.50 in model spend made a large, ambiguous UI overhaul economically practical for us, particularly for a free and open-source project like Ghostwriter.

These figures support a focused claim: intentional human engagement, deliberate model selection, and well-chosen tools allowed us to complete a substantial engineering task quickly and at a model cost we found remarkable. The result came from using the agents with purpose—planning the work, controlling the direction, and verifying the outcome—rather than simply allowing them to generate more work.

One rule for leaders

If an engineering leader asked for one rule their team should follow when using coding agents, ours would be this:

> Treat agents as delegated implementation capacity. You are still the architect.

That principle applies to commercial enterprises and government organizations alike. The operational controls will differ. A government team may need stricter authorization boundaries, data-handling rules, evidence retention, model-access controls, and approval gates. An enterprise may have more flexibility, but it still has obligations around sensitive data, production access, intellectual property, and release accountability. Neither environment can transfer judgment or responsibility to a model.

This guidance cuts against a common push toward greater agent autonomy and routine use of the most capable model. Strong models are most useful when humans make the problem and the boundaries legible. Clear leadership gives the agent room to execute while leaving implementation details to the agent.

We are beginning to see the same pattern beyond this project. People who have adopted the framework report completing the same work with no noticeable decline in quality while avoiding unnecessary model spend. We treat that as an early qualitative signal. A controlled quality study and company-wide savings benchmark remain for future work. Our prior baseline is weak because token cost was largely invisible to users. What we can say is that raising awareness of each model’s cost and providing model guidance has changed behavior without producing an obvious loss of capability.

Capability follows judgment

Model names, prices, and rankings will change. They changed while we wrote this article, and they may change again by the time you’ve read it. The need to frame a problem, challenge assumptions, verify evidence, make decisions, and own the result will not.

That is especially important in cybersecurity research and engineering. Sometimes the right choice will be an inexpensive model answering a bounded question. Sometimes it will be a frontier model investigating an ambiguous failure. Sometimes a specialized cyber model will be worth several times the price because the task genuinely requires it. Intentional engagement means matching the model to each of those states.

The Ghostwriter refresh succeeded because the human and agent roles changed with the work. We explored before we committed, made decisions before we scaled implementation, escalated when the evidence called for more capability, and verified before accepting the result.

The agent performed much of the labor. The human remained the architect and the problem owner.