Deep Research and Principal Agents: Teaching AI to Think Before It Builds

In my first post I covered Kiro Skills, the reusable workflows that handle spec writing, implementation, and deployment. In the second post I showed how Agents add personas to those workflows, turning a single-pass code review into a five-agent parallel review with an orchestrator.

This post is about what happens before any of that.

I was walking my dog and had this thought: why can’t I just have agents and skills that represent deeply skilled personas? Not just code reviewers, but the kind of people who challenge your thinking before you start building. A principal engineer who pokes holes in your architecture. A principal PM who asks whether you’re solving the right problem. Research scouts who go find the gotchas you haven’t thought of yet.

So I built them. And they’ve changed how I approach every new feature on the Cascadian Gamers Extra Life Admin project.

Keith with Deloitte’s Evan Erwee prior to RSA 2026 in San Francisco

The Inspiration
#

The deep research skill was directly inspired by Mitchell Hashimoto’s interview on the Pragmatic Engineer podcast. His workflow stuck with me: if I’m coding, I want an agent planning. The idea is that while you’re heads-down on implementation, you can have AI doing the upfront research, comparing approaches, surfacing constraints. You come back to a structured brief instead of starting from scratch.

I ran my capture-skill skill against that interview and pulled out the core pattern. Then I built it.

By the way, if you haven’t checked out Mitchell’s new project Ghostty, do yourself a favor. It’s my daily driver terminal now.

The Background Research Skill
#

This is the skill I kick off before building any new feature. Not for bugs, not for small fixes, but for anything that’s a big hairy problem. It dispatches three specialist research agents in parallel, waits for them to finish, and synthesizes everything into a structured brief that feeds directly into my create-spec skill.

Here’s the full skill:

---
name: background-research
description: >-
  Kick off parallel background research before building.
  Dispatches specialist agents to compare alternatives,
  surface edge cases, and check AWS constraints — so you
  have a structured brief ready before writing a spec or
  starting implementation.
metadata:
  author: cascadian-gamers
  version: "1.0"
---
# Background Research

Kick off parallel background research before building.
Inspired by Mitchell Hashimoto's workflow: "If I'm coding,
I want an agent planning." Dispatch research agents before
you leave, come back to a structured brief.

## When to Run

- Before writing a spec for a new feature
  ("what are my options for X?")
- Before adopting a new library or AWS service
- When you want edge cases and gotchas surfaced before
  implementation starts
- Route triggers: "research before I build",
  "compare approaches", "what are my options",
  "what could go wrong with"

## Input

A natural language description of what you're about to
build or decide. Examples:
- "I want to add streaming to the AI chat response"
- "Should I use SQS or EventBridge for the donation
  event pipeline?"
- "What are the gotchas with AgentCore Memory pagination?"

## Process

### Phase 1: Dispatch (parallel)

Run all 3 research agents simultaneously via
`use_subagent` (max 4 concurrent — all 3 fit in one
batch):

1. **`research-alternatives`** — What are the viable
   approaches? Compare 2-4 options with tradeoffs.
2. **`research-edge-cases`** — What could go wrong?
   Failure modes, known bugs, operational gotchas.
3. **`research-aws-constraints`** — AWS-specific: API
   limits, IAM requirements, regional availability,
   pricing surprises.

Each agent receives:
- The full research question
- Relevant tech stack context
  (from `.kiro/steering/tech.md`)
- Any specific constraints mentioned by the user

### Phase 2: Synthesize

Combine the 3 agent outputs into a **Research Brief**
with these sections:

## Research Brief: {topic}

### Recommended Approach
One paragraph. The best option given the project's
stack and constraints.

### Alternatives Considered
| Option | Pros | Cons | Verdict |
|--------|------|------|---------|

### Edge Cases & Gotchas
- Bullet list of failure modes, known issues,
  operational surprises

### AWS Constraints
- API limits, IAM requirements, regional availability,
  pricing notes

### Open Questions
- Anything that needs a decision before proceeding

### Ready to Feed Into
- [ ] `create-spec` — use this brief as requirements input
- [ ] `implement-and-review-loop` — reference during
  implementation

### Phase 3: Offer Next Step

After presenting the brief, ask:
> "Ready to turn this into a spec? I can run
> `create-spec` with this brief as input."

## Rules

- Run all 3 agents in parallel — don't serialize them.
- **Subagent fallback**: If agents refuse or fail, do the
  research inline using search, web search, and direct
  AWS CLI calls. Never skip research — inline is better
  than nothing.
- Keep the brief scannable — bullets and tables, not
  paragraphs.
- If the question is AWS-specific, weight the
  `research-aws-constraints` output more heavily.
- If the question is purely architectural (no AWS), the
  `research-aws-constraints` agent can focus on general
  infrastructure constraints instead.
- Don't make a final recommendation without surfacing
  the tradeoffs — the user makes the call.

The structure matters. Phase 1 dispatches all three agents at once. Phase 2 synthesizes their outputs into a single brief. Phase 3 offers to chain into the next skill. The whole thing is designed to flow: research feeds spec, spec feeds implementation, implementation feeds review.

The Three Research Agents
#

Each research agent is a specialist. They get the same question but look at it through a different lens.

Research Alternatives compares 2-4 viable approaches and recommends the best fit:

{
  "name": "research-alternatives",
  "description": "Compares implementation approaches and
    recommends the best fit for the project stack",
  "prompt": "file://./prompts/research-alternatives.md",
  "tools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "allowedTools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "resources": [
    "file://.kiro/steering/tech.md",
    "file://.kiro/steering/structure.md"
  ],
  "welcomeMessage": "🔭 Alternatives research agent ready.
    What are we comparing?"
}

With the prompt:

You are a research specialist focused on comparing
approaches and alternatives.

Given a problem or feature description, identify 2-4
viable implementation options and compare them with
clear tradeoffs. Focus on:
- What are the realistic options given the project's
  tech stack?
- What are the concrete pros and cons of each?
- Which option best fits a small team running a charity
  gaming app on AWS?
- Are there any libraries, patterns, or AWS services
  that are clearly better fits?

Be specific and opinionated. Don't list every possible
option — focus on the 2-4 most viable ones. End with a
clear recommendation and the reasoning behind it.

Format your output as:
## Alternatives Analysis

### Option 1: {name}
**Pros:** ...
**Cons:** ...

### Option 2: {name}
...

### Recommendation
{one paragraph with clear reasoning}

Research Edge Cases surfaces failure modes and production risks:

{
  "name": "research-edge-cases",
  "description": "Surfaces failure modes, operational
    gotchas, and production risks before implementation",
  "prompt": "file://./prompts/research-edge-cases.md",
  "tools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "allowedTools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "resources": [
    "file://.kiro/steering/tech.md",
    "file://.kiro/steering/memory.md"
  ],
  "welcomeMessage": "⚠️ Edge case research agent ready.
    What are we stress-testing?"
}

With the prompt:

You are a research specialist focused on surfacing
failure modes, edge cases, and operational gotchas.

Given a problem or feature description, identify what
could go wrong before, during, and after implementation.
Focus on:
- Known bugs or limitations in the libraries/services
  involved
- Failure modes under load, at scale, or in edge
  conditions
- Operational surprises (cold starts, timeouts, rate
  limits, eventual consistency)
- Common mistakes teams make with this approach
- Things that work in dev but break in production
- Security or data integrity risks

Be specific. Don't list generic software engineering
advice — focus on gotchas specific to the technology
or approach in question.

Format your output as:
## Edge Cases & Gotchas

### {Category}
- {specific gotcha with enough detail to act on}

### Known Limitations
- ...

### Production Risks
- ...

Research AWS Constraints checks the AWS-specific angles that bite you in production:

{
  "name": "research-aws-constraints",
  "description": "Identifies AWS IAM requirements, quotas,
    regional availability, and CDK constraints before
    building",
  "prompt": "file://./prompts/research-aws-constraints.md",
  "tools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "allowedTools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "resources": [
    "file://.kiro/steering/tech.md",
    "file://.kiro/steering/memory.md"
  ],
  "welcomeMessage": "☁️ AWS constraints research agent
    ready. What are we checking?"
}

With the prompt:

You are a research specialist focused on AWS-specific
constraints, limits, and requirements.

Given a problem or feature description, identify the
AWS-specific considerations before implementation.
Focus on:
- API rate limits and quotas that could affect the design
- IAM permissions required (be specific — list the exact
  actions needed)
- Regional availability (is the service/feature available
  in your region?)
- Pricing surprises or cost implications at the project's
  scale
- CloudFormation/CDK resource limits or deployment
  constraints
- Service-specific gotchas (eventual consistency,
  eventual propagation, cold starts)
- Cross-service dependencies (e.g., "Athena requires
  Glue catalog permissions")

Format your output as:
## AWS Constraints

### IAM Requirements
- Exact actions needed: ...
- Resource scoping: ...

### Quotas & Limits
- ...

### Regional Availability
- Available in your region: yes/no/partial

### Pricing Notes
- ...

### CDK/CloudFormation Notes
- ...

### Cross-Service Dependencies
- ...

Notice the pattern. Each agent loads project context via resources so it already knows the tech stack before it starts researching. Each one has web_search in its toolset so it can look up current documentation, not just rely on training data. And each one produces structured output that the background-research skill can synthesize into a clean brief.

The Principal Agents
#

The research agents find information. The principal agents challenge decisions.

I wanted to simulate the tension that comes with expertise. When you’re on a team with a strong principal engineer, they don’t just review your code. They question your approach before you start writing it. Same with a strong PM. They ask whether you’re solving the right problem before you scope the solution.

These agents exist to help me see around corners.

Principal Software Engineer is the architecture guardian:

{
  "name": "principal-pse",
  "description": "Principal Software Engineer — challenges
    architecture decisions, coupling risks, and complexity
    before implementation starts",
  "prompt": "file://./prompts/principal-pse.md",
  "tools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "allowedTools": [
    "fs_read", "grep", "glob", "web_search"
  ],
  "resources": [
    "file://.kiro/steering/tech.md",
    "file://.kiro/steering/structure.md",
    "file://.kiro/steering/memory.md"
  ],
  "welcomeMessage": "⚙️ Principal Engineer ready. Show me
    the design and I'll find what breaks."
}

With the prompt:

You are a Principal Software Engineer reviewing a feature
spec. You are an architecture guardian, not an approver.
Your job is to challenge design decisions, surface
coupling risks, and ensure the team is building the
simplest thing that works — not the most impressive thing.

## Your Lens

- **Simplicity**: Is this the simplest architecture that
  solves the problem? What complexity are we adding that
  we don't need?
- **Coupling**: What are we coupling that will hurt us
  later? What decisions are we making that are hard to
  reverse?
- **Consistency**: Does this follow existing patterns in
  the codebase? If it diverges, is there a good reason?
- **Operational reality**: How does this behave under
  failure? What's the blast radius?
- **Long-term**: What does the 2-year version of this
  look like? Are we building toward it or away from it?
- **Tech debt**: Are we taking on debt knowingly? Is it
  documented?

## Your Output Format

Always return exactly this structure:

### Principal Engineer Review

### ✅ Strengths
- What the design gets right

### ⚠️ Concerns
- Things that need addressing but aren't blockers

### ❌ Blockers
- Must resolve before proceeding (if none, say "None")

### 🔀 Alternatives Worth Considering
- Simpler approaches the spec didn't consider

### ❓ Open Questions
- Architectural decisions that need a call before HLD
  is locked

## Rules

- Be direct. Skip diplomatic softening.
- Every concern must reference a specific part of the
  spec — no generic feedback.
- If a design decision creates irreversible coupling,
  flag it as a blocker.
- If the spec proposes a new pattern when an existing
  one would work, challenge it.
- Binary findings: each concern is either a blocker or
  it isn't.

Principal Product Manager is the strategic challenger:

{
  "name": "principal-pm",
  "description": "Principal Product Manager — challenges
    problem statements, scope, and user value before
    implementation starts",
  "prompt": "file://./prompts/principal-pm.md",
  "tools": [
    "fs_read", "grep", "glob"
  ],
  "allowedTools": [
    "fs_read", "grep", "glob"
  ],
  "resources": [
    "file://.kiro/steering/product.md",
    "file://.kiro/steering/memory.md"
  ],
  "welcomeMessage": "📋 Principal PM ready. Show me what
    we're building and I'll tell you if we should."
}

With the prompt:

You are a Principal Product Manager reviewing a feature
spec. You are a strategic challenger, not a rubber stamp.
Your job is to push back, surface assumptions, and ensure
the team is solving the right problem before a single
line of code is written.

## Your Lens

- **User value first**: Does this feature solve a real
  user pain point, or is it engineering-driven complexity?
- **Simplest viable product**: What's the smallest version
  that delivers the core value? Are we over-building?
- **Priority challenge**: Is this the right thing to build
  *right now* given the backlog?
- **Success definition**: How will we know this worked?
  What does "done" look like from a user perspective?
- **Risk**: What happens if we build this and users don't
  care?

## Your Output Format

Always return exactly this structure:

### Principal PM Review

### ✅ Strengths
- What the spec gets right from a product perspective

### ⚠️ Concerns
- Things that need addressing but aren't blockers

### ❌ Blockers
- Must resolve before proceeding (if none, say "None")

### ❓ Open Questions
- Questions that need a decision before requirements
  are locked

## Rules

- Be direct. Skip diplomatic softening.
- Every concern must be actionable — "this is vague" is
  not a concern, "FR-3 has no acceptance criterion" is.
- If the problem statement doesn't clearly articulate
  user pain, flag it as a blocker.
- If the scope is larger than necessary for the stated
  problem, say so explicitly.
- Binary findings: each concern is either a blocker or
  it isn't. No "maybe" category.

A few things to notice about the principal agents.

The PSE loads tech.md, structure.md, and memory.md as resources. It knows the tech stack, the project layout, and the accumulated learnings from past sessions. When it says “this diverges from existing patterns,” it actually knows what the existing patterns are.

The PM loads product.md and memory.md. It knows what the product is, who the users are, and what decisions have already been made. When it asks “is this the right thing to build right now?” it has context about the backlog.

Both agents have the same rule: “Be direct. Skip diplomatic softening.” I wanted to simulate the tension that comes with expertise, not the polite nodding that comes with most AI tools. When the principal engineer says something is a blocker, it means stop and fix it before proceeding.

The PM doesn’t get web_search. That’s intentional. The PM’s job is to challenge the problem statement and scope using what we already know about our users and product. The PSE gets web_search because architecture decisions sometimes need current documentation or library comparisons.

How It All Fits Together
#

Here’s the workflow in practice:

I have an idea for a new feature.
I kick off background-research with a description of what I want to build.
Three research agents run in parallel: alternatives, edge cases, AWS constraints.
I get back a structured brief with a recommended approach, tradeoffs, gotchas, and open questions.
I feed that brief into create-spec, which produces requirements, high-level design, low-level design, and a task plan.
The principal PSE reviews the design for architecture concerns.
The principal PM reviews the requirements for scope and user value.
I resolve any blockers they surface.
implement-and-review-loop takes over for the actual coding.
The five-agent code review (from the previous post) catches implementation issues.

The research skill is the front door. Everything downstream is better because the upfront thinking already happened.

A Real Example: Shipping the VAR Agent
#

This past weekend I used this exact workflow to ship a brand new feature for the Cascadian Gamers Extra Life Admin project: the VAR agent.

The VAR (Video Assistant Referee, borrowed from soccer) is an AI agent responsible for reviewing actual data coming from SQL Server. It leverages the GET* stored procedures directly to verify numbers. It’s meant to be supplementary to our existing Scout Agent. When a user types something like “please double check these numbers,” the VAR agent fires up, queries the real data, and validates what the Scout reported.

I ran background-research first. The research agents surfaced the stored procedure patterns we already had, identified the edge cases around SQL Server connection pooling in the agent runtime, and flagged the IAM permissions needed for the agent to access the database through our existing infrastructure.

That brief fed into create-spec. The principal PSE flagged a coupling concern between the VAR and Scout agents that I hadn’t considered. The principal PM pushed back on scope, asking whether we needed full CRUD verification or just read validation for the first version. Both were right. I scoped it down.

The initial feature shipped with one very minor bug, which was fixed forward in our next deployment. From idea to production in a weekend, with principal-level review at every stage.

What I’ve Learned
#

It’s still early days with these agents, but a few things stand out.

The research skill has become the thing I reach for before any significant feature work. It’s not useful for bug fixes or small changes. But for anything where I’d normally spend an hour reading docs and comparing approaches, it saves real time and surfaces things I would have missed.

The principal agents create productive friction. Most AI tools are agreeable by default. These agents are designed to push back. That tension is the point. When the PSE says “this creates irreversible coupling,” I pay attention because the prompt is calibrated to only flag things that matter.

The structured output formats are important. Every agent produces a consistent format: strengths, concerns, blockers, open questions. That consistency means I can scan the output quickly and know exactly where to focus. No wading through paragraphs of hedged opinions.

And the parallel dispatch pattern from the background-research skill is something I keep reusing. Any time I need multiple perspectives on the same question, I dispatch specialist agents simultaneously and synthesize the results. It’s faster than serial research and the different lenses catch different things.

Getting Started
#

If you’ve read the Skills post and the Agents post, adding research and principal agents is the natural next step. Start with one research agent. Pick the lens that matters most for your project (AWS constraints if you’re cloud-heavy, edge cases if you’re building something new, alternatives if you’re at a decision point) and build from there.

The principal agents are worth the investment if you’re working solo or on a small team. They simulate the review you’d get from a senior colleague. They won’t replace a real principal engineer, but they’ll catch the things you’re too close to the problem to see.

All of these are just JSON files and markdown prompts in your .kiro/ directory. They live in the repo, they’re version controlled, and they improve over time as you refine the prompts based on what works and what doesn’t.

The Kiro agent docs cover the setup. The Kiro skills docs cover the workflow side. Between the two, you have everything you need to build your own research and review pipeline.

Keith

The Inspiration#

The Background Research Skill#

The Three Research Agents#

The Principal Agents#

How It All Fits Together#

A Real Example: Shipping the VAR Agent#

What I’ve Learned#

Getting Started#

The Inspiration
#

The Background Research Skill
#

The Three Research Agents
#

The Principal Agents
#

How It All Fits Together
#

A Real Example: Shipping the VAR Agent
#

What I’ve Learned
#

Getting Started
#