Test Format Reference

Compliance tests are written in structured Markdown with YAML frontmatter.

Basic Structure

---
name: test-name
version: 1.0
transport: completions
tags: [core, required]
---

# Test Title

Brief description of what this test validates.

## Setup

Agent and environment configuration.

## Test Cases

### 1. Test Case Name

**Request:**
HTTP request details

**Assertions:**
- Natural language assertion 1
- Natural language assertion 2

**Strict:**
```yaml
# Optional deterministic assertions
status: 200
body:
  key: value
```

Frontmatter

Field	Type	Required	Description
`name`	string	Yes	Unique test identifier
`version`	string	Yes	Test spec version
`transport`	string	Yes	Transport being tested (completions, realtime, a2a)
`tags`	array	No	Categorization tags
`type`	string	No	`single-agent` (default) or `multi-agent`
`timeout`	number	No	Test timeout in seconds

Setup Section

Single Agent

## Setup

Create an agent with the following configuration:
- Name: `echo-agent`
- Instructions: "Repeat back exactly what the user says, prefixed with 'Echo: '"
- Tools: [get_weather, search]

Multi-Agent

## Setup

### Agent: router
- Name: `router`
- Instructions: "Route requests to appropriate specialists"
- Handoffs: [weather-agent, search-agent]

### Agent: weather-agent
- Name: `weather-agent`
- Instructions: "Provide weather information"

Environment Variables

## Setup

### Environment
- `BASE_URL`: SDK server URL (default: http://localhost:8765)
- `API_KEY`: Test API key

Test Cases

Request Formats

HTTP Request:

**Request:**
POST `/chat/completions`
```json
{
  "model": "test-agent",
  "messages": [{"role": "user", "content": "Hello"}]
}
```

With Headers:

**Request:**
POST `/chat/completions`
Headers:
- `Authorization: Bearer test-token`
- `X-Custom-Header: value`

Body:
```json
{...}
```

Streaming Request:

**Request:**
POST `/chat/completions` (streaming)
```json
{"stream": true, ...}
```

Assertions

Natural Language

Human-readable assertions for agentic validation:

**Assertions:**
- Response status is 200
- The assistant responded with a greeting
- Response contains weather information for the requested location
- Tool call was made to `get_weather`
- finish_reason is "stop"

Strict (Deterministic)

Optional YAML block for exact matching:

**Strict:**
```yaml
status: 200
body:
  choices[0].message.role: assistant
  choices[0].message.content: /^Echo:/
  choices[0].finish_reason: stop
headers:
  content-type: application/json
```

Strict Assertion Operators

Operator	Example	Description
`equals` (default)	`status: 200`	Exact match
`regex`	`content: /^Hello/`	Regex match (prefix with `/`)
`contains`	`content: contains("weather")`	Substring match
`exists`	`choices: exists`	Field is present
`not_null`	`choices[0]: not_null`	Field is not null
`length`	`choices: length(1)`	Array/string length
`type`	`content: type(string)`	Type check

JSONPath

Use JSONPath for nested values:

body:
  choices[0].message.content: "Hello"
  usage.total_tokens: type(number)
  choices[*].finish_reason: "stop"  # All choices

Flow (Multi-Agent)

For multi-agent tests, describe the expected flow:

**Flow:**
1. User sends "What's the weather in NYC?" to `router`
2. Router recognizes weather intent
3. Router hands off to `weather-agent`
4. Weather-agent responds with weather information

**Assertions:**
- Handoff event was triggered
- Final response came from weather-agent
- Response mentions temperature or weather

Advanced Features

Conditional Tests

### 2. Tool Call (skip if no tools)

**Condition:** Agent has tools configured

**Request:**
...

Expected Failures

### 3. Error Handling

**Request:**
POST `/chat/completions` with invalid JSON

**Assertions:**
- Response status is 400
- Error message explains the issue

**Expected:** failure

Test Dependencies

---
name: auth-flow
depends_on: [session-create]
---

Data Generation

**Request:**
POST `/chat/completions`
```json
{
  "model": "{{agent_name}}",
  "messages": [{"role": "user", "content": "{{random_greeting}}"}]
}
```

**Variables:**
- `agent_name`: From setup
- `random_greeting`: One of ["Hello", "Hi", "Hey"]

Complete Example

---
name: completions-basic
version: 1.0
transport: completions
tags: [core, required]
---

# Basic Chat Completion

Tests that the `/chat/completions` endpoint handles simple requests correctly.

## Setup

Create an agent with the following configuration:
- Name: `echo-agent`
- Instructions: "Repeat back exactly what the user says, prefixed with 'Echo: '"

## Test Cases

### 1. Simple Message

**Request:**
POST `/chat/completions`
```json
{
  "model": "echo-agent",
  "messages": [{"role": "user", "content": "Hello"}]
}
```

**Assertions:**
- Response status is 200
- Response contains `choices` array with at least one item
- The assistant message starts with "Echo:"
- `finish_reason` is "stop"

**Strict:**
```yaml
status: 200
body:
  choices[0].message.role: assistant
  choices[0].message.content: /^Echo:/
  choices[0].finish_reason: stop
```

### 2. Empty Message Handling

**Request:**
POST `/chat/completions`
```json
{
  "model": "echo-agent",
  "messages": []
}
```

**Assertions:**
- Response status is 400
- Error message explains the issue

**Strict:**
```yaml
status: 400
body:
  error.type: invalid_request_error
```

On this page