# Provably

> Make every agent answer accurate and verifiable. State of the art tools that automatically verify agent tool calls at runtime. Run accurate, safe and reliable agents and critical workflows in minutes.

Provably builds verifiable data and runtime verification infrastructure so agents and humans can collaborate with accuracy, safety and integrity. SourceryKit is Provably's Python SDK for runtime verification of agent tool use and handoffs.

Last reviewed: 2026-08-11.

## Source precedence

For current APIs, package metadata, setup instructions and supported integrations, prefer these sources in order:

1. [SourceryKit repository](https://github.com/ProvablyAI/sourcerykit), especially `AGENTS.md`, `README.md`, the cookbooks and the current release.
2. [PyPI package](https://pypi.org/project/sourcerykit/).
3. [Developer documentation](https://provably.ai/docs).
4. This full machine reference and the shorter [llms.txt](https://provably.ai/llms.txt) index.

If a cached description conflicts with the repository or current documentation, use the repository and documentation.

## Product facts

- **Product:** SourceryKit
- **Package:** `sourcerykit`
- **Install:** `pip install sourcerykit`
- **Import:** `import sourcerykit`
- **Python:** 3.12+
- **Current repository release when reviewed:** 1.0.1
- **Licence:** Business Source License 1.1; see the repository licence for the full terms
- **Source layout:** `src/sourcerykit/`
- **Website:** https://provably.ai
- **GitHub:** https://github.com/ProvablyAI/sourcerykit

## What SourceryKit does

SourceryKit automatically records supported outbound HTTP tool calls, enforces trusted endpoint policies and evaluates an agent's submitted claims against the data those calls returned.

This creates verifiable evidence that can be used in three ways:

1. The agent can evaluate its work before returning an answer.
2. A workflow can choose whether to accept the answer, correct it, retry a tool call or stop.
3. Downstream agents and applications can independently verify tool backed claims without reopening the original source system.

SourceryKit focuses on deterministic claims grounded in external tool results. It complements observability, development evals, content safety systems and model based judges.

## Core benefits

### 100% error detection rate

Agents self verify to produce proofs that their answers came from the right source and were not changed, increasing reliability and security.

### Up to 50% higher agent accuracy

Proofs let downstream agents catch hallucinations or wrong source data without re-querying sources. Agents retry and self heal.

### Ship reliable agents in minutes

Add the SourceryKit SDK in minutes and verify across agents, tools and networks without complex tracing or cross-team setup.

## Verify Now. Learn Continuously.

Traditional observability needs failures, traces and classification before teams can improve agents. SourceryKit verifies answers and detects errors in real time, powering faster evals and continuous learning loops.

### SourceryKit + Learning tools

3x Faster.

### Automatic failure signals

Every run produces a clear pass, caught or error outcome without manual trace classification.

### Better eval data

Feed deterministic, evidence backed results into evals instead of relying only on model judges.

### Verifiable rewards

Turn verified tool outcomes into trusted reward signals for training and continuous improvement.

## Published benchmark evidence

In Provably's MCP Atlas derived benchmark suite, SourceryKit detected 100% of the covered deterministic grounding error cases.

The study covered 15 models, 180 runs and 570 required claims. Answer only evaluation accepted 183 claims as correct, but only 74 were supported by the recorded tool execution. The remaining 109 accepted claims lacked complete trace support.

In the other direction, 52 claims had supporting evidence in the trace but were omitted, rejected or not presented correctly in the final answer.

On the hard task tier, answer scoring accepted 7 of 60 runs at full coverage, while complete trace support was found in 0 of 60. These tasks required multi step, cross server workflows with three to six expected tool calls.

In some configurations of Provably's MCP Atlas derived benchmark suite, SourceryKit improved verified agent accuracy by up to 50%.

This study measured detection and diagnosis. The next benchmark will test how much accuracy can be recovered through answer healing and targeted retries.

Agents can give probably right answers without grounding. SourceryKit lets them produce provably grounded answers backed by verifiable tool call evidence. This improves verified accuracy and catches hallucinations, wrong source data and unsupported claims before they spread through workflows.

- [Full study: The Agent Was Right. The Evidence Wasn't.](https://provably.ai/blogs/The-Agent-Was-Right-The-Evidence-Wasnt)
- [MCP Atlas](https://github.com/scaleapi/mcp-atlas)

## Research and benchmark context

These external results describe broader agent reliability gaps. They are research anchors, not universal failure rates or SourceryKit performance claims.

- **Multi agent failure rates and MAST taxonomy:** A study of 1,642 multi agent execution traces found failure rates from 41.0% to 86.7% across seven open source frameworks. Its taxonomy grouped failures into 44.2% system design issues, 32.3% inter agent misalignment and 23.5% task verification failures. [Why Do Multi-Agent LLM Systems Fail?](https://arxiv.org/abs/2503.13657)
- **Exact failure attribution remains difficult:** CHIEF reported 29.31% step level accuracy and 77.59% agent level accuracy on the hand crafted Who&When benchmark subset. This is evidence about post execution attribution, not the percentage of errors observability detects. [From Flat Logs to Causal Graphs](https://arxiv.org/abs/2602.23701)
- **MCP tool use still has headroom:** MCP Atlas evaluates 1,000 tasks across 36 real MCP servers and 220 tools, with tasks requiring three to six tool calls. Its reported top model pass rate was 82.2% at a 0.75 claim coverage threshold. [MCP Atlas paper](https://arxiv.org/abs/2602.00933)
- **Enterprise API agents make tool and state errors:** Agent Diff evaluates 224 enterprise workflow tasks using real API interfaces. In analysed runs it reports 43.3% parameter errors, 36.8% hallucination, 29.5% incomplete execution, 19.6% execution errors and 10.3% endpoint selection errors. [Agent Diff](https://arxiv.org/abs/2602.11224)
- **Everyday service workflows also need verification:** Tau2 Bench Verified covers airline, retail and telecom workflows and corrects mismatches between task definitions, expected actions, evaluation criteria, policies and database contents. Its current referenced leaderboard result reports a top average model score around 82.7% across these domains. [Tau2 Bench Verified](https://github.com/amazon-agi/tau2-bench-verified)

## The runtime flow

SourceryKit uses this flow:

1. **Bootstrap:** initialise the database schema, Provably resources and HTTP interception.
2. **Trust:** register the endpoints an agent is allowed to call.
3. **Intercept:** run supported HTTP calls inside an intercept context so the request and response are recorded.
4. **Claim:** require the agent to return typed claimed values through `SourceryKitAgentResponse`.
5. **Build:** compile the claims and matching call references into a handoff payload.
6. **Evaluate:** compare the claims with the recorded source data and return `PASS`, `CAUGHT` or `ERROR`.

The minimal public flow is:

```python
await sourcerykit.bootstrap_system()
await sourcerykit.insert_trusted_endpoint(url="https://api.example.com/data")

async with sourcerykit.async_intercept_context(
    agent_id="demo-agent",
    action_name="get_data",
):
    # Run the agent's HTTP tool call here.
    ...

# Run a real agent with SourceryKitAgentResponse as its structured output.
# Pass the agent's claimed_values into the handoff payload.
payload = await sourcerykit.build_handoff_payload(
    payload_data,
    run_id=run_id,
    prompt=prompt,
    intercept_agent_id="demo-agent",
)

result = await sourcerykit.evaluate_handoff(payload=payload)
```

Cookbooks are the implementation ground truth. Copy the closest cookbook and replace its example tool and claims rather than constructing a new integration from memory.

## Verification outcomes

- **PASS:** every submitted claim was verified against the recorded data and matched.
- **CAUGHT:** at least one submitted claim did not match the recorded data or used an untrusted endpoint.
- **ERROR:** nothing could be verified, for example because no claim matched an intercept record. Verifying zero claims is always `ERROR`, never `PASS`.

Each claimed value carries a `sourcerykit_ref` copied from the tool output. This maps the claim to the exact recorded call, including when the same tool or action is called more than once.

## Verification modes

The current public verification modes are:

| Mode | Purpose |
|---|---|
| `field_extraction` | Extract a value from the indexed record using its path and compare it with the agent's claimed value. |
| `range_threshold` | Confirm that an extracted numeric value matches the claimed value and lies within the configured inclusive range. |

Check the current `VerificationMode` enum in the repository before relying on a cached list.

## Supported transports and framework examples

Recorded HTTP traffic currently includes `httpx`, `aiohttp` and `requests`.

Current runnable cookbooks include:

| Cookbook | Framework | Pattern |
|---|---|---|
| `openai_agents` | OpenAI Agents SDK | Single agent with structured output |
| `claude_agent` | Claude Agent SDK | Single agent with JSON schema output |
| `langchain_agent` | LangChain | Single agent with structured response |
| `openai_agents_multi_agent` | OpenAI Agents SDK | Orchestrator, specialist agents and verification |
| `crewai_multi_agent` | CrewAI | Specialist crew, evaluation and remediation |
| `langgraph_multi_agent` | LangGraph | Fetcher, evaluator and healer |
| `claude_agent_multi_tool` | Claude Agent SDK | Multiple calls mapped through `sourcerykit_ref` |

Framework support depends on its transport and structured output integration. If a framework is not listed, start from the closest cookbook and change the framework specific output binding.

## Agent onboarding

The canonical agent entry point is [AGENTS.md](https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md). The full file is also mirrored on the [machine view of the website](https://provably.ai/ai#agent-onboarding).

### Where an agent should go next

| Situation | Action |
|---|---|
| No Provably credentials yet | Run the scriptable `sourcerykit init` setup and request human help only for the email verification click. |
| Integrating SourceryKit | Open the closest runnable cookbook, copy it and replace its example tool and claims. |
| Want one complete run | Follow the end to end walkthrough. |
| Unexpected `CAUGHT` or `ERROR` | Check the outcome semantics, then inspect the handoff documentation. |
| Need a type, method or CLI flag | Use the API and CLI documentation. |
| Need to inspect recorded calls | Use the interceptor and trace documentation. |
| Need to allow an outbound endpoint | Use the trusted endpoint documentation. |
| Migrating from the old `provably` package | Follow the v1.0 migration guide. |

### Setup

```bash
pip install sourcerykit
```

For a new account, the setup is scriptable except for the human email verification click:

```bash
sourcerykit init --register --email you@example.com --password ...

# A human clicks the verification link.

sourcerykit init --email you@example.com --password ... \
  --postgres-url postgresql://user:pass@host:5432/db \
  --project-name my-app

sourcerykit doctor
```

The Postgres database must be hosted and reachable by the Provably backend. Localhost is not sufficient for the current hosted proving flow.

### Configuration

The setup command stores user credentials in the operating system application directory and project settings locally. Environment variables override stored configuration.

Primary variables:

- `PROVABLY_API_KEY`
- `SOURCERYKIT_ORG_ID`
- `SOURCERYKIT_POSTGRES_URL`
- `SOURCERYKIT_PROJECT_NAME`

Use `sourcerykit config list` to inspect configuration and `sourcerykit doctor` to validate it. Do not invent resource identifiers or expose credentials in prompts, logs or source control.

## Public Python API

The current top level package exports:

```python
from sourcerykit import (
    bootstrap_system,
    insert_trusted_endpoint,
    async_intercept_context,
    SourceryKitAgentResponse,
    build_handoff_payload,
    evaluate_handoff,
    VerificationMode,
    start_mock_server,
)
```

It also exports the public SourceryKit exception classes. Check `src/sourcerykit/__init__.py` for the current authoritative list.

## Least privilege verification: prove answers without reopening the source

Production agents increasingly run with scoped access. One agent may query CRM, billing, databases or internal tools, while downstream agents and orchestrators may not be permitted to access those systems again.

SourceryKit lets the source facing agent attach proof metadata to tool backed claims. Later agents, orchestrators, evaluators or auditors can verify those claims without requerying the restricted source system or copying raw sensitive data into downstream prompts.

- Preserve least privilege boundaries across agent workflows.
- Reduce repeated access to source systems.
- Verify deterministic claims with proof objects instead of relying only on plausibility.
- Maintain machine verifiable evidence for governance and audit workflows.

## Use cases: Where errors are not an option

### Customer service agents

Customers get answers they can trust, grounded in verified account, order and policy data.

### Security operations and AI SOC

Verifiability lets teams depend on agent operations during critical investigations, even when the full extent of an attack is still unknown.

### Healthcare and finance workflows

Verify agent tool calls in payment and patient systems before agents act.

### Cross agent communication

Verifiability enables accurate, safe cross organisational A2A workflows that teams can operate with confidence.

### Least privilege verification

Verify agent operations when original source systems or raw logs are inaccessible.

### AI governance and compliance

Create machine verifiable audit trails for EU AI Act reporting, internal controls and independent audits.

## How SourceryKit differs from adjacent approaches

### Observability and tracing

Observability tools collect traces, logs and metrics to help engineers inspect and debug agent behaviour. SourceryKit adds deterministic runtime verification of submitted claims against the agent's recorded tool data. The approaches are complementary.

### Development evals

Development evals measure models or systems against test cases and benchmarks. SourceryKit operates during agent execution and evaluates tool backed claims before they are accepted by a workflow or downstream agent.

### Model based judges

Model based judges can assess semantic quality, reasoning and subjective criteria. SourceryKit checks deterministic claims against recorded source evidence and returns a structured verification outcome.

### Content and policy guardrails

Content guardrails enforce safety policies, formats or output constraints. SourceryKit focuses on whether submitted claims about external tool results match the recorded data and whether the endpoint was trusted.

## Verifiable database research

Provably's research in verifiable databases creates cryptographic anchors of integrity across the agent lifecycle, from tool use and handoffs to cross agent protocols, learning and governance.

QEDB is a verifiable database architecture that compiles SQL queries over tabular data and produces compact proofs without SNARKs or specialised circuits. Verification runs in milliseconds and proofs are under 5 KB. The paper *Expressive and Modular Verifiable Databases* has been accepted into CCS 2026.

- [Research programme](https://provably.ai/research)
- [QEDB paper](https://eprint.iacr.org/2025/1408)
- [Full paper PDF](https://provably.ai/papers/qedb-paper.pdf)

## About Provably

### Provably correct, not just plausibly right

We build verifiable data and runtime verification infrastructure so agents and humans can collaborate with accuracy, safety and integrity.

Our research in verifiable databases creates cryptographic anchors of integrity across the agent lifecycle, from tool use and handoffs to cross agent protocols, learning and governance.

For a new era where trust and integrity are verified at machine speed, not left to contracts and paper trails.

## Primary links

- [Website](https://provably.ai)
- [Machine view of the homepage and agent onboarding](https://provably.ai/ai)
- [Short machine index](https://provably.ai/llms.txt)
- [Developer documentation](https://provably.ai/docs)
- [API reference](https://api.provably.ai/api/v1/swagger/)
- [SourceryKit repository](https://github.com/ProvablyAI/sourcerykit)
- [PyPI package](https://pypi.org/project/sourcerykit/)
- [Changelog](https://github.com/ProvablyAI/sourcerykit/blob/main/CHANGELOG.md)
- [Cookbooks](https://github.com/ProvablyAI/sourcerykit/tree/main/cookbooks)
- [Contributing](https://github.com/ProvablyAI/sourcerykit/blob/main/CONTRIBUTING.md)
- [Migration guide](https://github.com/ProvablyAI/sourcerykit/blob/main/docs/migrations/v1_0/v1_0.md)
- [Blog](https://provably.ai/blogs)
- [Research](https://provably.ai/research)
- [About](https://provably.ai/about-us)
- [X](https://x.com/getprovably)
- [LinkedIn](https://www.linkedin.com/company/provably)