> For the complete documentation index, see [llms.txt](https://docs.appmixer.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.appmixer.com/build-connectors-with-a-coding-agent/testing.md). # Testing in Three Layers Green component tests do not mean a working connector. A `MakeApiCall` component can pass its test — exit 0, a plausible payload — while ignoring every query parameter and header it is given. Two separate causes produce exactly that: * The Designer's key-value input arrives as a **JSON string**, and a parser that only accepts an array turns it into `{}` without a word. * **Eventbrite ignores the query string of a GET that carries `Content-Type: application/json`.** The test cannot see either: it asks only whether a response came back, and an ignored filter looks like a filter that matched. To catch it, send a filter value that must fail — and expect a `400`, not a `200`. A test proves what it asks. `appmixer test component` asks whether your code runs. Whether the output matches what you promised the Designer, and whether the connector holds up as a flow, are two more questions with two more commands. ## The three layers | Layer | Command | Answers | Cannot see | | ----------------- | --------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **1 · Component** | `appmixer test component ` | Does my code run against the real API and emit what I expect? | Whether the payload matches the declared schema | | **2 · Connector** | `appmixer connector verify ` | Does what the API returns match what the manifests promise? | Anything the Designer does — dropdowns, variables, bindings | | **3 · Flow** | `appmixer e2e run ` | Does it hold up as a flow on a real instance? | Nothing below it; it needs the connector published ([On the Instance](/build-connectors-with-a-coding-agent/on-the-instance.md)) | Each layer's blind spot is the next layer's subject. Layer 1 is the loop you live in while writing components; layers 2 and 3 are gates once it is green. ## Layer 1 — component tests The agent writes a **test plan** before running anything, and checks authentication first. The plan sets the **order** — a Create test needs an id a List provides, a Get test needs something a Create made — and says **what it will write** to your account. Read that part before you say yes: the plan is the one point where the agent reliably asks before writing. Outside it, while exploring an API, it may create records without asking. If writes need your approval, say so at the start. ```bash appmixer test component src///core/GetEvent -i '{"in":{"eventId":"..."}}' ``` * It takes a **directory**, not a component type. * `-i` takes JSON or a file path, and may be repeated. A file path is resolved **against the current directory, even an absolute one** — `/tmp/in.json` fails with `Cannot find module`. * For triggers, `--test` runs the component's `test()` method and returns a sample at once; `-t` sets the tick period for a real polling run. * **Dropdowns cannot be tested here.** Their cached source path needs the engine's `staticCache`, which the CLI does not provide. Check them in the Designer and in layer 3. {% hint style="warning" %} A component with an `outputType: file` input writes a **real file of real data** into its own directory when tested. Delete it before you stage anything. {% endhint %} ## Layer 2 — connector verify `appmixer connector verify` runs the behaviour files against the real API and compares each payload, leaf by leaf, with the declared output: * **`✗` declared but absent** — the manifest promises a field the API does not send; in the Designer it is a variable that resolves to nothing. * **`~` undeclared, or optional and not seen** — the API sends a field no flow can reach, or an optional field did not appear. * **`✓`** — every declared field turned up. `connector validate` checks the *shape* of your source; `verify` checks its *truthfulness*. ``` ✓ [schema-conformance] GetMyProfile: 8 declared field(s), all present ~ [schema-conformance] ListOrganizations: returned in the live payload but undeclared (unreachable from flows): _type ✗ [schema-conformance] FindEvents: declared but absent from the live payload (dead in the variable picker): published ~ [schema-conformance] GetEvent: optional field(s) never observed in the live payload — confirm they exist or record a sample that has them: published; returned in the live payload but undeclared (unreachable from flows): tx_time_limit, is_locked, … ✓ [enum-roundtrip] CreateEvent.currency: 25 option(s) round-tripped, labels and values match eventbrite: 2 pass, 1 fail, 0 error, 3 warn, 0 skip ``` **Undeclared fields are the useful finding.** A first `verify` on a new connector typically lists a dozen fields the API returns and nobody declared — real data no flow can reach. Declare the ones worth having. **`✗` on a field you know is optional means the schema never said so.** `published` above is optional — draft events lack it — and `GetEvent` rightly reports it as `~`. Verify can only tell an optional field from a required one when the schema it reads declares `required`; where nothing does, it treats every leaf as required. That happens with a static schema without `required`, with an out port declared through `options` alone, and with a component whose port is built from an `outputType` input and which exports no `ITEM_SCHEMA`. The fix is in your connector, not in the finding: declare `required` on the schema, and export `ITEM_SCHEMA` from components with a dynamic out port. Then optional fields come back as `~` and `verify` can gate CI ([Day-to-Day Work](/build-connectors-with-a-coding-agent/daily.md#continuous-integration)). ### Fixtures: `verify.json` Components that need inputs get them from **recipes, never concrete values**, so the file works on any account: ```json { "fixtures": { "organizationId": { "from": "ListOrganizations", "path": "id" }, "eventId": { "from": "FindEvents", "inputs": { "organizationId": "{organizationId}", "status": "" }, "path": "id" } }, "read": [ { "component": "ListOrganizations", "inputs": {} }, { "component": "FindEvents", "inputs": { "organizationId": "{organizationId}", "status": "" } }, { "component": "GetEvent", "inputs": { "eventId": "{eventId}" } } ], "roundtrip": [ { "component": "CreateEvent", "input": "currency", "base": { "organizationId": "{organizationId}", "name": "Appmixer verify currency round-trip", "startDate": "2030-06-01T17:00:00Z", "endDate": "2030-06-01T18:00:00Z", "timezone": "UTC", "listed": false }, "valueField": "currency", "cleanup": { "component": "MakeApiCall", "inputs": { "url": "/events/{id}/", "method": "DELETE" } } } ] } ``` Each fixture runs a component and takes a value from the first item it returns. A fixture needs something to find, so an empty account needs one seeded record. Without the file, verify still checks every List, Find and Get component that needs no inputs. ### The flags ```bash appmixer connector verify # live, read-only appmixer connector verify --write # + enum round-trips appmixer connector verify --record # save output shapes appmixer connector verify --offline # re-check without credentials ``` `--write` runs the **enum round-trip**: for each `select` listed under `roundtrip`, it creates a record per option and reads the stored value back. It needs: * a `select` on a **Create** component whose value the service echoes back (a filter on a List or Find does not qualify); * **a `valueField`** — the check compares values only when the spec names one, and the label only with a `labelField`. With neither, it creates a record per option and compares nothing; * a `cleanup`, which runs for every created record, pass or fail; * an account you are willing to have written to. The result line is fixed text: it says "labels and values match" whatever the spec compared. With a `valueField` alone — Eventbrite returns a currency code and no name — read it as "values match". A `labelField` is what catches an inverted label/value map. `--record` saves each component's output **shape** — values replaced by type placeholders — to `artifacts/samples/.json`, safe to commit. `--offline` then re-checks against those samples with no credentials. Exit code is `0` with no `fail` or `error` findings, `1` otherwise. {% hint style="info" %} **On an OAuth 2.0 connector in CLI 2.6.0**, `verify` cannot find the token `appmixer test auth login` stored and stops with `No stored credentials for ""`. Later versions read it. Until you are on one, pass the token explicitly: ```bash (umask 077; echo '{"accessToken":""}' > /tmp/auth.json) appmixer connector verify --auth /tmp/auth.json rm /tmp/auth.json ``` {% endhint %} ## Layer 3 — the flow A flow on a real instance is the only test in which the Designer calls your connector the way it does for users — including every dropdown. `appmixer e2e import` validates every variable the way the Designer does, so it can fail before any flow runs: ``` [ERROR]: src/acme/eventbrite/artifacts/test-flows/test-flow-events.json: variable fetch error: … {"keyword":"required","params":{"missingProperty":"organizationId"}, "message":"must have required property 'organizationId'"}, {"instancePath":"/status","keyword":"enum", … "message":"must be equal to one of the allowed values"} … "property":"eventId" ``` That is a dependent dropdown that can never load: Get Event's event dropdown called Find Events without the organization it required, and with a status value it did not accept. Component tests and `verify` never run a dropdown, so they stayed green. [Scope and Research](/build-connectors-with-a-coding-agent/research.md#what-to-check-in-the-result) shows how to wire such a dropdown correctly. ### The loop ```bash appmixer e2e import ./src///artifacts/test-flows appmixer e2e list -c : appmixer e2e run --fix appmixer e2e results -c : ``` `import` creates or updates the flows, binds and validates an account, and checks every variable. `run` starts one flow and watches its logs. `results` reads the result stores; if it returns `{}`, trust `run`'s own output. `--fix` is a deterministic repair loop: it rebinds the account on a token error and re-runs on a transient failure. When nothing matches, it stops and hands a **fix brief** to your agent. | Exit | Meaning | | ---- | -------------------------------------- | | `0` | Passed | | `1` | Failed, or timed out | | `2` | A fix brief was emitted — `--fix` only | Exit 2 is not a failure. It is a handover — paste it back to the agent. It looks like this (shortened, ids replaced): ``` Attempt 1: result=error, 1 error(s). [appmixer.utils.test.Assert] Assertion failed: expected true to equal true; expected false to equal false ═══ NEEDS_FIX — fix brief for the calling agent ═══ { "status": "NEEDS_FIX", "reason": "1 error(s), no deterministic rule matched — escalating to AI.", "flowId": "", "flowName": "E2E Eventbrite - events", "connector": "acme:eventbrite", "errors": [ { "componentId": "", "componentType": "appmixer.utils.test.Assert", "message": "Assertion failed: expected true to equal true; expected false to equal false", "timestamp": "…" } ], "assertsFired": [ … ], "assertsSilent": [], "recentLogs": [ … ] } Fix the local flow JSON, then `appmixer e2e import` and `appmixer e2e run --fix` again. RESULT | NEEDS_FIX | E2E Eventbrite - events | … flowId= ``` That particular fault is in the test, not the connector. Assert turns a boolean field into the string `"true"` and compares it loosely with what you declared: a JSON `true` becomes `1`, `"true"` becomes `NaN`, and the two never match. Write expected booleans as strings. The message reads as nonsense because its operands are printed the wrong way round. {% hint style="warning" %} **Exit 1 is not always your connector.** A start failure inside the platform's own test components (`commons.registerSignalReceiver is not a function`) or a provider outage both look like failures. Run it once more, and check the service is up, before you take the connector apart. {% endhint %} ## What "done" looks like ```bash appmixer test component # per component, exit 0 appmixer connector verify # no fail or error you cannot explain appmixer e2e run # exit 0 ``` Layer 1 says your code runs, layer 2 that your manifests are honest, layer 3 that the connector survives being used. Each catches defects the others pass: dropped parameters, undeclared fields, a dropdown that never loads. Next: [Review Before Publishing](/build-connectors-with-a-coding-agent/review.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.appmixer.com/build-connectors-with-a-coding-agent/testing.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.