CI/CD

Gate a pull request on a Roverly run. Your pipeline uploads a build, starts a run, and turns the verdict into a check.

The integration is an HTTP API — GitHub Actions, GitLab CI, Jenkins, Bitbucket and CircleCI all work the same way, with nothing to install.

Prerequisites

Two things a person sets up once. CI cannot do either, and a pipeline wired before they exist fails every time.

A paired agent host, online. Runs execute on your own hardware, not on Roverly's — a developer machine, a dedicated test box, a device farm, or a mix. Install the desktop app, pair it, and connect the device or simulator. iOS on real hardware needs provisioning in place first.

Whichever host polls first claims the run, so pair as many as you like; two will never pick up the same one. If none is online the run never starts and your step times out.

A pinned production run, for regression. Do a learn run from the desktop app, then mark it Production run on the runs page.

CI cannot do this: pinning needs a learn run, and learn is refused over the API. Without it, POST /v1/runs returns 400.

Get an API key

Dashboard → Settings → API. Give the key both scopes:

Scope	Allows
trigger	Start runs, upload builds
read	Poll a run, read findings

Store it as ROVERLY_API_KEY in your CI secret store. It is a bearer token — anyone holding it can spend your run allowance.

Projects. A key reaches every project in your workspace by default. Uncheck All projects to narrow it to the ones a given pipeline needs — a key for one repo then cannot start runs or read findings anywhere else, so a leak from that repo's CI settings stays contained. A project outside the key's list answers 404, the same as one that does not exist.

Your desktop's key is a different kind and cannot call this API.

Quickstart

curl -fsSL https://roverly.ai/cli/roverly -o roverly && chmod +x roverly

ROVERLY_API_KEY="$ROVERLY_KEY" ./roverly test \
  --project-id "$ROVERLY_PROJECT" \
  --run-type feature \
  --pr-body "$PR_BODY" --pr-title "$PR_TITLE" --pr-number "$PR_NUMBER" \
  --build ./app/build/outputs/apk/release/app-release.apk \
  --summary-file roverly-comment.md

--run-type feature is the CLI's default and is shown here only for clarity. A feature run needs a goal, so a pipeline with no PR context — push-to-main, nightly, a tag build — must pass --run-type regression explicitly or supply --goal. (The API itself has no default: run_type is required in the body.)

Branch and commit are detected for you

On GitHub Actions the CLI fills branch and commit from the runner's own environment, so the run is tagged with nothing to configure. Those tags are what the dashboard shows on a run and what ?branch= filters the release gate by.

On a pull_request event it reads the PR's head SHA out of the event payload rather than using GITHUB_SHA. GITHUB_SHA there is an ephemeral merge commit that GitHub creates to test the merge result — it is not a commit in your repository and it stops resolving once the PR closes, so a run tagged with it would point at nothing.

Tag	Where it comes from
branch	GITHUB_HEAD_REF on a PR (the source branch), else GITHUB_REF_NAME
commit	the PR head SHA on a PR, else GITHUB_SHA
environment	never detected — pass --environment yourself

Pass --branch / --commit explicitly to override the detected value; an explicit flag always wins. On CI other than GitHub Actions nothing is detected, so pass all three.

./roverly test ... --environment staging          # tag the deployment target
./roverly test ... --branch "$MY_BRANCH"          # override the detected branch

Exit code	Meaning
0	Check passes
1	Check fails
2	The tool was invoked wrong

--summary-file writes a pull-request comment you can post as-is.

For a copy-paste version, roverly.ai/cli/snippet.sh is the same gate in about fifteen lines of curl and jq. It skips the build upload and the PR comment.

API reference

Base URL https://api.roverly.ai. Every call needs Authorization: Bearer <key>.

Every request names a project, and every response returns one. Reads take ?project_id=; writes carry it in the body. Responses echo it back, so a pipeline can check the verdict it is about to post belongs to the project it asked about rather than trusting a run id from elsewhere. A run id that belongs to a different project than the one you named is a 404, even when your key reaches both.

Upload a build

Mobile only. Web projects test a URL and install nothing.

POST /v1/artifacts               → { id, project_id, upload_url, upload_content_length, expires_at_iso }
PUT  <upload_url>                → the bytes, direct to storage
POST /v1/artifacts/{id}/complete?project_id=…

Bytes go straight to storage, never through the API. Send exactly the Content-Length returned as upload_content_length — the signature binds it.

Declare size_bytes, sha256, and package_id (Android package or iOS bundle id). The package id is checked against the project's: a mismatch would install one app and launch another.

Builds expire after 7 days. Max 200 MB.

For an iOS simulator build, zip the .app and upload the zip. The archive must contain exactly one .app.

Start a run

POST /v1/runs

Field	Notes
project_id	Required
run_type	regression or feature. learn is refused.
goal	Required for feature — what to exercise. Map your PR description to it.
artifact_id	The build to install
target_url	Web only — a preview deployment to test instead of the project URL
branch, commit_sha, environment	Free-text traceability
pr_number, pr_url, pr_title	Optional PR context
jira_issue_key	Optional — the Jira ticket behind this PR (see below)

Jira context. Connect Jira under Settings → Integrations and Roverly folds the linked ticket's summary and description into a feature run's goal, so the agent tests against the acceptance criteria and not just the PR description.

Roverly then distils the two into a short goal — a sentence or two naming what to exercise, plus a few numbered steps — because that is what the agent reads on every step of the run. Send your PR description as goal and let it do the rest; there is no format to follow, and a ## Test plan section is used when you have one. The distilled goal appears on the run in the dashboard, with the original behind a disclosure, and in the summary Roverly gives you to post. The ticket is found automatically when your project key appears in pr_title, branch or goal (PROJ-412); send jira_issue_key when it appears in none of them, or to override the guess. Enrichment is best-effort — if Jira is unreachable the run still starts with the goal you sent.

GitHub App (optional). Install the Roverly app on your repositories and the verdict is posted as a native check run, by Roverly rather than by your pipeline's token — which is also what lets a fork's pull request get a result. Your pipeline is unchanged: it still builds, uploads and triggers, and the summary endpoint still works. Install it from Settings → Integrations.

The App reports; it does not trigger — so your secret stays. ROVERLY_API_KEY and --project-id are still required exactly as above: the key authenticates the trigger, and the project id names which Roverly project this repo's PRs run against. The App knows neither — by the time it posts, the run already exists, and it reads only the repo (from pr_url) and finds the right installation. What the App can replace is the poll-and-comment step: with it installed you no longer need the CLI to post the summary comment, though keeping both is fine.

Two fields decide whether a check appears. A check run attaches to a commit, so commit_sha must be a real SHA — a branch name is ignored and no check is posted. The repository is read from pr_url, so that has to be the https://github.com/owner/repo/pull/N form. Send both on the trigger; miss either and the run still executes and summary still answers, you just get no check.

Failed runs get a check too, carrying the reason the run ended. Re-running the same run updates the existing check rather than adding a second one.

GitHub's "Re-run" button works. Clicking it re-tests the pull request using the build already on file — Roverly rebuilds nothing. If that build has expired (they are kept 7 days) the check says so instead of failing silently; push the commit again to refresh it.

Preview deployments. Previews live off your domain (*.vercel.app, *.netlify.app), so add those hosts to the project's preview-domain allowlist first. Otherwise the run is refused — a stray URL must not be able to aim a run at someone else's site.

Read the verdict

GET /v1/runs/{id}?project_id=…          → { project_id, status, ... }
GET /v1/runs/{id}/summary?project_id=…  → { project_id, status, reasons, blocking, markdown, superseded, steps, goal }
GET /v1/runs/{id}/findings?project_id=… → { project_id, run_id, findings: [...] }

Poll until status is one of succeeded, failed, cancelled.

processing is not finished. Findings are produced during processing. A client that stops when the status is no longer running reads the verdict before the findings exist and reports a pass on a broken change. Check membership of the three terminal statuses — never "not running".

Then read summary. Branch on status (pass / fail) and post markdown as a comment. It is always HTTP 200; an HTTP error means auth or infrastructure, never a verdict.

What Roverly did, not just whether it passed. On a run that succeeded, summary returns steps[] — the ordered actions the agent took, each naming the screen it started on, what it did, and where it landed. Taps the agent made by coordinate are resolved back to the element under them, falling back to tapped at (x, y) when nothing resolves. The same list is folded into markdown, so posting that comment gives a reviewer the evidence the feature was exercised rather than asking them to take "passed" on trust.

A failed run returns no steps, deliberately: it verified nothing, and a list of taps ending in a crash is noise on a pull request. It gets a one-or-two-sentence reason instead.

Progress while it runs — automatic with the GitHub App. If Roverly's App is installed, the check starts spinning ("Roverly is testing this pull request") the moment an agent picks up the run, then updates in place to the verdict when it finishes. No polling needed for that.

Without the App, polling before the run finishes is not wasted — the reasons carry the same plain-sentence waypoint you can post as a status update:

the agent has claimed the run	"A Roverly desktop agent has picked up this CI run"
the agent finished, analysis is running	"The Roverly desktop agent has completed this CI run; kicking off backend processing"

Neither is a verdict. status is "fail" until the run is terminal, which is why the same response also says "Not a verdict yet — poll until the run reaches succeeded, failed or cancelled."

Stop polling before 60 minutes. An unclaimed run is cleaned up at 60. Past that you are polling an abandoned run and reporting it as your app's verdict. 45 minutes is a sensible ceiling.

Concurrent pushes

Push six times and you get the in-flight run plus one run for the sixth commit. Not six runs.

A run an agent has started always finishes. A queued run nobody has picked up is replaced by the newer one. Replaced runs do not count against your plan.

When your job is the replaced one, summary returns superseded: true.

Branch on status, not on superseded alone. There are two cases behind that one flag:

	superseded	status	Do
Replaced before any agent picked it up	true	"pass"	Exit 0 — nothing ran, and a red check here is noise
Replaced after the run had already started	true	the real verdict	Honour status. The run finished and its findings are real, just about an older commit

A client that exits 0 on superseded alone will silently swallow a genuine failure in the second case. The reasons say the result is stale.

Failure modes

Cause	What you see
No paired desktop online	Run stays queued; your step times out
Another run in progress	Queued behind it; same timeout
No production run pinned	400 at trigger, naming the fix
Over your plan's allowance	402 at trigger
Build missing, expired or wrong package	400 at trigger; no run starts
Run errored	summary fails, naming the error
Replaced by a newer commit	superseded: true — exit 0 only if status is "pass"; see above

Anything that is not a pass renders as a failed check. The summary text says which.

Fork pull requests are not special-cased. The API cannot tell a fork PR from any other, so a fork PR starts a real run and spends a real slot. On a public repo that is usually wrong — both for cost and because a contributor's token generally cannot post the comment back. Gate it in your pipeline. On GitHub Actions:

if: github.event.pull_request.head.repo.full_name == github.repository

Plans and limits

No plan gates this API and there is no separate CI quota. A CI-triggered run draws on the same monthly pool as one started from the desktop.

Plan	Runs / month	Past the pool
Starter	8	402 until the month rolls over
Pro	40	Metered as overage; runs continue
Enterprise	Per contract	Per contract

On Starter that means eight gated pull requests a month; the ninth POST /v1/runs returns 402 rather than queueing.

On Pro, overage is funded by prepaid credits. A run needs $2.00 available to start once the pool is used up — add credits under Settings → AI.

Wiring this up with an AI assistant

Everything on this page is also published as one plain-text file: llms.txt. Point a coding assistant at it and it can write a working pipeline in one fetch, instead of crawling the site.

Two things are worth telling it explicitly, because they are what a correct-looking config gets wrong: the paired-desktop and pinned-production-run prerequisites are set up by a person, not by CI, and processing is not a terminal status.