Automation

SmarTT Runner CLI

A small program that runs on your machine, asks SmarTTest for work, executes the automated tests in a real browser, and reports the results back.

Why it exists#

The applications you test usually are not reachable from the internet: staging behind a VPN, a local build, an internal tool. The runner puts the browser inside the network where those applications already live.

The runner solves both problems by working outwards. It polls SmarTTest for work: every five seconds when idle, every three while executing a job. Nothing ever connects into it, so it needs no open port, no firewall rule, and no public address. If the machine can reach the internet and your application, it can run your tests.

Note

This is the same model GitHub self-hosted runners use, for the same reason.

The package#

@smarttest/runner on npm·Apache-2.0·command smarttest-runner

Node 18 or newer. Windows, macOS, and Linux.

Installing#

1
Create a runner in SmarTTest
Execution Settings → Runners → New runner. Name it after the machine. You get a token, shown exactly once. Copy it now, because only a hash is stored.
2
Save the token on the machine
terminal
```
npx @smarttest/runner login --token <your-token>
```
This verifies the token against the server and writes it to ~/.smarttest/runner.json. You only do this once.
3
Install the browser and the encoder
terminal
```
npx @smarttest/runner install
```
Downloads Chromium and ffmpeg. This also runs automatically on install, so you usually only need it to repair a broken setup.
4
Start it
terminal
```
npx @smarttest/runner start
```
It shows as Online in the app and begins picking up work.

For a machine that runs permanently, install it globally instead of through npx:

terminal

npm install -g @smarttest/runner
smarttest-runner start

Once started it reports the machine it is on, and waits:

terminal

SmarTTest runner v0.1.2
  machine: build-01 (win32)
  server:  https://www.smarttest.cloud
Waiting for work. Press Ctrl+C to stop.

Commands#

login

smarttest-runner login --token <token> [--url <url>]

Verifies the token and saves it. Run it once per machine, or again to point the runner at a different SmarTTest instance.

Note

Verifying deliberately does not claim work. An earlier version used the same call the runner uses to pick up jobs, which meant logging in could quietly steal a job and leave it running with nothing executing it.

start

smarttest-runner start [--token <token>] [--url <url>]

The main loop. Asks for work, executes it, reports back, repeats. Leave it running in a terminal, a service, or a container.

Ctrl+C finishes the job in flight, then stops.
Ctrl+C twice stops immediately.
If the server is unreachable it says so once and keeps retrying, instead of filling the terminal with the same error.

stop

smarttest-runner stop [--force]

Stops a runner started from another terminal, and closes any browser windows left behind by an execution that crashed. Without --force it lets the current job finish first.

Tip

It only kills browsers Playwright launched for testing. Your own Chrome, and any other process, is left alone.

status

smarttest-runner status

Shows the saved configuration: which server it points at and where the config file lives. The token is not printed.

install

smarttest-runner install

Downloads Chromium and ffmpeg. Run it if executions start failing with a missing-binary error.

Configuration#

Settings are resolved in order: command-line flag, then environment variable, then the saved config file.

Parameter	Type	Description
--token / SMARTT_RUNNER_TOKENreq	string	The runner token, created in Execution Settings → Runners.
--url / SMARTT_API_BASE_URL	urldefault https://www.smarttest.cloud	Only needed for a self-hosted instance.
--force	flag	With stop, kill immediately instead of finishing the current job.

The saved config lives at ~/.smarttest/runner.json.

For CI or a container

Skip login entirely and pass SMARTT_RUNNER_TOKEN as an environment variable. There is then no state on disk to manage.

What a runner token can reach#

The token is scoped tightly on purpose. A runner can read the tests it has been asked to execute and write back their results. It cannot create or edit tests, browse your library, or reach another project.

That matters because a runner often lives on a shared build machine. If the token leaks, what it grants is the ability to run tests and post results, not access to your test estate.

Heads up

Revoking a runner in the app kills its token immediately and cancels anything queued for it. A runner still polling will start reporting that it cannot reach the server.

Sending it work#

Two ways, both covered elsewhere:

From the app: open a run and press SmarTT Execute. See Test Runs.
From an agent: run_on_runner, then poll get_runner_job. See the tool reference.

Either way only automated tests are executed. Manual ones are skipped and stay yours to fill in.

Troubleshooting#

The runner shows as Offline

It has not been seen for over 60 seconds. Check the process is still running, and that the machine can reach the server. Jobs queued for it are not lost; they wait until it comes back.

Every test fails in a couple of seconds

Almost always a missing binary rather than a broken test, ffmpeg in particular. Run smarttest-runner install. SmarTTest reports this kind of failure as an environment problem, so it is not mistaken for the tests actually failing.

Browsers left open after a crash

smarttest-runner stop closes them.

Could not reach the server

Either the machine lost connectivity, or the token was revoked. Check the runner still exists in Execution Settings → Runners.

Tests interfere with each other

Lower Tests at once to 1. Concurrency only works when tests do not share a login, a fixture, or account state.