SmarTTest

Automation

MCP tool reference

All forty tools, grouped by what they touch. Setup instructions are in SmarTT MCP.

Remote or local#

Most tools work on both the hosted connector and the local server. Four exist only locally, because they drive a real browser and the server does not have one.

Note

On the remote connector those four are not listed at all. The remote path to execution is run_on_runner, which hands the work to a machine that has a browser. See SmarTT Runner CLI.

Projects & folders#

Where an agent starts. Almost every other tool needs a projectId, and this is where it comes from.

list_projects

remote + local
Lists the projects your organization can reach, with the ids every other tool needs.

Parameters

ParameterTypeDescription
includeInactivebooleandefault falseInclude deactivated projects. They are read-only, so writes to them will be refused.

Use it when

  • First call of almost any session, because nothing else works without a project id.
  • Confirming which project a request refers to when the user says "the checkout tests" and there are three projects.

list_project_folders

remote + local
Every active folder in a project, each with its full path so the tree can be reconstructed.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Finding the folder id needed to create a test.
  • Understanding how a team organizes its library before adding to it.
  • Deciding where a new test belongs, rather than dropping it in the first folder found.
On a large library this returns a lot of text at once, and it is not paginated. Ask for the specific area you need rather than the whole tree when the project is big.

create_folder

remote + local
Creates a root folder, or a subfolder under an existing one.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
namereqstringMaximum 30 characters.
parentIdstringOmit for a root folder. Otherwise the folder this one nests under.

Use it when

  • Setting up structure for a new feature area before writing its tests.
  • Splitting a folder that has grown past the point of being navigable.

Tags#

Cross-cutting labels. Note that tests are tagged by name through create_test and update_test, so these two are mostly for discovery and cleanup.

list_tags

remote + local
The tags in a project, each with a count of how many active tests carry it.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Checking what conventions already exist before inventing a new tag.
  • Finding the right tag name to scope a run to.
  • Spotting tags with a count of one, usually a typo of a real tag.

create_tag

remote + local
Creates a tag in a writable project.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
namereqstringShort and lowercase works best.
descriptionstringWhat qualifies a test for this tag.

Use it when

  • Introducing a genuinely new category, after checking list_tags for something equivalent.
A name only has to be free within the target project, so the same tag can exist in several projects. Creating one that the project already has is refused rather than silently reused.

Tests#

The core of the API. Reading, writing, archiving, and checking the quality of automation.

list_tests

remote + local
Lightweight metadata for the tests in a project, enough to choose one, without pulling every step.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
folderIdstringRestrict to one folder.
recursivebooleandefault trueWhether subfolders are included when folderId is set.
includeArchivedbooleandefault falseInclude archived tests. Needed to find something to restore.

Use it when

  • Finding a test by title before reading it in full.
  • Surveying what is automated and what is not across an area.
  • Locating an archived test so it can be restored.

get_test

remote + local
The complete test: steps, expected result, automation plan, tags, and attached reusable preconditions.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
includeArchivedbooleandefault trueWhether an archived test can be returned.

Use it when

  • Reading the written steps before writing automation for them.
  • Inspecting an existing automation plan before changing it.
  • Checking which reusable preconditions are attached, and in what order.

create_test

remote + local
Creates a test inside an existing folder. Written steps and automation can both be supplied at once.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
folderIdreqstringMust already exist.
titlereqstringWhat the test verifies, in one line.
priorityLOW | MEDIUM | HIGH | CRITICALDefaults if omitted.
descriptionstringContext the reader needs.
preconditionsstringWritten state required before step 1.
stepsreqstep[]At least one. Each may carry automation.
expectedResultstringWhat must be true at the end.
expectedResultAutomationaction[]Commands that verify the expected result, normally the assertions.
expectedResultAttachmentsstring[]URLs of supporting images.
referencesstringTickets, requirement ids.
automatedNo | Yes | Impossible | Discardeddefault "No"Only these four values. Anything else is rejected.
automationStatusCompleted | In Progress | Needs Attention | BrokenOnly allowed when automated is Yes.
automatedRejectReasonstringWhy the test was not automated. Only for Impossible and Discarded.
tagNamesstring[]By name, not id. Tags must already exist.

Use it when

  • Turning a requirement or a bug report into a written test.
  • Creating a test and its automation together, in one call.
  • Bulk-creating a set of cases for a new feature.
The four automated values are exact. "yes", "true", or "automated" are all rejected rather than coerced.

update_test

remote + local
Updates a test in place. Every field is optional: only what you pass changes.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
folderIdstringMove the test to a different folder.
titlestring
prioritystring
descriptionstring | nullPass null to clear.
preconditionsstring | nullPass null to clear.
stepsstep[]Replaces the whole list. It is not a merge.
expectedResultstring | null
expectedResultAutomationaction[] | nullReplaces the verification actions entirely.
referencesstring | null
automatedNo | Yes | Impossible | Discarded
automationStatusstring | nullOnly when automated is Yes.
automatedRejectReasonstring | nullOnly for Impossible or Discarded.
addTagNamesstring[]Tags to add, by name.
removeTagNamesstring[]Tags to remove, by name.

Use it when

  • Attaching automation to a test that was written manually.
  • Marking a test Broken after the UI changed under it.
  • Adding tags in bulk when a new suite is defined.
  • Moving a test to the folder it should have been in.
steps replaces the entire list. To change one step, read the test first, modify the array, and send all of it back.
The whole automation vocabulary: every command with the exact input keys it accepts, their types, and which are required.

Parameters

ParameterTypeDescription
commandstringReturn only this command. Omit it to get all twenty-six.

Use it when

  • Before writing any automation plan. It is the difference between knowing a command exists and knowing what to put in it.
  • After a rejected save, to see what the command actually accepts instead of guessing again.
Keys outside this list are rejected when the test is saved, and the error names the valid ones. Nothing is silently ignored, so there is no reason to guess.

analyze_test_quality

remote + local
Runs the deterministic Quality Engine on a test's automation. Returns a 0-100 scorecard with findings and concrete suggestions. No AI, no cost.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.

Use it when

  • Immediately after writing automation. This is the single most valuable tool for an agent, because it turns "looks right" into a checkable result.
  • Auditing an existing suite to find the tests worth fixing first.
  • Confirming a fix actually improved things rather than moving the problem.
It checks assertions, locator brittleness, hard waits, hardcoded credentials, unused runtime variables, and steps with no automation behind them. The findings name the step, so they can be acted on directly.

archive_test

remote + local
Soft-deletes a test. It leaves the tree but keeps its history and every past run result.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.

Use it when

  • Retiring a test for a feature that no longer exists.
  • Removing a duplicate, once you have confirmed which one to keep.

restore_test

remote + local
Brings an archived test back, and records the restoration in its history.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.

Use it when

  • Undoing an archive.
  • Recovering a test after a folder was deleted. Find it first with list_tests and includeArchived: true.

Reusable actions#

Shared blocks of steps, and how tests attach to them. Editing a block changes every test that uses it. See Reusable Actions for the model.

list_reusable_actions

remote + local
The blocks defined for a project, each with the tests that currently use it.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Checking whether a login block already exists before writing another one.
  • Seeing the blast radius of a change before making it.
  • Finding unused blocks worth deleting.

get_reusable_action

remote + local
The full detail of one block, including its steps and every test linked to it.

Parameters

ParameterTypeDescription
reusableActionIdreqstringFrom list_reusable_actions.

Use it when

  • Reading a block's steps before modifying them.
  • Understanding what a precondition actually does when debugging a failing test.
Creates a shared block of steps in a project.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
namereqstringDescribe the outcome: "Log in as admin", not "Fill form".
descriptionstringWhat state the system is in once it finishes.
usableAsPRECONDITION | STEP | BOTHdefault "PRECONDITION"Where the block may be used.
stepsreqstep[]At least one.

Use it when

  • Extracting a sequence that has been copied into several tests.
  • Defining login once, at the start of automating a project.
A block cannot contain smartt_run_reusable_action. Blocks do not nest. If two share steps, inline them.
Changes a block. Every test that uses it is affected immediately.

Parameters

ParameterTypeDescription
reusableActionIdreqstring
namestring
descriptionstring | null
usableAsPRECONDITION | STEP | BOTH
stepsstep[]Replaces the whole list.

Use it when

  • Fixing every test at once after a login form changed.
  • Making a block more robust, replacing a CSS selector with a role-and-name locator.
Deletes a block.

Parameters

ParameterTypeDescription
reusableActionIdreqstring

Use it when

  • Removing a block nothing uses any more.
Refused while tests are still attached. Detach them first.
Attaches a block to a test as a precondition. It will run before the test's own first step.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
reusableActionIdreqstringMust be in the same project.

Use it when

  • Giving a new test the standard login setup.
  • Replacing steps that were inlined into a test with the shared block, usually right after analyze_test_quality flags the duplication.
Removes one precondition link from a test. The block itself is untouched.

Parameters

ParameterTypeDescription
linkIdreqstringThe link id, from get_test, not the reusable action id.

Use it when

  • A test that no longer needs that setup.
  • Clearing links before deleting a block.
Sets the order in which a test's preconditions run. They execute top to bottom.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
orderedLinkIdsreqstring[]Every attached link id, exactly once, in the order you want.

Use it when

  • Fixing setup that runs in the wrong order: selecting an account before logging in, for instance.
  • Putting a newly attached precondition in the right place instead of last.
Every currently attached link must appear exactly once. A partial list is rejected.

Environments & variables#

What an execution needs. These tools tell an agent which keys exist and where they are defined, never what a secret contains.

The browser and resolution configured for a project.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Reporting how tests will run, without guessing.
  • Deciding whether to override the browser for one execution.
execute_test and execute_run already apply these. You only need this tool to show or reason about them.

list_environments

remote + local
The execution environments available to your organization.

Parameters

Takes no parameters.

Use it when

  • Getting the environmentId that every execution tool requires.
  • Confirming which target the user means before running anything against it.

get_org_variables

remote + local
Organization-level variable keys. Secret values are never returned.

Parameters

Takes no parameters.

Use it when

  • Discovering what is available to reference as {{orgVar.KEY}} before writing automation.
  • Checking a base URL is defined before writing a test that depends on it.

get_project_variables

remote + local
Project-level variable keys. Secret values are never returned.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Finding what to reference as {{projectVar.KEY}}.
  • Understanding which values are project-specific rather than shared.

get_test_variables

remote + local
Variable keys attached to a single test. Secret values are never returned.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.

Use it when

  • Checking what a test defines for itself as {{testVar.KEY}}.
  • Diagnosing a test that fails only in one environment.
Resolves the actual values a test would receive in a given environment.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
environmentIdreqstringFrom list_environments.

Use it when

  • Confirming a test will get the values it expects before running it.
  • Working out why the same test passes against staging and fails against production. Usually it is a variable defined for one environment and not the other.

Runs#

Creating runs, recording results, and closing them. A run stays open until complete_run is called.

list_runs

remote + local
Runs for a project, newest first, with summary counts and status.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
limitnumberdefault 50Between 1 and 200.

Use it when

  • Finding the run a user is referring to.
  • Checking whether an open run already exists before creating another.
  • Summarising recent testing activity.

get_run

remote + local
One run with every test result in it: status, comment, duration, and recording.

Parameters

ParameterTypeDescription
runIdreqstringFrom list_runs.

Use it when

  • Reading what failed and why, in order to act on it.
  • Watching results land while a runner executes.
  • Producing a report from a completed run.

create_run

remote + local
Creates a run, optionally scoped to folders, tags, or specific tests. A result slot is created for every test in scope.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.
namereqstringWhat this run is verifying.
includeTimestampbooleandefault falseAppend a date and time to the name, the same option the interface offers.
commentstringWhat this run is for. Worth filling in.
folderIdsstring[]Include everything in these folders.
tagNamesstring[]Include every test carrying these tags.
testCaseIdsstring[]Include these specific tests.

Use it when

  • Building a smoke run from a tag before a release.
  • Creating a targeted run for the area a change touched.
  • Setting up a run to hand straight to a runner.
With no scope at all, every test in the project is included. The interface asks you to pick something instead, so an agent creating an unscoped run is making a choice a person would have been stopped from making by accident.
Sets the result of one test inside a run. A comment is always required.

Parameters

ParameterTypeDescription
runIdreqstring
testCaseIdreqstringFrom list_tests or get_test.
statusreqPENDING | IN_PROGRESS | PASSED | FAILED | SKIPPEDCase-insensitive.
commentreqstringCannot be empty.
videoUrlurlRecording of the execution. Left by the runner.
executionIdstringWhich runner execution produced this.
durationMsnumberHow long it took.

Use it when

  • Recording the outcome of a test executed outside SmarTTest.
  • Marking a test skipped, with the reason it was out of scope.
  • Correcting a result after investigating a failure.
The comment is not optional by accident. A failure with no note is a failure nobody can act on later.

complete_run

remote + local
Closes a run. SmarTTest reads through the results and appends a SmarTT Summary of what happened.

Parameters

ParameterTypeDescription
runIdreqstring
commentreqstringThe closing note. Cannot be empty.

Use it when

  • Finishing a run once every test has a result.
  • Closing a run so it starts counting toward the Health Dashboard.
Only completed runs feed pass-rate trends, so leaving runs open quietly loses data.

Execution#

Two paths. Queue the work onto a runner, which works from anywhere, or execute it directly, which only the local server can do.

list_runners

remote + local
The runners registered for a project: the machines that can execute its automated tests.

Parameters

ParameterTypeDescription
projectIdreqstringFrom list_projects.

Use it when

  • Getting the runnerId for run_on_runner.
  • Checking one is online before queueing work, so the user is not left waiting on a stopped machine.
A runner not seen in the last 60 seconds is offline and will not pick up work until it returns.

run_on_runner

remote + local
Queues an existing run for a runner to execute. Returns immediately; the work happens on the runner's machine.

Parameters

ParameterTypeDescription
runIdreqstringThe run to execute.
runnerIdreqstringFrom list_runners.
environmentIdreqstringFrom list_environments.
concurrency1 – 8default 1How many tests at once. Raise it only when the tests are genuinely independent.
onlyall | pending | faileddefault "all"Which results to execute. failed is the usual second pass.
headlessbooleandefault trueWhether the browser window is hidden on the runner machine.

Use it when

  • The only way to execute tests from the remote connector.
  • Running a full regression suite on a dedicated machine instead of your laptop.
  • Re-running just the failures after a fix.
Only automated tests are executed; manual ones are skipped. A run can have one job queued or running at a time. Poll get_runner_job, or read the run to watch results land.

get_runner_job

remote + local
The progress of a job queued with run_on_runner.

Parameters

ParameterTypeDescription
jobIdreqstringReturned by run_on_runner.

Use it when

  • Polling until execution finishes.
  • Distinguishing "the runner never picked this up" from "the tests are failing". The job carries its own error separate from the results.

validate_automation

local server only
Opens a page and checks whether locators resolve, without running a test or creating a run.

Parameters

ParameterTypeDescription
urlreqstringThe page to open. Resolve any variable placeholder yourself first.
locatorsreqobject[]The locators to check, each the same input object an action would use.
headlessbooleandefault trueWhether the browser window is hidden.

Use it when

  • Before asking anyone to execute a run. A locator that matches four elements fails the whole test, and this finds that out in one step instead of one execution.
  • When a test fails and you cannot tell whether the element moved or the page did. The reply includes the final URL and title, which is how a redirect gives itself away.
For anything ambiguous it returns how many elements matched and the text of the first few, so you can tell them apart. It creates no run and records nothing.

execute_test

local server only
Compiles and executes one test's automation plan, in a real browser, on this machine.

Parameters

ParameterTypeDescription
testCaseIdreqstringFrom list_tests or get_test.
environmentIdreqstringFrom list_environments.
headlessbooleandefault trueWhether the window is visible.
stopOnFailurebooleandefault trueStop at the first failing step, rather than carrying on.
timeoutMsnumberdefault 180000Overall limit, from 1s to 15 minutes.
runIdstringLog this result into an existing run. Omit it while iterating, and the execution then leaves no trace anywhere.
recordVideobooleanDefaults to true when runId is set, false otherwise.
videoWidthnumberdefault 1280Viewport and recording width.
videoHeightnumberdefault 720Viewport and recording height.
browserChannelchromium | chrome | msedgeWhich browser to drive. Defaults to the Chromium bundled with Playwright.

Use it when

  • The develop-run-develop loop: write automation, run it, see the failure, fix it, without polluting any run with attempts.
  • Reproducing a failure locally with a visible browser to see what is happening.
  • Recording a real result, by passing runId.
Omitting runId is the important detail. Without it the execution is recorded nowhere, which is exactly what you want while iterating.

get_test_execution

local server only
The current state of an execution started with execute_test.

Parameters

ParameterTypeDescription
executionIdreqstringFrom execute_test.

Use it when

  • Polling a long execution.
  • Reading which step failed, and the error it produced.

execute_run

local server only
Executes every automated test in a run on this machine, logging each result back into it. Returns immediately.

Parameters

ParameterTypeDescription
runIdreqstring
environmentIdreqstring
concurrency1 – 8default 1Each test gets its own browser. Only raise it for independent tests.
onlyall | pending | faileddefault "all"Which results to run.
headlessboolean
recordVideoboolean
timeoutMsnumberPer test.
videoWidthnumber
videoHeightnumber
browserChannelchromium | chrome | msedge

Use it when

  • Executing a whole suite locally, without setting up a runner.
  • Re-running the failures from an earlier pass on your own machine.
Non-automated tests are skipped and reported as such. Raising concurrency when tests share a login or a fixture makes them fail in ways that look like product bugs.

get_run_execution

local server only
The progress of a run execution started with execute_run, including the state of every test in it.

Parameters

ParameterTypeDescription
runExecutionIdreqstringFrom execute_run.

Use it when

  • Polling until the suite finishes.
  • Reporting progress while a long run is in flight.