# SmarTTest > Test management for everyone who ships software. Written tests and browser > automation live on the same record, an MCP server lets an AI agent read and > write them, and a CLI runner executes them on a machine you control. Full documentation, 15 pages, generated from https://www.smarttest.cloud/docs --- Source: https://www.smarttest.cloud/docs Documentation # What SmarTTest is A testing platform for everyone who ships software, built on features that earn their place, an engine that lets AI agents do real work, and a refusal to make testing slow. ## Who it is for Testing tools have historically been built for one job title. They assume a dedicated QA team, a formal test plan, and someone whose full-time work is maintaining all of it. That describes fewer and fewer teams. Today the person writing a test might be a QA engineer. It might just as easily be the developer who built the feature, or someone shipping a product with an AI agent doing most of the typing. All three need the same things: somewhere to keep what has been verified, a way to run it, and an honest answer to "is this working?" SmarTTest is built for all of them. Nothing in it requires a QA background to use, and nothing in it is dumbed down for people who have one. ## Three things it is built on ### Features that actually add value Most test management tools are a long feature list where a handful of things get used and the rest exist to win comparisons. Every feature in SmarTTest is here because it answers a question teams genuinely ask. - Four automation states instead of a yes/no flag, because "not automated" and "deliberately never automating this, here is why" are not the same answer, and the difference is what you need in a review a year later. - A Quality Engine that scores automation out of 100 and names what is wrong. Automation that runs is easy. Automation you can trust is not, and the difference is usually a missing assertion or a fixed sleep that nobody looked for. - Health metrics about decay, not vanity. Which areas have not been executed in months, which tests fail at random, whether coverage is keeping up. - Nothing is ever destroyed. A deleted test is archived, because every run that references it would otherwise be left with a hole. ### An engine agentic AI can actually use Plenty of tools have added an AI feature. That usually means a chat box that writes test cases nobody asked for. SmarTTest is built the other way around: the platform exposes its real capabilities to an agent, and the agent does the work you would otherwise do by hand. Writing browser automation is a loop: read the test, write a plan, run it, see what broke, fix it, run it again. An agent connected through the MCP server can close that loop on its own. It reads the test from SmarTTest, writes the automation back into it, executes it in a real browser, and calls `analyze_test_quality` to check its own work against a deterministic scorecard rather than its own judgement. The agent is a participant, not a feature All forty tools are the same operations the interface performs, with the same permissions and the same rules. There is no separate AI mode that behaves differently from the rest of the product. ### Usability and speed, kept in view A testing tool that slows down development gets abandoned, no matter how thorough it is. Two decisions follow from that. - A test is one document. The written steps and the automation live on the same record. There is no separate manual suite and automated suite drifting apart in two systems. - Iterating leaves no trace. While an agent is developing automation it can execute without recording anything anywhere. Only real executions land in a run, so your history stays a record of what happened rather than of every attempt. ## The three pieces - The web app: where tests, folders, tags, runs, and results live. The source of truth, and the only thing that stores anything. - SmarTT MCP: the server that exposes SmarTTest to an AI agent. See SmarTT MCP. - SmarTT Runner CLI: a small program on a machine you control that executes the automated tests. See SmarTT Runner CLI. Why execution happens on your machine The tests you write point at your own applications: staging behind a VPN, local builds, internal tools. Something running elsewhere usually cannot reach any of them. The runner works the other way around, asking SmarTTest for work and reporting back, so nothing ever connects into your network. SmarTTest Cloud, coming soon If you would rather not run the machine yourself, we are building hosted execution: you point SmarTTest at your environment and we take care of the infrastructure your tests run on. The runner is not going away either, because plenty of applications are only reachable from inside your own network. ## How things relate The hierarchy is deliberately short. An organization owns everything. Inside it are projects, and inside a project are folders, which hold tests. A run takes a set of those tests and records what happened when somebody, or something, executed them. Two things cut across that tree rather than living inside it: tags, which group tests that sit in different folders, and reusable actions, which let many tests share the same setup. ## Where to start If you have never used SmarTTest, follow the quickstart. It takes an empty account all the way to automated tests running on your own machine. QuickstartFrom an empty account to a run executing on your machine, end to end.Read TestsThe core entity. Steps, expected results, and the automation plan.Read ## Everything in these docs The platform ProjectsThe top-level container. Everything you create lives inside one.Read TestsSteps, expected results, priority, and the automation plan.Read TagsCross-cutting labels for slicing your library any way you need.Read Test RunsA snapshot of executing a set of tests, with a result per test.Read Reusable ActionsNamed blocks of steps shared across many tests.Read Execution SettingsBrowser, environments, variables, and runners.Read Health DashboardThe state of your library: coverage, automation, stale areas.Read Test ArchiveArchived tests are never destroyed. Find and restore them.Read Export DataTake your tests and results out whenever you want.Read Users & RolesWho can do what: Tester and Org Admin.Read Automation SmarTT MCPConnect an AI agent so it can read and write your tests.Read MCP tool referenceEvery tool the server exposes, and when to reach for it.Read SmarTT Runner CLIThe command-line runner that executes tests on your machine.Read ## Reading this with an AI agent The whole documentation is published as plain text, generated from these same pages, so an agent can read it without scraping HTML. Point it at either file: - /llms.txt: the index, with one line per page. Useful when the agent should pick what to read. - /llms-full.txt: every page, end to end, in one file. Tip If your agent is already connected through the MCP server, it can read your actual tests instead of only the documentation, which is usually the faster way to get an answer about your own project. NextQuickstart --- Source: https://www.smarttest.cloud/docs/quickstart Start here # Quickstart This walks an empty account all the way to automated tests running on your machine. Ten minutes if you already have Node installed. ## Part 1: Get your first tests written - 1Create a projectGo to Organization → Projects and create one. A project is the container for everything else, and nothing can be created until one exists. See Projects. - 2Create a root folderIn the sidebar, New Root Folder. Folders are how tests are organized. One per feature area is a good default. You can nest them as deep as you need. - 3Create a testOpen the folder and add a test. Give it a title, a priority, and the steps a person would follow. You do not need automation yet; a written test is already useful on its own. See Tests. - 4Tag itAdd a tag like smoke or checkout. Tags cut across folders, which is what makes it possible to build a run out of "everything smoke" later. See Tags. _A test is one document: title, priority, steps, expected result, and optionally an automation plan._ ## Part 2: Tell SmarTTest about your app Before anything can be executed, SmarTTest needs to know where to execute it and with what. An environment is exactly that: a named set of values, such as the URL to open and the account to log in with. The same test then runs against staging or production without being rewritten. - 1Create an environmentExecution Settings → Environments. Name it after the target, like Staging. - 2Add the variables it needsA BASE_URL at minimum. Credentials go in as secret variables, which are stored encrypted and never displayed again. See Execution Settings. Tip Variables can be defined at three levels (organization, project, and test) and overridden per environment. Put anything shared at the organization level so you write it once. ## Part 3: Connect a machine that can run browsers Automated tests run on a runner: a small program you install on a machine you control, which is where the browser actually opens. It asks SmarTTest for work and reports the results back, so it works fine behind a VPN or a corporate firewall. SmarTTest Cloud, coming soon Hosted execution is on the way, so you will be able to hand us the infrastructure instead of running a machine yourself. - 1Create a runner in the appExecution Settings → Runners → New runner. You get a token, shown once. Copy it. - 2Install the CLI on that machineterminal ``` npm install -g @smarttest/runner ``` Node 18 or newer. If you would rather not install it globally, every command below also works with npx @smarttest/runner. - 3Connect it to your accountterminal ``` smarttest-runner login --token smarttest-runner install ``` The second command downloads the browser and the video encoder the runner needs. - 4Start itterminal ``` smarttest-runner start ``` It now shows as Online in the app and will pick up any work you send it. Full details in SmarTT Runner CLI. _Runners registered for the project, with their status and the machine they are installed on._ ## Part 4: Automate a test Automation is a plan attached to the test: an ordered list of browser commands. You can write it by hand, but the intended path is to let an AI agent do it, because the agent can read the test, write the plan, execute it, see what broke, and fix it, without you translating between the two. - 1Connect the MCP serverThis is what gives your agent access to SmarTTest. Setup for both the remote and local options is in SmarTT MCP. - 2Ask the agent to automate the testIt reads the written steps and turns them into an automation plan. It can then run analyze_test_quality to score its own work and fix what the scorecard flags. You do not have to automate everything A test can be marked `No`, `Impossible`, or `Discarded` instead of `Yes`. The last two ask for a reason, which is what keeps "why isn't this automated?" answerable a year later. ## Part 5: Execute a run - 1Create a runTest Runs → New run. Scope it to folders, tags, or hand-picked tests. See Test Runs. - 2Send the automated tests to your runnerOpen the run and press SmarTT Execute. Pick the runner and the environment, choose how many tests run at once, and queue it. Manual tests are skipped and stay yours to fill in. - 3Watch results landThe run updates as each test finishes, with a recording attached to every automated result. - 4Close the runWhen everything has a result, close it with a final comment. That freezes it and generates the summary. _Each test in the run carries its own status, comment, and, for automated tests, a recording of the execution._ ## What to read next - Tests: the automation plan in detail, and the Quality Engine. - Execution Settings: variables, secrets, and how they resolve. - MCP tool reference: every tool an agent can call. PreviousOverviewNextProjects --- Source: https://www.smarttest.cloud/docs/projects The platform # Projects A project is the boundary around a body of testing work. Folders, tests, tags, runs, reusable actions and runners all belong to exactly one. ## What a project is for A project is an isolation boundary. Everything inside one is invisible from the others: tests, folders, tags, runs, reusable actions, variables and runners all belong to exactly one project and never leak across. That matters in two common situations. A company with several unrelated products does not want one team browsing another team's test library, or picking the wrong tag when building a run. An agency or consultancy testing for several clients has a harder requirement: one client's data must never be reachable from another client's workspace, and that has to hold whether the access is through the interface, an export, or an AI agent connected to the MCP. The rule for splitting is not size. It is whether the two sets of tests would ever belong in the same run. If they never would, they belong in separate projects. Only one project is active at a time. The active project decides what the sidebar shows, what a new test gets attached to, and what a run can pull from. Switching projects switches the entire workspace. _Three projects in one organization: a storefront, a payment integration, and an inherited legacy portal. Nothing in any of them is visible from the others._ ## What a project holds Parameter | Type | Description --- | --- | --- name (required) | string | Unique within your organization. description | string | Free text. Useful when several projects have similar names. runnerBrowser | chrome | msedge | chromiumdefault chrome | Which browser the runner drives for this project's tests. Configured in Execution Settings. runnerViewportWidth | numberdefault 1920 | Window width used during execution, and the width of the recorded video. runnerViewportHeight | numberdefault 1080 | Window height used during execution. ## Active and inactive projects Projects are never deleted outright. Deactivating one puts it in read-only mode: everything stays visible and exportable, but nothing inside it can be created, edited, or executed. This is what you want when a product is retired but its test history still matters for audits. - An inactive project still appears in exports and in the archive. - Runs that were open when it was deactivated stay open, but cannot be modified. - Reactivating restores full access. Nothing is lost in between. Read-only is enforced everywhere The restriction is applied on the server, not just hidden in the interface. An AI agent working through the MCP gets the same refusal: `getWritableProject` rejects an inactive project before any write happens. ## Folders inside a project Folders are the tree in the left sidebar. They nest as deep as you like, and a test lives in exactly one of them. Folder names are capped at 30 characters, which is deliberate: a folder tree stops being navigable when the names are sentences. ### Choosing a structure The structure that survives is usually the one that mirrors how the product is discussed: by feature area, not by test type. "Checkout", "Search", "Account" ages well. "Regression", "Smoke", "P1" does not, because a single test belongs to several of those at once. That is what tags are for. Note Deleting a folder does not destroy the tests inside it. They are archived, and remain recoverable from the Test Archive. ## Who can manage projects Creating, renaming, and deactivating projects is restricted to Org Admins. Testers work inside whatever projects exist. See Users & Roles. ## Through the MCP An agent lists the projects it can reach with `list_projects`, which returns the id every other tool needs. There is no tool to create or delete a project. That stays a human decision made in the app. PreviousQuickstartNextTests --- Source: https://www.smarttest.cloud/docs/tests The platform # Tests A test is one document that both a person and a machine can execute. The written steps and the automation plan live side by side, so they cannot drift apart. ## Anatomy of a test Every test has a written half and, optionally, an executable half. The written half is what a person reads during a manual pass and what a reviewer checks. The executable half is an ordered list of browser commands. Both describe the same behaviour. _The test editor. Written steps on one side, the automation plan attached to the same record._ ## Fields These are the fields the test editor gives you, in the order they appear. Parameter | Type | Description --- | --- | --- Title (required) | text | What the test verifies, in one line. Priority (required) | Low | Medium | High | Critical | Used for filtering, and for deciding what belongs in a smoke run. Tags | multi-select | Cross-cutting labels. See Tags. References | text | Ticket ids or links. What explains why this test exists. Automated? | No | Yes | Impossible | Discardeddefault No | Whether this test has automation. See below. Automation status | Completed | In Progress | Needs Attention | Broken | How the automation is holding up. Only appears when `Automated?` is `Yes`. Why can't it be automated? / Discard reason | text | Only appears for `Impossible` and `Discarded`. Maximum 500 characters. Description | text | The context a reader needs before the steps make sense. Preconditions | text | The state the system must be in before step 1. Setup that has to actually run belongs in a reusable action instead. Steps (required) | ordered list | Each step has an action, its own expected result, and optionally the automation commands that perform it. Note The editor has two tabs over the steps: Manual Test Design for the written version, and SmarTT Playwright Automation for the executable one. They describe the same test, so changing one is a prompt to check the other. ## The four automation states "Automated: yes or no" loses the most useful information, which is why a test that could be automated still isn't. SmarTTest keeps four states instead: - Yes: it has an automation plan. Pair it with an automationStatus so you can tell finished automation from work in progress and from automation that is currently broken. - No: not automated yet. The default, and the honest state for a backlog. - Impossible: cannot be automated at all: a physical device, a third-party flow you do not control, a visual judgement. Requires a reason. - Discarded: could be automated, but the team decided it isn't worth it. Requires a reason. Why the reason is mandatory `Impossible` and `Discarded` are the two states that get questioned in a review a year later. Forcing a reason at the moment of the decision is the only time anyone actually remembers it. It also makes the Health Dashboard honest: unautomated tests with a justification are not the same problem as unautomated tests without one. Heads up `automatedRejectReason` is not a description of the automation. Putting a summary of the script there when the test is automated is rejected. The field only exists for the two negative states. ## The automation plan Automation is stored as an ordered list of actions. Each action has a `command`, an `input` object, and optionally a `saveAs` that captures its result into a runtime variable. an action ``` { "command": "playwright_browser_click", "input": { "role": "button", "name": "Sign in" } } ``` ### Available commands Grouped by how often you will reach for them. Parameter | Type | Description --- | --- | --- playwright_browser_navigate | common | Open a URL. Usually the first action of a test. playwright_browser_click | common | Click an element. playwright_browser_type | common | Type into a field. playwright_browser_fill_form | common | Fill several fields in one action, instead of a click-and-type pair for each. playwright_browser_select_option | common | Choose from a dropdown. playwright_browser_press_key | common | Press a key. Only real key names are accepted, so a typo is rejected on save. playwright_browser_hover | common | Hover, to reveal menus or tooltips. playwright_browser_wait_for_element | common | Wait until an element is there. The correct alternative to a fixed sleep. playwright_browser_wait_for | common | Wait for text to appear or disappear, or for a fixed time. playwright_browser_take_screenshot | common | Capture the page, or a single element. The image is attached to the run result. smartt_read_element | capture | Read a value off the page into a runtime variable. Requires saveAs. smartt_assert | assertion | Compare a captured value against an expectation. See below. smartt_run_reusable_action | common | Run a shared block of steps here. Editing the block updates every test that uses it. playwright_browser_navigate_back | less common | Browser back. playwright_browser_drag | less common | Drag an element. playwright_browser_drop | less common | Drop onto a target. playwright_browser_file_upload | less common | Attach a file to an input. playwright_browser_handle_dialog | less common | Accept or dismiss a native alert / confirm / prompt. playwright_browser_resize | less common | Resize the window mid-test. playwright_browser_tabs | less common | Open, switch, or close tabs. playwright_browser_get_url | capture | Current URL into a variable. Requires saveAs. playwright_browser_get_title | capture | Page title into a variable. Requires saveAs. playwright_browser_console_messages | capture | Console output into a variable, useful to assert no errors were logged. playwright_browser_network_requests | capture | All network activity into a variable. playwright_browser_network_request | capture | One matched request into a variable, to assert on its status or body. playwright_browser_close | advanced | Close the browser explicitly. ### Runtime variables Six commands return a value, and all six must capture it. That is what `saveAs` is for. A command that reads something and throws the result away is almost always a mistake, so it is rejected rather than silently accepted. Once captured, refer to the value anywhere as `{{runtime.NAME}}`. capture, then assert ``` [ { "command": "smartt_read_element", "input": { "role": "heading", "name": "Order confirmed" }, "saveAs": "confirmation" }, { "command": "smartt_assert", "input": { "actual": "{{runtime.confirmation}}", "operator": "contains", "expected": "Order confirmed" } } ] ``` The most common automation mistake `smartt_assert` requires `actual` to hold a `{{runtime.NAME}}` reference, not a literal. Writing `"actual": "Order confirmed"` compares a constant to a constant, which proves nothing about the page, so the Quality Engine flags it. Capture the real value first, then assert on the capture. ### Assertion operators `smartt_assert` takes `actual`, an `operator`, and usually an `expected`: - Comparison: equals, notEquals, contains, notContains, matches (regular expression). - Numeric: greaterThan, lessThan, greaterOrEqual, lessOrEqual. - No expected needed: isTrue, isFalse, isEmpty, isNotEmpty. ### How to point at an element Most commands need to identify an element. The order below is not stylistic. It is how resistant each one is to a redesign that does not change behaviour: - Role and name: { "role": "button", "name": "Sign in" }. Matches what a user sees, and survives markup changes. - Label: for form fields, the visible label. - Test id: stable, but only if your app actually maintains them. - CSS selector: the last resort. It breaks on refactors that changed nothing a user would notice, and that is exactly what makes a suite feel unreliable. Names match by substring, so a link named `Free` also matches "Start for Free". When more than one element matches, the step fails instead of picking one at random, which is the right call: a test that silently chose would pass or fail depending on render order. - exact: true requires the whole name to match. This is the fix you want most of the time. - nth: 0 picks one when several elements really are identical, like a row in a list. Tip `validate_automation` tells you how many elements a locator matches before you run anything, and shows the text of each one. See the tool reference. ## The Quality Engine Automation can be syntactically valid and still be worthless: no assertions, fixed sleeps, a password typed in as a literal. The Quality Engine reads a test's automation and returns a score out of 100 with specific findings. It is entirely deterministic: no AI, no cost, same input gives the same output. ### The same screen, two tests The scorecard sits above the automation plan. These two tests were written against the same application on the same afternoon. _49/100, Fragile. The line that matters is 8 actions, 0 verifications: this test clicks its way through checkout and never checks that anything happened, so it would pass on a completely broken payment flow. The ten findings are listed per step and per action, not as a vague overall complaint._ _100/100 on a test doing the same kind of work. It verifies a result, finds its elements by role and name, and reads its credentials from variables instead of having them typed into the steps._ Nothing about the first test is unusual. It was written quickly to move a coverage number, it runs, and it goes green. That is exactly why a deterministic score is worth having: nobody would have caught this by reading it. It checks eight things: - Verifications: does the test actually assert anything, or does it just click around? - Locators: how brittle the element references are. - Hard waits: fixed sleeps, which are the main source of flakiness. - Text waits: waiting on text that will not reliably appear. - Step actions: whether the automation matches the written steps. - Hardcoded data: credentials, URLs, and other values that belong in a variable. - Runtime variables: values captured but never used, or asserted without being captured. - Empty steps: written steps with no automation behind them. ### It catches flakiness before you have any A flaky test is usually not born flaky. It is written with a pattern that will eventually flap, and the flapping only shows up weeks later on a slower day. Much of what the engine flags is exactly that pattern, caught while the automation is still on screen: - A fixed sleep instead of a wait. Waiting three seconds passes on your machine and fails on a loaded CI box. The fix is waiting for the element, and the finding says so. - A CSS selector where a role and name would do. It survives until somebody reorders a div, and then it fails for a reason that has nothing to do with the product. - Waiting on text that is not guaranteed to appear. A wait that sometimes has nothing to wait for is a timeout waiting to happen. - A value read and never asserted. The test looks like it checks something and does not, so it passes through real breakages and fails on unrelated noise. Note This is the preventive half. The Health Dashboard handles the other half: tests already producing inconsistent results across runs, ranked by how unstable they are. One stops flakiness being written, the other finds what slipped through. ### Every finding comes with a prompt Each finding has a copy fix prompt action. It generates an instruction written for that specific finding, on that specific step, ready to paste into a terminal agent such as Claude Code or Cursor. The prompts are built per rule rather than generically, because "fix this test" makes an agent guess, and a guess is how a brittle locator becomes a differently brittle locator. Tip An agent connected through the MCP skips the copying entirely: it calls `analyze_test_quality` on its own work and fixes what comes back. That loop of write, score, fix is what makes the difference between automation that runs and automation you can trust. ### A side effect worth having: accessibility The engine pushes you towards locating elements by their role and accessible name, because that is what survives a redesign. That happens to be the definition of an accessible interface too: a button a screen reader can announce is a button your test can find by name. Plenty of legacy applications have neither. When a test cannot be written against roles and names because the page exposes none, that is not a testing problem to work around with a CSS selector. It is an accessibility gap the automation just found. Teams automating older applications with SmarTTest tend to end up adding labels and roles to the product itself, which makes the tests stable and the application usable by more people at the same time. ## Flakiness tracking As a test accumulates results, SmarTTest tracks whether it produces consistent outcomes. A test is `STABLE`, `FLAKY`, or a `REGRESSION`, with a score behind it. Flaky tests surface on the Health Dashboard, because a suite where failures are routinely ignored is worse than no suite at all. ## Archiving Tests are never hard-deleted. Archiving is a soft delete: the test disappears from the tree but keeps its history and its past run results. See Test Archive. ## Through the MCP `list_tests`, `get_test`, `create_test`, `update_test`, `archive_test`, `restore_test`, and `analyze_test_quality`. Full parameters in the tool reference. PreviousProjectsNextTags --- Source: https://www.smarttest.cloud/docs/tags The platform # Tags Folders answer where a test lives. Tags answer everything else, and a test can carry as many as it needs. ## Why tags exist A test belongs to exactly one folder, which is the right model for "what part of the product is this?". But the same test is also part of the smoke suite, also touches payments, also only matters on mobile. Those are not places. They are properties, and a folder tree cannot hold more than one of them at a time. Tags are that second axis. Their real payoff shows up when you build a run: scoping it to a tag pulls in the right tests no matter which folders they live in. _Manage → Tags. Each tag shows how many active tests carry it._ ## What a tag holds Parameter | Type | Description --- | --- | --- name (required) | string | Short and lowercase works best. description | string | What qualifies a test for this tag. Worth filling in, because it is what stops a tag from meaning three different things to three people. ## Using them well - Suites: smoke, regression, release-blocker. These are the ones you will scope runs to most often. - Risk and domain: payments, auth, gdpr. Useful when a change lands in one area and you want everything that touches it. - Context: mobile, slow, needs-real-data. These tell you what a test costs to run. Keep the list short A tag that is on one test is a note, not a tag. If a label does not eventually group several tests, it belongs in the test's description. Tag lists rot when nobody prunes them, and a rotten list stops being usable for scoping runs, which was the whole point. ## Scope Tags belong to a project, and you see the ones for whichever project is active. Removing a tag from a test does not delete the tag; deleting the tag removes it from every test that carried it, but does not touch the tests themselves. Note A tag name only has to be unique inside its project. Two projects, or two organizations, can each have their own `smoke` without knowing about each other, so you can name tags for what they mean instead of working around what someone else already took. ## Through the MCP `list_tags` and `create_tag`. Tags are also applied and removed as part of `create_test` and `update_test`, by name rather than by id, so an agent can tag a test without looking anything up first. See the tool reference. PreviousTestsNextTest Runs --- Source: https://www.smarttest.cloud/docs/test-runs The platform # Test Runs A run is the record of executing a set of tests at a moment in time. Tests change; a run does not. That is what makes it worth keeping. ## What a run is When you create a run, SmarTTest takes the tests you scoped it to and creates one result slot for each. Every slot starts as `PENDING` and ends up with a status and a comment. The run stays open until you close it deliberately. _Each row is one test's result inside this run. Automated results carry a recording of the execution._ ## Creating a run - 1Name it after the thing being verified"Release 4.2 regression", not "Run 17". Include timestamp adds a date and time to the name, which is worth turning on when you run the same suite repeatedly and off when the name already identifies the moment. - 2Scope itBy folders, by tags, or both. You have to pick something: a run with no scope used to quietly include every test in the project, which is rarely what anyone means. Select all next to the folder tree takes the whole project when that genuinely is what you want. - 3Add an opening commentWhat this run is for, what build it targets. It is the first thing a reader sees later. _Until a folder or a tag is picked there is nothing to put in the run, so the button stays disabled. Select all takes the whole project._ ## Result statuses Parameter | Type | Description --- | --- | --- PENDING | status | Nobody has executed this test yet. The starting state. IN_PROGRESS | status | Somebody has started, or a runner has picked it up and is executing it right now. PASSED | status | Behaved as the expected result describes. FAILED | status | Did not. The comment is where the detail goes. SKIPPED | status | Deliberately not executed: out of scope for this build, blocked by an environment problem, or superseded. Write a comment, especially on failures The interface does not force one, but a run full of bare pass/fail marks is unreadable three weeks later, and a failure with no note is a failure nobody can act on. Through the MCP a comment is mandatory, precisely because an agent recording results has even less excuse for leaving them unexplained. ## Executing automated tests with SmarTT Execute Manual tests you work through yourself, marking each result. For automated tests, press SmarTT Execute in the run header and hand the work to a runner. _Choose the runner, the environment, how many tests run at once, and which results to execute._ ### What the dialog asks for Parameter | Type | Description --- | --- | --- Runner | select | The machine that executes. An offline runner can still be chosen; the job waits in the queue until it comes back. Environment | select | Which set of variables the tests get: which URL, which credentials. Tests at once | 1 – 8default 1 | Raise it only when the tests are independent. Tests that share a login or a fixture will collide with each other. What to run | all | pending | faileddefault all | Re-running only the failures is the usual second pass, once you have fixed whatever broke. Browser window | hidden | visibledefault hidden | Whether the browser is visible on the runner machine. Hidden is faster; visible is useful when you are watching a failure happen. Manual tests are skipped, not failed SmarTT Execute only touches tests marked `automated: Yes`. The rest keep their current status and stay yours to fill in. Heads up A run can only have one job queued or running at a time. Queueing the same run twice would let the second job overwrite results the first one is still producing, so it is refused. ## Recordings and evidence Every automated result carries a video of the execution and how long it took. A failure you can watch is a failure you can diagnose without reproducing it locally first. On manual results, the comment field is a rich text editor: paste screenshots or drop a screen recording straight into it. ## Closing a run Closing freezes the run. You can add a final comment, though nothing forces you to. Either way SmarTTest reads through the results and writes a SmarTT Summary: what passed, what failed, what the failures had in common, and whether the run is a reasonable basis for shipping. That summary is appended to the closing comment, so the outcome is readable without opening every row. - A closed run cannot receive new results, and cannot be sent to a runner. - It still appears in exports and feeds the Health Dashboard. - Reopen it if you genuinely need to add something. Close runs promptly Only completed runs count toward pass-rate trends. A run left open for a month is a month of missing data on the Health Dashboard. ## Through the MCP `list_runs`, `get_run`, `create_run`, `update_run_test_result`, and `complete_run`. To execute, an agent calls `run_on_runner` and polls `get_runner_job`. See the tool reference. PreviousTagsNextReusable Actions --- Source: https://www.smarttest.cloud/docs/reusable-actions The platform # Reusable Actions Log in. Accept the cookie banner. Seed a cart. Every suite has a handful of blocks that appear in dozens of tests, and that all break on the same day when the UI changes. ## The problem they solve Copying a five-step login into forty tests works right up until the login form changes. Then you have forty tests to fix, you fix thirty-eight, and the two you missed fail for a week before anyone notices. A reusable action is that block, defined once. Tests reference it. When the login changes you edit one place, and every test that uses it is correct again immediately. _Execution → Reusable Actions. Each block shows which tests currently use it._ ## What a reusable action holds Parameter | Type | Description --- | --- | --- name (required) | string | Describe the outcome, not the mechanics: "Log in as admin" rather than "Fill form". description | string | What state the system is in once it finishes, and anything it assumes beforehand. usableAs | PRECONDITION | STEP | BOTHdefault PRECONDITION | Where this block is allowed to be used. See below. steps (required) | step[] | The automation commands that make up the block. ## Precondition or step The distinction is about intent, and it keeps the library from turning into a pile of loose fragments. - PRECONDITION: setup that runs before the test begins. It is not what the test is verifying; it is what has to be true first. Logging in is the classic case. - STEP: a block used inside the test flow, as one of the actions being exercised. - BOTH: when a block genuinely serves either purpose. Reach for it last: a block that is allowed anywhere gives no signal about how it is meant to be used. ## Attaching preconditions to a test A test can have several preconditions attached, and they run top to bottom. Order matters: logging in before selecting an account is not the same as the reverse, so the order is explicit and reorderable rather than incidental. Note Preconditions run before the test's own first step, every time it executes. If a block is slow, every test that attaches it pays that cost on every run. ## Calling one from inside a test Within an automation plan, the `smartt_run_reusable_action` command runs a block at that exact point: calling a reusable action mid-test ``` { "command": "smartt_run_reusable_action", "input": { "reusableActionId": "" } } ``` ## Blocks cannot call other blocks A reusable action may not contain `smartt_run_reusable_action`. This is enforced, not a convention. Nesting sounds convenient and turns bad quickly: a failure three levels deep gives you a stack of names instead of the step that actually broke, and a change to a low-level block silently alters tests whose authors never referenced it. Keeping blocks one level deep means a test's behaviour is always readable from the test plus the blocks it names. If two blocks share steps, inline them. ## Editing and deleting - Editing a block changes behaviour for every test that uses it, immediately. That is the point, and it is also the risk, so check the usage list before you change one. - Each block shows which tests reference it, so the blast radius is never a guess. - Deleting a block that is still attached to tests is refused. Detach it first. Keep blocks small and single-purpose "Log in" is a good block. "Log in, go to settings, enable the beta flag, and open the dashboard" is four blocks pretending to be one, and no test will want exactly that combination twice. ## Scope and permissions Reusable actions belong to a project. Managing them is restricted to Org Admins, since a single edit propagates across the whole suite. ## Through the MCP `list_reusable_actions`, `get_reusable_action`, `create_reusable_action`, `update_reusable_action`, `delete_reusable_action`, `attach_reusable_action_to_test_preconditions`, `detach_reusable_action_from_test_preconditions`, and `reorder_test_reusable_preconditions`. See the tool reference. PreviousTest RunsNextExecution Settings --- Source: https://www.smarttest.cloud/docs/execution-settings The platform # Execution Settings Everything a test needs in order to actually run: which browser, against which environment, with which values, on which machine. ## Five tabs - Browser & Resolution: how the browser is launched for this project. - Runners: the machines allowed to execute this project's tests. - Environments: the targets you run against: staging, production, local. - Organization Variables: values shared by every project. - Project Variables: values for the active project only. Note This whole screen is restricted to Org Admins. It holds credentials and grants machine access, so it is not open to every tester. ## Browser & Resolution Sets how the browser is launched for this project's tests. Both the window and the recorded video use this size, so a low resolution here produces recordings you cannot read. Parameter | Type | Description --- | --- | --- Browser | chrome | msedge | chromiumdefault chrome | chrome and msedge drive the real browser installed on the runner machine. chromium uses the build that ships with Playwright, which is more reproducible but not what your users have. Width | numberdefault 1920 | Viewport and recording width. Height | numberdefault 1080 | Viewport and recording height. Tip Test at the resolution your users actually have. A layout bug that only appears on a 1366-wide laptop will never be caught by a suite that always runs at 1920. ## Environments An environment is a named target: `Staging`, `Production`, `Local`. On its own it holds almost nothing; what makes it useful is that variables can be given a different value per environment. The same test then runs against staging or production without a single edit. Every execution picks exactly one environment. It is a required choice, so there is no default that would let you accidentally point a destructive test at production. ## Variables Variables keep URLs, accounts, and credentials out of the tests themselves. In an automation plan you reference one by namespace and key: referencing variables ``` { "command": "playwright_browser_navigate", "input": { "url": "{{orgVar.BASE_URL}}/checkout" } } ``` ### Three namespaces The namespace is part of the reference, so which variable you mean is always explicit. There is no invisible cascade to reason about. Parameter | Type | Description --- | --- | --- {{orgVar.KEY}} | organization | Shared by every project. Where anything used in more than one place belongs. {{projectVar.KEY}} | project | Only for the active project. {{testVar.KEY}} | test | Attached to a single test. For values only that test cares about. ### Environment overrides Within a namespace, a key can have a base value plus a value for a specific environment. At execution time the environment-specific value wins if there is one; otherwise the base value is used. So `orgVar.BASE_URL` might be `https://staging.acme.com` for the Staging environment and `https://acme.com` for Production, while the test just says `{{orgVar.BASE_URL}}`. _Execution Settings, Organization Variables. A secret shows a mask instead of its value, and cannot be read back._ ### Secrets Marking a variable secret encrypts the value at rest and stops it ever being displayed again. The interface shows a mask, and the MCP returns only whether a value exists, never the value itself. It is decrypted at execution time and handed to the runner. Never put a credential in a test A password typed into an automation step is stored in plain text, appears in exports, and shows up in the recording. The Quality Engine flags hardcoded credentials for exactly this reason. Put them in a secret variable. ## Runners A runner is a small program you install on a machine you control. That machine is where the browser opens and the automated tests actually execute; the runner reports the results back to SmarTTest. _Each runner shows its status, the machine it is installed on, and when it was last seen._ ### Nothing connects into your network The runner polls SmarTTest for work: every five seconds when idle, every three while executing. Nothing ever opens a connection towards the runner. That is what lets it sit behind a VPN, a corporate firewall, or on a laptop with no public address, with no networking changes at all. ### Status Parameter | Type | Description --- | --- | --- Online | status | Seen within the last 60 seconds. Ready to pick up work. Offline | status | Not seen for over 60 seconds, which is more than ten missed polls, so this is a stopped runner rather than a network hiccup. Jobs sent to it wait in the queue until it returns. Never connected | status | Created in the app, but the token was never installed on a machine. Revoked | status | Its token no longer works. Permanent. ### Setting one up - 1Create it hereGive it a name that identifies the machine. You get a token, shown exactly once. - 2Install it on that machineterminal ``` npx @smarttest/runner login --token npx @smarttest/runner install npx @smarttest/runner start ``` Full command reference in SmarTT Runner CLI. ### What a runner token can do A runner token is deliberately narrow. It can read the tests it has been asked to execute and write back their results, and nothing else. It cannot create or edit tests, read your variables outside an execution it was given, or reach another project. The token is shown once Only a hash is stored, so it cannot be recovered, not even by us. If you lose it, revoke the runner and create a new one. ### Revoking Revoking stops the token working immediately and cancels anything queued for that runner. It cannot be undone. If the runner is online at the time it may be executing tests right now, and that work is cancelled. The app warns you before you confirm. ## Through the MCP `get_execution_settings`, `list_environments`, `get_org_variables`, `get_project_variables`, `get_test_variables`, and `list_runners`. Note that the variable tools report which keys exist, never secret values. See the tool reference. PreviousReusable ActionsNextHealth Dashboard --- Source: https://www.smarttest.cloud/docs/health-dashboard The platform # Health Dashboard A test suite decays quietly. Tests stop being run, automation breaks and nobody notices, failures get ignored because they are 'always like that'. This screen is where that becomes visible. ## Availability Note The Health Dashboard is available on paid plans. On the Free tier the entry does not appear in the sidebar, and the page itself redirects to the dashboard. ## What it measures Parameter | Type | Description --- | --- | --- Health Score | 0 – 100 | A single composite figure for the project. Useful as a trend line, not as a target. Chase the underlying metrics, not this number. Completed runs | count | How many runs were closed in the range. Only completed runs count, which is a good reason to close them promptly. Pass Rate Trend | % over time | Where the suite is heading. A falling pass rate can mean the product is getting worse or the tests are. Both are worth knowing. Freshness | recency | How recently tests were actually executed. Surfaces the areas nobody has run in months, which are the ones most likely to be silently broken. Automation & Suite Growth | % and counts | Share of tests marked `Yes`. Tests marked `Impossible` or `Discarded` are a deliberate decision rather than a gap, which is exactly why those states exist. Flaky Test Index | list | Tests that pass and fail without the product changing. Every one of them trains the team to ignore a red result. Test Growth | count over time | New tests over the range. Flat growth on an actively developed product usually means coverage is falling behind. ## The three bands The Health Score is a single number per project, averaged over the completed runs in the range. It lands in one of three bands, and the colour is the fastest read on the screen. Parameter | Type | Description --- | --- | --- 80 to 100 | Healthy | Green. Failures are rare and the ones that happen are real, not noise. 50 to 79 | Warning | Amber. Something is eroding: a rising failure rate, a test that will not settle, or both. below 50 | At Risk | Red. The suite is no longer telling you whether the product works. The score starts at 100 for each run and comes down as failures land: a code error costs 10, a flaky failure costs 5, and an environment or infrastructure failure costs 2. A flaky failure is cheaper on purpose, because it says less about the product than a genuine break does. ### Healthy: a suite you can trust _Flat pass rate, nothing in the Flaky Test Index. A red result here means something, which is the whole point of keeping a suite._ There is nothing to do on this screen. Watch the trend rather than the number, and spend the attention on coverage instead: the Automation & Suite Growth panel is the one worth reading here. ### Warning: something is eroding _The pass rate slid from 78% to the mid-fifties over a handful of runs, and one test is failing about a fifth of the time._ This is the band where the screen earns its keep, because the problem is still small enough to fix. The shape matters more than the score: a steady decline is usually the product getting worse, while a jagged line is usually the tests. Start with the Flaky Test Index, because an unreliable test also drags the pass rate down and hides whatever else is happening. ### At Risk: the suite has stopped answering the question _A pass rate swinging between 20% and 55% run after run, and a test at 91% instability. Nobody reads these results any more._ A suite in this state costs more than it returns: every run produces failures the team has learned to scroll past, so a real regression arrives looking exactly like the noise. Do not try to fix the score. Pick the worst offenders in the Flaky Test Index, and either repair them or archive them. A smaller suite you believe is worth more than a large one you ignore. ## SmarTT Insight Alongside the numbers, the dashboard produces a written reading of them: what stands out, what is getting worse, and what is worth doing about it. It is meant to answer "so what?", the step between having metrics and acting on them. ## Scope and range Everything shown is for the active project. Switching projects switches the dashboard. The time range filter changes which runs and results are counted. ## Reading it well - Trends beat snapshots. A 78% pass rate means nothing on its own. A 78% that was 94% last month means quite a lot. - Flaky tests first. They cost more than a low automation ratio, because they erode trust in every other result. - Freshness reveals dead zones. An area that has not been executed in three months is untested regardless of how many tests it has. - Automation ratio is a means, not an end. A high ratio built on tests with no assertions is worse than a low one. That is what the Quality Engine is for. PreviousExecution SettingsNextTest Archive --- Source: https://www.smarttest.cloud/docs/test-archive The platform # Test Archive & History Nothing in SmarTTest is ever really deleted. Deleting a test archives it, and this is where archived tests and the full edit history of every test live. ## Deleting is archiving When you delete a test it gets a deletion timestamp and disappears from the folder tree, from searches, and from new runs. The record itself stays exactly where it was, along with every result it ever produced. This matters more than it sounds. A test deleted last March is still referenced by every run it appeared in. Destroying it would leave those runs with holes, and a run with holes cannot be used as evidence of anything. _Manage → Test Archive. The audit trail for every test in the project, and the place to restore archived ones._ ## The history trail Beyond archived tests, this screen holds the change history for tests that are still active. Every meaningful change is recorded with who made it and when: creation, edits, archiving, and restoration. - Answer "when did this test change, and who changed it?" without guessing. - Understand why a test that used to pass now fails: sometimes the product changed, sometimes the test did. - Keep the audit trail regulated environments ask for. ## Restoring Restoring puts the test back where it was, with its steps, automation, tags, and history intact. The restoration itself is recorded too. If the folder it belonged to is gone, restore the folder first. Note A restored test does not retroactively join runs that were created while it was archived. Those runs are a record of what was executed at the time, and that record does not change. ## Deleting a folder Deleting a folder archives the tests inside it rather than destroying them. They all show up here and can be restored individually. Heads up This is why deleting a folder is recoverable but still disruptive: every test inside leaves the tree at once, and each has to be brought back on its own. ## Inactive projects In a deactivated project the archive is readable but restoration is blocked, like every other write. See Projects. ## Through the MCP `archive_test` and `restore_test`. To see archived tests in a listing, pass `includeArchived: true` to `list_tests`: they are excluded by default. See the tool reference. PreviousHealth DashboardNextExport Data --- Source: https://www.smarttest.cloud/docs/export-data The platform # Export Data Your tests are yours. Export them as CSV at any time, from any folder, without asking anyone. ## Exporting - 1Pick the foldersSelecting a folder automatically includes every subfolder under it, so exporting a whole area is one click. Select All takes the entire project. - 2ExportA CSV downloads with one row per test. _Manage → Export Data. Choose the folders; subfolders come along automatically._ ## What the file contains Parameter | Type | Description --- | --- | --- id | string | The test's identifier in SmarTTest. title | string | What the test verifies. priority | string | LOW, MEDIUM, HIGH, or CRITICAL. description | string | The context field. preconditions | string | Written preconditions. steps | string | The written steps. expectedResult | string | What must be true at the end. tags | string | The tags on the test. folderPath | string | Full path, so the tree can be reconstructed. createdAt | date | When it was created. updatedAt | date | Last modification. The export covers the written test Automation plans, run results, and recordings are not in this file. It is the human-readable test, in the shape other test management tools expect. For programmatic access to runs and results, use the MCP: `list_runs` and `get_run` return the full structure. ## What it is good for - Sharing with people outside SmarTTest: auditors, clients, a stakeholder who wants a spreadsheet. - Bulk review: reading three hundred titles in one window is faster than clicking through a tree. - Your own backup: no reason to depend on us for a copy of your own work. - Migrating out. Yes, really. A tool that makes leaving hard is a tool you should be suspicious of. ## Coming the other way Migrating in is a separate path: Import Data → TestRail in the sidebar brings an existing TestRail library across, preserving the folder structure. Universal CSV import, planned A generic CSV import is on the roadmap, so a library exported from any tool, or maintained in a spreadsheet, can be brought in by mapping its columns instead of needing a dedicated integration. ## Secrets are never exported Encrypted variables are not part of any export, in any form. If a credential appears in your CSV, it is because it was typed into a test step as literal text, which is what the Quality Engine flags as hardcoded data, and a good reason to fix it. PreviousTest ArchiveNextUsers & Roles --- Source: https://www.smarttest.cloud/docs/users The platform # Users & Roles Everyone in an organization can read and write tests. The roles differ in who can change the things that affect everybody else. ## The organization An organization is the account boundary. All its users share the same projects, tests, runs, and variables, and nothing crosses from one organization to another. _Organization → Users. Adding people and changing roles happens here._ ## The two roles Parameter | Type | Description --- | --- | --- TESTER | roledefault assigned on creation | Full day-to-day access: create and edit tests, folders and tags, create and execute runs, record results, browse the archive, export data. ORG_ADMIN | role | Everything a Tester can do, plus the shared configuration: projects, reusable actions, execution settings, runners, users, and the subscription. ## What the extra permissions cover The split is not about trust in someone's testing. It is about which actions are hard to undo, or affect everyone at once: - Projects: creating and deactivating them changes what the whole team sees. - Reusable actions: one edit changes behaviour across every test that uses the block. - Execution Settings: this screen holds credentials and issues runner tokens, so it is genuinely sensitive. - Users and subscription: who is in the account, and what it costs. - TestRail import: a bulk write across the whole library. Note Restrictions are enforced on the server, not just hidden in the interface. That applies to the MCP too: an agent acts with the permissions of the user whose account it is connected through, so it cannot do anything that user could not do themselves. ## Adding people Org Admins add users from Organization → Users. New users are created as Testers; promote them afterwards if they need more. Seats depend on your plan The Free tier is limited to a single user. Adding a second requires upgrading. See Organization → Subscription. ## Choosing roles - Default to Tester. It covers everything the job needs day to day. - Org Admin for whoever maintains the setup: usually the person who owns environments, variables, and the runners. PreviousExport DataNextSmarTT MCP --- Source: https://www.smarttest.cloud/docs/mcp Automation # SmarTT MCP An MCP server that gives an AI agent direct access to your SmarTTest workspace: reading tests, writing automation, scoring its own work, and executing runs. ## What it is for Writing browser automation is a loop: read the test, write a plan, run it, see what broke, fix it, run it again. Doing that through a chat window means a person copying context in one direction and code back in the other, which is slow and lossy. The MCP server removes the copying. The agent reads the test from SmarTTest itself, writes the automation back into it, runs the Quality Engine on what it produced, and fixes what the scorecard flags, without you in the middle of each step. The agent acts as you It connects through your account and inherits your permissions exactly. It cannot touch a project you cannot touch, and it cannot read a secret variable. The variable tools report which keys exist, never their values. ## The package @smarttest/mcp on npm·Apache-2.0 ## Two ways to connect There is a hosted server you connect to over the network, and a local server you run on your own machine. They expose the same tools, with one difference that decides which you want. Parameter | Type | Description --- | --- | --- Remote (HTTP + OAuth) | hosted | Nothing to install. You log in and approve access in the browser. Works with any client that supports remote MCP connectors. Cannot drive a browser itself; it queues execution on a runner instead. Local (stdio) | npx | Runs on your machine, so it can drive a real browser directly. This is what you want while writing and debugging automation, because the develop-run-develop loop happens in place with no queue. Tip Most people end up with both: the local server while writing automation, the remote connector for everything else. ## Setting up the remote connector The remote server speaks Streamable HTTP and authenticates with OAuth 2.1, using dynamic client registration and PKCE, so there is no key to create, paste, or rotate. - 1Add the connector in your clientPoint it at:MCP server URL ``` https://www.smarttest.cloud/api/mcp/http ``` In Claude, this is Settings → Connectors → Add custom connector. - 2Approve accessYour browser opens on SmarTTest. Log in if you are not already, review what the connector is asking for, and approve. - 3DoneThe tools appear in your client. Access tokens are short-lived and refresh on their own. Execution is not available remotely Driving a browser has to happen on a machine you control, so `execute_test` and `execute_run` are not exposed over HTTP. Use `run_on_runner`, which queues the work on a machine that can run it. See SmarTT Runner CLI. ## Setting up the local server ### 1. Create an API key In SmarTTest, go to Settings → MCP API Keys and create one. It is shown once, and only a hash is stored, so it cannot be recovered later. _Settings → MCP API Keys. Keys can be revoked at any time, which immediately cuts off anything using them._ ### 2. Add it to your client Most clients use the same shape. For Claude Desktop or Claude Code: mcp config ``` { "mcpServers": { "smarttest": { "command": "npx", "args": ["-y", "@smarttest/mcp"], "env": { "SMARTT_MCP_API_KEY": "your-key-here" } } } } ``` ### 3. Restart the client The tools appear once it reconnects. ### Configuration Parameter | Type | Description --- | --- | --- SMARTT_MCP_API_KEY (required) | string | The key from Settings → MCP API Keys. SMARTT_API_BASE_URL | urldefault https://www.smarttest.cloud | Only needed if you point at a different SmarTTest instance. SMARTT_ARTIFACTS_DIR | path | Where local execution artifacts (screenshots, recordings) are written. ## Requirements for local execution Running tests locally needs a browser and a video encoder. The package installs Playwright as a dependency; the browser binaries themselves are a separate download: terminal ``` npx playwright install chromium npx playwright install ffmpeg ``` Missing ffmpeg fails in a confusing way Without ffmpeg, executions die almost immediately with a path error rather than anything about video. If a whole run fails in a couple of seconds, this is the first thing to check. ## How to use it The agent works best when you let it close its own loop rather than approving each step. A productive session looks like: - 1Point it at a test"Automate the checkout test in the Payments folder." It finds the project, the folder, and the test on its own. - 2Let it iterateWith the local server it can execute without a runId, which leaves no trace in any run. That is the point: the develop-run-develop loop should not pollute your history. - 3Make it check its own work"Run analyze_test_quality and fix what it finds." The scorecard is deterministic, so this is a real check rather than the agent grading itself. - 4Then execute for realPass a runId when you want the result recorded, or queue the whole run with run_on_runner. ## The tools Forty tools, grouped by what they touch. Each one is documented with its parameters and what it is for in the tool reference. - Projects, folders and tags: finding your way around. - Tests: reading, creating, updating, archiving, and quality analysis. - Reusable actions: the shared blocks and how tests attach to them. - Environments and variables: what execution needs, without exposing secrets. - Runs: creating them, recording results, closing them. - Execution: locally, or queued onto a runner. ## If something is not working - No tools appear: the client did not connect. Restart it, and check the key is in env and not in args. - Everything returns unauthorized: the key was revoked, or belongs to a different instance than SMARTT_API_BASE_URL points at. - Execution tools are missing: you are on the remote connector. They only exist locally; use run_on_runner. - Runs fail in seconds with a path error: ffmpeg is not installed. See above. PreviousUsers & RolesNextMCP tool reference --- Source: https://www.smarttest.cloud/docs/mcp/tools Automation # MCP tool reference All forty tools, grouped by what they touch. Setup instructions are in SmarTT MCP. ## Remote or local Most tools work on both the hosted connector and the local server. Four exist only locally, because they drive a real browser and the server does not have one. Note On the remote connector those four are not listed at all. The remote path to execution is `run_on_runner`, which hands the work to a machine that has a browser. See SmarTT Runner CLI. ## Projects & folders Where an agent starts. Almost every other tool needs a projectId, and this is where it comes from. ### list_projects remote + local Lists the projects your organization can reach, with the ids every other tool needs. Parameters Parameter | Type | Description --- | --- | --- includeInactive | booleandefault false | Include deactivated projects. They are read-only, so writes to them will be refused. Use it when - First call of almost any session, because nothing else works without a project id. - Confirming which project a request refers to when the user says "the checkout tests" and there are three projects. ### list_project_folders remote + local Every active folder in a project, each with its full path so the tree can be reconstructed. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Finding the folder id needed to create a test. - Understanding how a team organizes its library before adding to it. - Deciding where a new test belongs, rather than dropping it in the first folder found. On a large library this returns a lot of text at once, and it is not paginated. Ask for the specific area you need rather than the whole tree when the project is big. ### create_folder remote + local Creates a root folder, or a subfolder under an existing one. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. name (required) | string | Maximum 30 characters. parentId | string | Omit for a root folder. Otherwise the folder this one nests under. Use it when - Setting up structure for a new feature area before writing its tests. - Splitting a folder that has grown past the point of being navigable. ## Tags Cross-cutting labels. Note that tests are tagged by name through create_test and update_test, so these two are mostly for discovery and cleanup. ### list_tags remote + local The tags in a project, each with a count of how many active tests carry it. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Checking what conventions already exist before inventing a new tag. - Finding the right tag name to scope a run to. - Spotting tags with a count of one, usually a typo of a real tag. ### create_tag remote + local Creates a tag in a writable project. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. name (required) | string | Short and lowercase works best. description | string | What qualifies a test for this tag. Use it when - Introducing a genuinely new category, after checking list_tags for something equivalent. A name only has to be free within the target project, so the same tag can exist in several projects. Creating one that the project already has is refused rather than silently reused. ## Tests The core of the API. Reading, writing, archiving, and checking the quality of automation. ### list_tests remote + local Lightweight metadata for the tests in a project, enough to choose one, without pulling every step. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. folderId | string | Restrict to one folder. recursive | booleandefault true | Whether subfolders are included when folderId is set. includeArchived | booleandefault false | Include archived tests. Needed to find something to restore. Use it when - Finding a test by title before reading it in full. - Surveying what is automated and what is not across an area. - Locating an archived test so it can be restored. ### get_test remote + local The complete test: steps, expected result, automation plan, tags, and attached reusable preconditions. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. includeArchived | booleandefault true | Whether an archived test can be returned. Use it when - Reading the written steps before writing automation for them. - Inspecting an existing automation plan before changing it. - Checking which reusable preconditions are attached, and in what order. ### create_test remote + local Creates a test inside an existing folder. Written steps and automation can both be supplied at once. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. folderId (required) | string | Must already exist. title (required) | string | What the test verifies, in one line. priority | LOW | MEDIUM | HIGH | CRITICAL | Defaults if omitted. description | string | Context the reader needs. preconditions | string | Written state required before step 1. steps (required) | step[] | At least one. Each may carry automation. expectedResult | string | What must be true at the end. expectedResultAutomation | action[] | Commands that verify the expected result, normally the assertions. expectedResultAttachments | string[] | URLs of supporting images. references | string | Tickets, requirement ids. automated | No | Yes | Impossible | Discardeddefault "No" | Only these four values. Anything else is rejected. automationStatus | Completed | In Progress | Needs Attention | Broken | Only allowed when `automated` is `Yes`. automatedRejectReason | string | Why the test was not automated. Only for `Impossible` and `Discarded`. tagNames | string[] | By name, not id. Tags must already exist. Use it when - Turning a requirement or a bug report into a written test. - Creating a test and its automation together, in one call. - Bulk-creating a set of cases for a new feature. The four `automated` values are exact. `"yes"`, `"true"`, or `"automated"` are all rejected rather than coerced. ### update_test remote + local Updates a test in place. Every field is optional: only what you pass changes. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. folderId | string | Move the test to a different folder. title | string | priority | string | description | string | null | Pass null to clear. preconditions | string | null | Pass null to clear. steps | step[] | Replaces the whole list. It is not a merge. expectedResult | string | null | expectedResultAutomation | action[] | null | Replaces the verification actions entirely. references | string | null | automated | No | Yes | Impossible | Discarded | automationStatus | string | null | Only when automated is Yes. automatedRejectReason | string | null | Only for Impossible or Discarded. addTagNames | string[] | Tags to add, by name. removeTagNames | string[] | Tags to remove, by name. Use it when - Attaching automation to a test that was written manually. - Marking a test Broken after the UI changed under it. - Adding tags in bulk when a new suite is defined. - Moving a test to the folder it should have been in. `steps` replaces the entire list. To change one step, read the test first, modify the array, and send all of it back. ### list_automation_commands remote + local The whole automation vocabulary: every command with the exact input keys it accepts, their types, and which are required. Parameters Parameter | Type | Description --- | --- | --- command | string | Return only this command. Omit it to get all twenty-six. Use it when - Before writing any automation plan. It is the difference between knowing a command exists and knowing what to put in it. - After a rejected save, to see what the command actually accepts instead of guessing again. Keys outside this list are rejected when the test is saved, and the error names the valid ones. Nothing is silently ignored, so there is no reason to guess. ### analyze_test_quality remote + local Runs the deterministic Quality Engine on a test's automation. Returns a 0-100 scorecard with findings and concrete suggestions. No AI, no cost. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. Use it when - Immediately after writing automation. This is the single most valuable tool for an agent, because it turns "looks right" into a checkable result. - Auditing an existing suite to find the tests worth fixing first. - Confirming a fix actually improved things rather than moving the problem. It checks assertions, locator brittleness, hard waits, hardcoded credentials, unused runtime variables, and steps with no automation behind them. The findings name the step, so they can be acted on directly. ### archive_test remote + local Soft-deletes a test. It leaves the tree but keeps its history and every past run result. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. Use it when - Retiring a test for a feature that no longer exists. - Removing a duplicate, once you have confirmed which one to keep. ### restore_test remote + local Brings an archived test back, and records the restoration in its history. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. Use it when - Undoing an archive. - Recovering a test after a folder was deleted. Find it first with list_tests and includeArchived: true. ## Reusable actions Shared blocks of steps, and how tests attach to them. Editing a block changes every test that uses it. See Reusable Actions for the model. ### list_reusable_actions remote + local The blocks defined for a project, each with the tests that currently use it. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Checking whether a login block already exists before writing another one. - Seeing the blast radius of a change before making it. - Finding unused blocks worth deleting. ### get_reusable_action remote + local The full detail of one block, including its steps and every test linked to it. Parameters Parameter | Type | Description --- | --- | --- reusableActionId (required) | string | From list_reusable_actions. Use it when - Reading a block's steps before modifying them. - Understanding what a precondition actually does when debugging a failing test. ### create_reusable_action remote + local Creates a shared block of steps in a project. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. name (required) | string | Describe the outcome: "Log in as admin", not "Fill form". description | string | What state the system is in once it finishes. usableAs | PRECONDITION | STEP | BOTHdefault "PRECONDITION" | Where the block may be used. steps (required) | step[] | At least one. Use it when - Extracting a sequence that has been copied into several tests. - Defining login once, at the start of automating a project. A block cannot contain `smartt_run_reusable_action`. Blocks do not nest. If two share steps, inline them. ### update_reusable_action remote + local Changes a block. Every test that uses it is affected immediately. Parameters Parameter | Type | Description --- | --- | --- reusableActionId (required) | string | name | string | description | string | null | usableAs | PRECONDITION | STEP | BOTH | steps | step[] | Replaces the whole list. Use it when - Fixing every test at once after a login form changed. - Making a block more robust, replacing a CSS selector with a role-and-name locator. ### delete_reusable_action remote + local Deletes a block. Parameters Parameter | Type | Description --- | --- | --- reusableActionId (required) | string | Use it when - Removing a block nothing uses any more. Refused while tests are still attached. Detach them first. ### attach_reusable_action_to_test_preconditions remote + local Attaches a block to a test as a precondition. It will run before the test's own first step. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. reusableActionId (required) | string | Must be in the same project. Use it when - Giving a new test the standard login setup. - Replacing steps that were inlined into a test with the shared block, usually right after analyze_test_quality flags the duplication. ### detach_reusable_action_from_test_preconditions remote + local Removes one precondition link from a test. The block itself is untouched. Parameters Parameter | Type | Description --- | --- | --- linkId (required) | string | The link id, from get_test, not the reusable action id. Use it when - A test that no longer needs that setup. - Clearing links before deleting a block. ### reorder_test_reusable_preconditions remote + local Sets the order in which a test's preconditions run. They execute top to bottom. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. orderedLinkIds (required) | string[] | Every attached link id, exactly once, in the order you want. Use it when - Fixing setup that runs in the wrong order: selecting an account before logging in, for instance. - Putting a newly attached precondition in the right place instead of last. Every currently attached link must appear exactly once. A partial list is rejected. ## Environments & variables What an execution needs. These tools tell an agent which keys exist and where they are defined, never what a secret contains. ### get_execution_settings remote + local The browser and resolution configured for a project. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Reporting how tests will run, without guessing. - Deciding whether to override the browser for one execution. `execute_test` and `execute_run` already apply these. You only need this tool to show or reason about them. ### list_environments remote + local The execution environments available to your organization. Parameters Takes no parameters. Use it when - Getting the environmentId that every execution tool requires. - Confirming which target the user means before running anything against it. ### get_org_variables remote + local Organization-level variable keys. Secret values are never returned. Parameters Takes no parameters. Use it when - Discovering what is available to reference as {{orgVar.KEY}} before writing automation. - Checking a base URL is defined before writing a test that depends on it. ### get_project_variables remote + local Project-level variable keys. Secret values are never returned. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Finding what to reference as {{projectVar.KEY}}. - Understanding which values are project-specific rather than shared. ### get_test_variables remote + local Variable keys attached to a single test. Secret values are never returned. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. Use it when - Checking what a test defines for itself as {{testVar.KEY}}. - Diagnosing a test that fails only in one environment. ### resolve_test_execution_variables remote + local Resolves the actual values a test would receive in a given environment. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. environmentId (required) | string | From list_environments. Use it when - Confirming a test will get the values it expects before running it. - Working out why the same test passes against staging and fails against production. Usually it is a variable defined for one environment and not the other. ## Runs Creating runs, recording results, and closing them. A run stays open until complete_run is called. ### list_runs remote + local Runs for a project, newest first, with summary counts and status. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. limit | numberdefault 50 | Between 1 and 200. Use it when - Finding the run a user is referring to. - Checking whether an open run already exists before creating another. - Summarising recent testing activity. ### get_run remote + local One run with every test result in it: status, comment, duration, and recording. Parameters Parameter | Type | Description --- | --- | --- runId (required) | string | From list_runs. Use it when - Reading what failed and why, in order to act on it. - Watching results land while a runner executes. - Producing a report from a completed run. ### create_run remote + local Creates a run, optionally scoped to folders, tags, or specific tests. A result slot is created for every test in scope. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. name (required) | string | What this run is verifying. includeTimestamp | booleandefault false | Append a date and time to the name, the same option the interface offers. comment | string | What this run is for. Worth filling in. folderIds | string[] | Include everything in these folders. tagNames | string[] | Include every test carrying these tags. testCaseIds | string[] | Include these specific tests. Use it when - Building a smoke run from a tag before a release. - Creating a targeted run for the area a change touched. - Setting up a run to hand straight to a runner. With no scope at all, every test in the project is included. The interface asks you to pick something instead, so an agent creating an unscoped run is making a choice a person would have been stopped from making by accident. ### update_run_test_result remote + local Sets the result of one test inside a run. A comment is always required. Parameters Parameter | Type | Description --- | --- | --- runId (required) | string | testCaseId (required) | string | From list_tests or get_test. status (required) | PENDING | IN_PROGRESS | PASSED | FAILED | SKIPPED | Case-insensitive. comment (required) | string | Cannot be empty. videoUrl | url | Recording of the execution. Left by the runner. executionId | string | Which runner execution produced this. durationMs | number | How long it took. Use it when - Recording the outcome of a test executed outside SmarTTest. - Marking a test skipped, with the reason it was out of scope. - Correcting a result after investigating a failure. The comment is not optional by accident. A failure with no note is a failure nobody can act on later. ### complete_run remote + local Closes a run. SmarTTest reads through the results and appends a SmarTT Summary of what happened. Parameters Parameter | Type | Description --- | --- | --- runId (required) | string | comment (required) | string | The closing note. Cannot be empty. Use it when - Finishing a run once every test has a result. - Closing a run so it starts counting toward the Health Dashboard. Only completed runs feed pass-rate trends, so leaving runs open quietly loses data. ## Execution Two paths. Queue the work onto a runner, which works from anywhere, or execute it directly, which only the local server can do. ### list_runners remote + local The runners registered for a project: the machines that can execute its automated tests. Parameters Parameter | Type | Description --- | --- | --- projectId (required) | string | From list_projects. Use it when - Getting the runnerId for run_on_runner. - Checking one is online before queueing work, so the user is not left waiting on a stopped machine. A runner not seen in the last 60 seconds is offline and will not pick up work until it returns. ### run_on_runner remote + local Queues an existing run for a runner to execute. Returns immediately; the work happens on the runner's machine. Parameters Parameter | Type | Description --- | --- | --- runId (required) | string | The run to execute. runnerId (required) | string | From list_runners. environmentId (required) | string | From list_environments. concurrency | 1 – 8default 1 | How many tests at once. Raise it only when the tests are genuinely independent. only | all | pending | faileddefault "all" | Which results to execute. failed is the usual second pass. headless | booleandefault true | Whether the browser window is hidden on the runner machine. Use it when - The only way to execute tests from the remote connector. - Running a full regression suite on a dedicated machine instead of your laptop. - Re-running just the failures after a fix. Only automated tests are executed; manual ones are skipped. A run can have one job queued or running at a time. Poll `get_runner_job`, or read the run to watch results land. ### get_runner_job remote + local The progress of a job queued with run_on_runner. Parameters Parameter | Type | Description --- | --- | --- jobId (required) | string | Returned by run_on_runner. Use it when - Polling until execution finishes. - Distinguishing "the runner never picked this up" from "the tests are failing". The job carries its own error separate from the results. ### validate_automation local server only Opens a page and checks whether locators resolve, without running a test or creating a run. Parameters Parameter | Type | Description --- | --- | --- url (required) | string | The page to open. Resolve any variable placeholder yourself first. locators (required) | object[] | The locators to check, each the same input object an action would use. headless | booleandefault true | Whether the browser window is hidden. Use it when - Before asking anyone to execute a run. A locator that matches four elements fails the whole test, and this finds that out in one step instead of one execution. - When a test fails and you cannot tell whether the element moved or the page did. The reply includes the final URL and title, which is how a redirect gives itself away. For anything ambiguous it returns how many elements matched and the text of the first few, so you can tell them apart. It creates no run and records nothing. ### execute_test local server only Compiles and executes one test's automation plan, in a real browser, on this machine. Parameters Parameter | Type | Description --- | --- | --- testCaseId (required) | string | From list_tests or get_test. environmentId (required) | string | From list_environments. headless | booleandefault true | Whether the window is visible. stopOnFailure | booleandefault true | Stop at the first failing step, rather than carrying on. timeoutMs | numberdefault 180000 | Overall limit, from 1s to 15 minutes. runId | string | Log this result into an existing run. Omit it while iterating, and the execution then leaves no trace anywhere. recordVideo | boolean | Defaults to true when runId is set, false otherwise. videoWidth | numberdefault 1280 | Viewport and recording width. videoHeight | numberdefault 720 | Viewport and recording height. browserChannel | chromium | chrome | msedge | Which browser to drive. Defaults to the Chromium bundled with Playwright. Use it when - The develop-run-develop loop: write automation, run it, see the failure, fix it, without polluting any run with attempts. - Reproducing a failure locally with a visible browser to see what is happening. - Recording a real result, by passing runId. Omitting `runId` is the important detail. Without it the execution is recorded nowhere, which is exactly what you want while iterating. ### get_test_execution local server only The current state of an execution started with execute_test. Parameters Parameter | Type | Description --- | --- | --- executionId (required) | string | From execute_test. Use it when - Polling a long execution. - Reading which step failed, and the error it produced. ### execute_run local server only Executes every automated test in a run on this machine, logging each result back into it. Returns immediately. Parameters Parameter | Type | Description --- | --- | --- runId (required) | string | environmentId (required) | string | concurrency | 1 – 8default 1 | Each test gets its own browser. Only raise it for independent tests. only | all | pending | faileddefault "all" | Which results to run. headless | boolean | recordVideo | boolean | timeoutMs | number | Per test. videoWidth | number | videoHeight | number | browserChannel | chromium | chrome | msedge | Use it when - Executing a whole suite locally, without setting up a runner. - Re-running the failures from an earlier pass on your own machine. Non-automated tests are skipped and reported as such. Raising concurrency when tests share a login or a fixture makes them fail in ways that look like product bugs. ### get_run_execution local server only The progress of a run execution started with execute_run, including the state of every test in it. Parameters Parameter | Type | Description --- | --- | --- runExecutionId (required) | string | From execute_run. Use it when - Polling until the suite finishes. - Reporting progress while a long run is in flight. PreviousSmarTT MCPNextSmarTT Runner CLI --- Source: https://www.smarttest.cloud/docs/cli Automation # SmarTT Runner CLI A small program that runs on your machine, asks SmarTTest for work, executes the automated tests in a real browser, and reports the results back. ## Why it exists The applications you test usually are not reachable from the internet: staging behind a VPN, a local build, an internal tool. The runner puts the browser inside the network where those applications already live. The runner solves both problems by working outwards. It polls SmarTTest for work: every five seconds when idle, every three while executing a job. Nothing ever connects into it, so it needs no open port, no firewall rule, and no public address. If the machine can reach the internet and your application, it can run your tests. Note This is the same model GitHub self-hosted runners use, for the same reason. ## The package @smarttest/runner on npm·Apache-2.0·command `smarttest-runner` Node 18 or newer. Windows, macOS, and Linux. ## Installing - 1Create a runner in SmarTTestExecution Settings → Runners → New runner. Name it after the machine. You get a token, shown exactly once. Copy it now, because only a hash is stored. - 2Save the token on the machineterminal ``` npx @smarttest/runner login --token ``` This verifies the token against the server and writes it to ~/.smarttest/runner.json. You only do this once. - 3Install the browser and the encoderterminal ``` npx @smarttest/runner install ``` Downloads Chromium and ffmpeg. This also runs automatically on install, so you usually only need it to repair a broken setup. - 4Start itterminal ``` npx @smarttest/runner start ``` It shows as Online in the app and begins picking up work. For a machine that runs permanently, install it globally instead of through `npx`: terminal ``` npm install -g @smarttest/runner smarttest-runner start ``` Once started it reports the machine it is on, and waits: terminal ``` SmarTTest runner v0.1.2 machine: build-01 (win32) server: https://www.smarttest.cloud Waiting for work. Press Ctrl+C to stop. ``` ## Commands ### login ``` smarttest-runner login --token [--url ] ``` Verifies the token and saves it. Run it once per machine, or again to point the runner at a different SmarTTest instance. Note Verifying deliberately does not claim work. An earlier version used the same call the runner uses to pick up jobs, which meant logging in could quietly steal a job and leave it running with nothing executing it. ### start ``` smarttest-runner start [--token ] [--url ] ``` The main loop. Asks for work, executes it, reports back, repeats. Leave it running in a terminal, a service, or a container. - Ctrl+C finishes the job in flight, then stops. - Ctrl+C twice stops immediately. - If the server is unreachable it says so once and keeps retrying, instead of filling the terminal with the same error. ### stop ``` smarttest-runner stop [--force] ``` Stops a runner started from another terminal, and closes any browser windows left behind by an execution that crashed. Without `--force` it lets the current job finish first. Tip It only kills browsers Playwright launched for testing. Your own Chrome, and any other process, is left alone. ### status ``` smarttest-runner status ``` Shows the saved configuration: which server it points at and where the config file lives. The token is not printed. ### install ``` smarttest-runner install ``` Downloads Chromium and ffmpeg. Run it if executions start failing with a missing-binary error. ## Configuration Settings are resolved in order: command-line flag, then environment variable, then the saved config file. Parameter | Type | Description --- | --- | --- --token / SMARTT_RUNNER_TOKEN (required) | string | The runner token, created in Execution Settings → Runners. --url / SMARTT_API_BASE_URL | urldefault https://www.smarttest.cloud | Only needed for a self-hosted instance. --force | flag | With stop, kill immediately instead of finishing the current job. The saved config lives at `~/.smarttest/runner.json`. For CI or a container Skip `login` entirely and pass `SMARTT_RUNNER_TOKEN` as an environment variable. There is then no state on disk to manage. ## What a runner token can reach The token is scoped tightly on purpose. A runner can read the tests it has been asked to execute and write back their results. It cannot create or edit tests, browse your library, or reach another project. That matters because a runner often lives on a shared build machine. If the token leaks, what it grants is the ability to run tests and post results, not access to your test estate. Heads up Revoking a runner in the app kills its token immediately and cancels anything queued for it. A runner still polling will start reporting that it cannot reach the server. ## Sending it work Two ways, both covered elsewhere: - From the app: open a run and press SmarTT Execute. See Test Runs. - From an agent: run_on_runner, then poll get_runner_job. See the tool reference. Either way only automated tests are executed. Manual ones are skipped and stay yours to fill in. ## Troubleshooting ### The runner shows as Offline It has not been seen for over 60 seconds. Check the process is still running, and that the machine can reach the server. Jobs queued for it are not lost; they wait until it comes back. ### Every test fails in a couple of seconds Almost always a missing binary rather than a broken test, ffmpeg in particular. Run `smarttest-runner install`. SmarTTest reports this kind of failure as an environment problem, so it is not mistaken for the tests actually failing. ### Browsers left open after a crash `smarttest-runner stop` closes them. ### Could not reach the server Either the machine lost connectivity, or the token was revoked. Check the runner still exists in Execution Settings → Runners. ### Tests interfere with each other Lower Tests at once to 1. Concurrency only works when tests do not share a login, a fixture, or account state. PreviousMCP tool reference