Home Docs Test Automation Running Functional Tests

Running Functional Tests

1. TestRunner description of parameters

Running functional tests from the command-line is straightforward using the included testrunner.bat/.sh script, which takes a number of arguments to control which tests to run, output, etc:

  • a : Turns on exporting of all test results, not only errors
  • A : Turns on exporting of all results using folders instead of long
  • c : The TestCase to run, used to narrow down the tests to run
  • D : Sets system property with name=value
  • d : The domain to use in any authentications, overrides any domain set for any TestRequests
  • e : The endpoint to use when invoking test-requests, overrides the endpoint set in the project file
  • E : Sets which environment to use (ReadyAPI only)
  • F : Sets the format of the report specified with the -R option, for Printable reports this is one of PDF, XLS, HTML, RTF, CSV, TXT, and XML. For Data Export this is either XML or CSV (ReadyAPI only)
  • f : Specifies the root folder to which test results should be exported (see below)
  • G : Sets global property with name=value
  • g : Sets the output to include Coverage HTML reports ( ReadyAPI only )
  • h : The host:port to use when invoking test-requests, overrides only the host part of the endpoint set in the project file
  • I : Do not stop if error occurs, ignore them: Execution does not stop if error occurs, but no detailed information about errors are stored to the log. (If you need full information about errors, do not use this option). 
  • i : Enables SoapUI UI-related components, required if you use the UISupport class for prompting or displaying information
  • j : Turns on exporting of JUnit-compatible reports, see below
  • M : Creates a Test Run Log Report in XML format
  • m : Sets the maximum number of TestStep errors to save for each
  • o : Opens the generated report in a browser (ReadyAPI only)
  • P : Sets project property with name=value, e.g. -Pendpoint=Value1 -PsomeOtherProperty=value2
  • p : The password to use in any authentications, overrides any password set for any TestRequests
  • r : Turns on printing of a small summary report (see below)
  • R : Selects which report to generate for the test objects executed, for example if running the entire project, this could specify the name of a test-suite-level report that would be generated for each TestSuite. The report is saved as specified with the -F option to the folder specified with the -f option. (ReadyAPI only)
  • S : Sets to save the project file after tests have been run
  • s : The TestSuite to run, used to narrow down the tests to run
  • t : Sets the soapui-settings.xml file to use, required if you have custom proxy, ssl, http, etc setting
  • u : The username to use in any authentications, overrides any username set for any TestRequests
  • v : Sets password for soapui-settings.xml file
  • w : Sets the WSS password type, either 'Text' or 'Digest'
  • x : Sets project password for decryption if project is encrypted

So for example:

testrunner.bat -FPDF -R"Project Report" c:\projects\my-soapui-project.xml

Runs all tests in the specified my-soapui-project.xml file and creates the default project report available with ReadyAPI.

2. Launching the TestRunner from within SoapUI

For getting started with the command line testrunner, SoapUI includes a "Launch TestRunner" action available from Project, TestSuite or TestCase popup menus, which launches the bundled command-line tools from inside SoapUI. (The IDE plugins do not include these runners, you will need to download/install SoapUI separately and point the "TestRunner Path" option in this dialog to the install directory). The dialog looks as follows:

launch-test-runner-basic

Here we've specified some initial settings to run the current TestCase, in the Report tab we can specify which reports to generate:

The Reports tab of the Launch TestTunner dialog

Now if we run the testrunner we get:

TestRunner execution finished

Scrolling back up in the opened window we can see the actual command issued at the command-line:

The Launch TestRunner command

Copy and paste this into your favorite automation tool for rerun these tests as configured.

TestEngine: An Optimized Test Runner

Execute tests from SoupUI Open Source or SoupUI Pro in parallel to speed up your API Testing.

  • Scale Based on Your Business Needs:

    Dynamically run larger regression tests alongside smaller tests with custom configuration parameters.

  • Job Queuing:

    Execute tests as they arrive, staggering test runs and avoiding bottlenecks at peak times.

  • Supports Multiple Environments:

    Use TestEngine on Docker, VMS, Mac, Windows or Linux.

Learn More About TestEngine

3. Exporting parsable results from SoapUI

When integrating SoapUI's testrunner with Continuous Integration (CI) you can parse the results of an automated test run in one of the following ways:

  1. You can use the "-r" option of testrunner script to output the results to the console and parse the results from there
  2. You can generate a JUnit style report that will create an HTML and XML report files you can parse
  3. You can also export your results in CSV or XML format to parse the results in your CI tool

No one knows APIs better than SmartBear. Find out what our Pro version of SoapUI can do to improve your testing.