3. The Command-Line Test Runner
The PHPUnit command-line test runner can be invoked through the phpunit command.
Running tests
The most common way to run tests is with a phpunit.xml configuration file (see Composing a Test Suite Using XML Configuration). When a configuration file is present in the current working directory, simply running phpunit without any arguments will execute all configured test suites:
$ ./tools/phpunit PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 Configuration: /path/to/project/phpunit.xml .. 2 / 2 (100%) Time: 00:00.077, Memory: 10.00 MB OK (2 tests, 2 assertions)
You can also run the tests of a specific test source file by passing its path as an argument:
$ ./tools/phpunit tests/ExampleTest.php PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 .. 2 / 2 (100%) Time: 00:00.077, Memory: 10.00 MB OK (2 tests, 2 assertions)
When invoked as shown above, the PHPUnit command-line test runner will look for a ExampleTest.php source code file in the current working directory, load it, and expect to find an ExampleTest test case class. It will then run the tests found in that class.
PHPT tests
In addition to test classes, PHPUnit’s test runner can also execute PHPT tests.
PHPT is a simple test format used by the PHP project itself.
Files with the .phpt suffix are automatically discovered by the test runner (see the --test-suffix option in Selection).
This can be useful, for instance, for end-to-end tests of CLI tools or for getting basic regression tests in place for legacy codebases.
Selecting tests
PHPUnit provides several command-line options for selecting which tests to run.
Filtering by test name
The --filter option selects tests whose name matches a pattern:
$ ./tools/phpunit --filter testDoesSomething PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 Configuration: /path/to/project/phpunit.xml . 1 / 1 (100%) Time: 00:00.077, Memory: 10.00 MB OK (1 test, 1 assertion)
The pattern is matched against the full test name, which has one of these forms:
Fully\Qualified\ClassName::testMethodName Fully\Qualified\ClassName::testMethodName with data set #0 Fully\Qualified\ClassName::testMethodName with data set "named set"
Matching is a partial match (the pattern does not need to cover the entire name) and not anchored. For example, --filter testAdd also matches a test named testAddress.
PHPT tests (see PHPT tests) are never selected by --filter.
The --exclude-filter option accepts the same pattern syntax described below and removes matching tests from the run. It can be combined with --filter.
Shortcut syntax
For common cases you do not need to write a regular expression. The following shortcut forms are recognized and translated into an appropriate pattern internally:
testMethodNameSelects all tests whose name contains
testMethodName.ClassName::testMethodNameSelects a specific test method in a specific class.
testMethodName#3Selects the test method together with data set
#3(a numeric data set key).testMethodName#1-3Selects the test method together with numeric data sets in the inclusive range
1to3.#1-3Selects numeric data sets in the inclusive range
1to3across all test methods.testMethodName@one plus oneSelects the test method together with the named data set
"one plus one". Quote the argument so the shell preserves spaces.testMethodName@one.*Selects the test method together with any named data set that matches the regular expression
one.*.
When the shortcut form is used, matching is case-insensitive.
Regular expression syntax
If the pattern begins with a non-alphanumeric character and is a valid PCRE regular expression, it is used verbatim (including its delimiters and modifiers). This form is useful when the shortcut syntax is not expressive enough, or when you need case-sensitive matching:
$ ./tools/phpunit --filter '/^App\\Domain\\.*Test::test/'
Caution
Fully-qualified class names contain backslashes, and backslashes are special in regular expressions. If you want to use --filter with a fully-qualified class name, either escape every backslash (App\\\\Domain\\\\FooTest) or wrap the pattern with explicit delimiters (/App\\\\Domain\\\\FooTest/). The filter is not passed through preg_quote(), so all regular expression metacharacters remain active.
Filtering by test suite
When you have multiple test suites defined in your XML configuration file, you can use the --testsuite option to run only the tests of a specific suite:
$ ./tools/phpunit --testsuite unit PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 Configuration: /path/to/project/phpunit.xml ............... 15 / 15 (100%) Time: 00:00.100, Memory: 10.00 MB OK (15 tests, 30 assertions)
The --exclude-testsuite option excludes a named test suite from the run.
The --list-suites option prints a list of all test suites defined in the XML configuration file without running any tests.
Filtering by group
The --group option runs only the tests that belong to a specified group. Groups are assigned to tests using the #[Group] attribute (see Group).
The --exclude-group option excludes tests belonging to a specified group.
Filtering by covered unit
The --covers option runs only tests that declare they intend to cover a specified class or function. The --uses option runs only tests that declare they intend to use a specified class or function.
Listing tests
The --list-tests option prints a list of all tests that would be executed without actually running them. This is useful for verifying that your selection options are working as expected.
The --list-tests-xml option writes this list in XML format to a specified file.
See Selection for the complete reference of all test selection options.
Outcome and Issues
PHPUnit separates the outcome (errored, failed, incomplete, skipped, or passed) of a test from the issues (considered risky, triggered a warning, …) of a test.
With regard to outcome, PHPUnit distinguishes between failures and errors. A test fails when an assertion failed. This is different from an unexpected exception or a PHP error that occur while a test is running. When this happens, the test errors.
Errors tend to be easier to fix than failures. If you have a big list of problems, it is best to tackle the errors first and see if you have any failures left when they are all fixed.
Output
The default output of PHPUnit’s test runner consists of up to four sections.
Runtime information
This section contains information about the PHPUnit version, PHP version, and PHPUnit’s XML configuration file.
This section can be disabled using the --no-output CLI option.
Progress
This section is printed to indicate progress while the tests are being run.
For each test run, the PHPUnit command-line tool prints one character to indicate progress:
.
Printed when a successful test has no issues
F
Printed when an assertion fails while running the test method
E
Printed when an error occurs while running the test method
W
Printed when the test triggered a warning
R
Printed when the test has been considered risky (see Risky Tests)
D
Printed when the test triggered a deprecation
N
Printed when the test triggered a notice
I
Printed when the test is marked as incomplete (see Incomplete Tests)
S
Printed when the test was skipped (see Skipping tests)
This section can be disabled using the --no-progress and --no-output CLI options.
Test results
This section contains information about errors, failures, skipped tests, incomplete tests, and issues. By default, only information about errors and failures is printed.
This section is only printed when there are errors, failures, or issues to report. It can be disabled using the --no-results and --no-output CLI options.
Failure output
Whenever a test fails, PHPUnit tries its best to provide you with as much context as possible that can help to identify the problem.
<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;
final class ArrayDiffTest extends TestCase
{
public function testEquality(): void
{
$this->assertSame(
[1, 2, 3, 4, 5, 6],
[1, 2, 33, 4, 5, 6],
);
}
}
Running the test shown above yields the output shown below:
./tools/phpunit tests/ArrayDiffTest.php
PHPUnit 12.5.2 by Sebastian Bergmann and contributors.
Runtime: PHP 8.5.4
F 1 / 1 (100%)
Time: 00:00, Memory: 25.89 MB
There was 1 failure:
1) ArrayDiffTest::testEquality
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
Array &0 [
0 => 1,
1 => 2,
- 2 => 3,
+ 2 => 33,
3 => 4,
4 => 5,
5 => 6,
]
/path/to/tests/ArrayDiffTest.php:8
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
In this example only one of the array values differs and the other values are shown to provide context on where the error occurred.
When the generated output would be long to read PHPUnit will split it up and provide a few lines of context around every difference.
<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;
final class LongArrayDiffTest extends TestCase
{
public function testEquality(): void
{
$this->assertSame(
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 33, 4, 5, 6],
);
}
}
Running the test shown above yields the output shown below:
./tools/phpunit tests/LongArrayDiffTest.php
PHPUnit 12.5.2 by Sebastian Bergmann and contributors.
Runtime: PHP 8.5.4
F 1 / 1 (100%)
Time: 00:00, Memory: 25.89 MB
There was 1 failure:
1) LongArrayDiffTest::testEquality
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
11 => 0,
12 => 1,
13 => 2,
- 14 => 3,
+ 14 => 33,
15 => 4,
16 => 5,
17 => 6,
]
/path/to/tests/LongArrayDiffTest.php:8
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
Edge cases
When a comparison fails PHPUnit creates textual representations of the input values and compares those. Due to that implementation a diff might show more problems than actually exist.
This only happens when using assertEquals() or other “weak” comparison
functions on arrays or objects.
<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;
final class ArrayWeakComparisonTest extends TestCase
{
public function testEquality(): void
{
$this->assertEquals(
[1, 2, 3, 4, 5, 6],
['1', 2, 33, 4, 5, 6],
);
}
}
Running the test shown above yields the output shown below:
./tools/phpunit tests/ArrayWeakComparisonTest.php
PHPUnit 12.5.2 by Sebastian Bergmann and contributors.
Runtime: PHP 8.5.4
F 1 / 1 (100%)
Time: 00:00, Memory: 25.89 MB
There was 1 failure:
1) ArrayWeakComparisonTest::testEquality
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
Array (
- 0 => 1
+ 0 => '1'
1 => 2
- 2 => 3
+ 2 => 33
3 => 4
4 => 5
5 => 6
)
/path/to/tests/ArrayWeakComparisonTest.php:8
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
In this example the difference in the first index between
1 and '1' is
reported even though assertEquals() considers the values as a match.
Summary
This section contains a summary of the test suite execution.
When no test errored or failed then the summary will be
OKWhen a test errored then the summary will be
ERRORED!When a test failed and no test errored then the summary will be
FAILED!When there were no errors or failures, but a test was skipped, then the summary will be
OK, but some tests were skipped!When there were no errors, failures, or skipped tests, but there were issues, then the summary will be
OK, but there were issues!
--fail-on-*
Please note that if you use CLI options such as --fail-on-deprecation, for example,
or their XML configuration counterparts, then this only affects the test runner’s shell
exit code. It does not have an effect on whether or not OK will be printed in the
output’s summary section.
This section can be disabled using the --no-output CLI option.
Alternative output formats
PHPUnit supports alternative output formats that replace the default progress and result output.
TestDox
The --testdox option replaces the default result output with TestDox format.
See TestDox for details.
Debug
The --debug option replaces the default progress and result output with detailed event-by-event information about the test execution process. When combined with --with-telemetry, timing and memory usage information is included.
TeamCity
The --teamcity option replaces the default progress and result output with TeamCity format. This is used for integration with PhpStorm or TeamCity.
Controlling output
The following options control what is displayed in the test results section. By default, only information about errors and failures is printed. Use these options to display additional details:
--display-incompleteshows details for incomplete tests--display-skippedshows details for skipped tests--display-deprecationsshows details for deprecations triggered by tests--display-noticesshows details for notices triggered by tests--display-warningsshows details for warnings triggered by tests--display-errorsshows details for errors triggered by tests--display-all-issuesenables all of the above
The --colors=<flag> option controls the use of colors in terminal output. Accepted values are never, auto (uses colors when the terminal supports it), and always.
The --columns option controls the width of the progress output. Use --columns max to use the full terminal width.
The --reverse-list option prints defects in reverse order, showing the most recently encountered defects first.
See Reporting for the complete reference of all reporting options.
Interaction between output options
The options described above fall into two groups:
--no-output,--no-progress, and--no-resultssuppress sections of the default output format described in Output.--testdox,--teamcity, and--debugselect an alternative output format that replaces the default progress and result output.
The --no-* options apply to the default output format only. They do not suppress the output produced by an alternative output format. For example, --testdox --no-output will still produce TestDox output.
The --testdox-html and --testdox-text options are logging options (see Logging): they write a TestDox-formatted report to a file. They are not intended to replace or augment the console output. Use --testdox when you want TestDox-formatted output on the console.
Exit codes
The PHPUnit command-line test runner exits with an exit code that indicates the outcome of the test run. This is particularly important for Continuous Integration (CI) pipelines and scripts that need to act on the result of a test run.
0
All tests passed (no errors, no failures)
1
At least one test failed
2
At least one test errored
By default, issues such as deprecations, notices, warnings, and risky tests do not affect the exit code. The --fail-on-* family of options changes this behaviour:
--fail-on-warningcauses a failure exit code when any test triggers a warning--fail-on-riskycauses a failure exit code when any test is considered risky--fail-on-deprecationcauses a failure exit code when any test triggers a deprecation--fail-on-noticecauses a failure exit code when any test triggers a notice--fail-on-skippedcauses a failure exit code when any test is skipped--fail-on-incompletecauses a failure exit code when any test is marked incomplete--fail-on-empty-test-suitecauses a failure exit code when no tests were executed--fail-on-all-issuesenables all of the above
These options are commonly used in CI pipelines to enforce strict quality standards:
$ ./tools/phpunit --fail-on-deprecation --fail-on-notice --fail-on-warning PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 Configuration: /path/to/project/phpunit.xml DW 2 / 2 (100%) Time: 00:00.077, Memory: 10.00 MB OK, but there were issues! Tests: 2, Assertions: 2, Deprecations: 1, Warnings: 1.
The --fail-on-* options have --do-not-fail-on-* counterparts that can be used to override settings from the XML configuration file.
See Execution for the complete reference.
Stopping early
When a test suite is large, it can be useful to stop execution as soon as a problem is encountered instead of waiting for the entire suite to finish.
The --stop-on-* family of options controls this:
--stop-on-defectstops after the first error, failure, warning, or risky test--stop-on-errorstops after the first error--stop-on-failurestops after the first failure--stop-on-warningstops after the first warning--stop-on-riskystops after the first risky test--stop-on-deprecationstops after the first deprecation--stop-on-noticestops after the first notice--stop-on-skippedstops after the first skipped test--stop-on-incompletestops after the first incomplete test
See Execution for the complete reference.
Logging
PHPUnit can write test results to log files in various formats:
--log-otr <file>writes test results in Open Test Reporting (OTR) XML format (recommended)--log-junit <file>writes test results in JUnit XML format (legacy)--log-teamcity <file>writes test results in TeamCity format (used by PhpStorm)--testdox-html <file>writes test results in TestDox format as an HTML file--testdox-text <file>writes test results in TestDox format as a plain text file--log-events-text <file>writes all test runner events as plain text--log-events-verbose-text <file>writes all test runner events with extended information
These logfiles are commonly used for CI tool integration and reporting.
The --testdox-html and --testdox-text options write a report to a file; they are not intended to replace or augment the console output. For TestDox-formatted console output use --testdox (see TestDox).
Logging can also be configured in the XML configuration file (see The <logging> Element).
The --no-logging option ignores all logging configured in the XML configuration file.
Test execution order
The --order-by option controls the order in which tests are executed. Supported values include:
default
Tests are executed in the order they are discovered
defects
Tests that failed in a previous run are executed first. This requires the result cache (enabled by default, see
--cache-result)
duration
Tests are ordered by duration, shortest first. This requires the result cache
random
Tests are executed in random order. Use
--random-order-seedto make the order reproducible
reverse
Tests are executed in reverse discovery order
size
Tests are ordered by size (small, medium, large, unknown)
depends
Tests are reordered to satisfy
#[Depends]declarations
no-depends
Tests are not reordered to satisfy
#[Depends]declarations
Multiple values can be combined: --order-by defects,random runs previously failing tests first, then executes the remaining tests in random order.
Convenience aliases are available: --resolve-dependencies (for --order-by depends), --ignore-dependencies (for --order-by no-depends), --random-order (for --order-by random), and --reverse-order (for --order-by reverse).
$ ./tools/phpunit --order-by random --random-order-seed 42 PHPUnit 12.5.0 by Sebastian Bergmann and contributors. Runtime: PHP 8.5.3 Configuration: /path/to/project/phpunit.xml Random seed: 42 .. 2 / 2 (100%) Time: 00:00.077, Memory: 10.00 MB OK (2 tests, 2 assertions)
See Execution for the complete reference.