Nette Tester 2.0.0 is out

by Miloslav Hůla 3 months ago in Tester

Nette Tester 2 is finally out. I'll describe changes and new features in detail.

Compatibility

New Tester requires PHP in version 5.6 or newer and supports PHP 7.2. Simultaneously, support for HHVM has been dropped. There has been small opinion poll on this topic.

Changes in tests runner

Tester's default interpreter (when called without the -p option) is CLI now. It used to be CGI. You will see notice in console about it.

When you pass directory as a tester argument, it will find all *.phpt files as usual, but newly it will find all *Test.php files additionally.

The Tester used to run tests in order as they have been found. Now, it will run last time failing tests as first. For this purpose, the Tester uses temporary directory. Default is got by sys_get_temp_dir() and you can change it by explicit --temp <path> option. The --info call will help you find out currently used directory. Try to think about this new feature in relation to --stop-on-fail or --watch options. It may improve your workflow.

As mentioned in documentation, every test is run by PHP in separated process with -n option. So without php.ini file. Sometimes it is painful to tune up own php.ini file for tests. A new option -C is introduced, it causes that -n is not used and as a consequence, tests run with system-wide PHP settings.

Sometimes is good to know whether test is running by tester myTest.phpt or as an ordinary PHP script by php myTest.phpt. An environment variable will help you to recognize it:

if (getenv(Tester\Environment::RUNNER)) {
    # by Tester
} else {
    # other way
}

And when talking about environment variables, there is another one. When tests run in parallel, by default in 8 threads, you can be interested in a current thread number.

$threadNumber = getenv(Tester\Environment::THREAD);

Changes in testing framework

New modifiers for Assert::match() have been added. The %w% matches alphanumeric chars 0-9, a-z, A-Z and _ (underscore). The %% matches one % char.

Behavior of FileMock has been fixed, mainly in an append mode.

Another change concerns the TestCase. Now, the TestCase::tearDown() run always, even aften an error occurred in a testing method. Small BC break is that the TestCase::run() method does not accept arguments anymore. If you need to run a single testing method, use the TestCase::runTest($method, array $args = []) method.

New test file annotation @phpExtension has been added. When some of given extensions is not loaded, test will be skipped. You can use one annotation with multiple extensions, or split it to multiple annotations. For example:

/**
 * @phpExtension pdo, pdo_pgsql
 * @phpExtension json
 */

Completely new helper FileMutator has been added. Its purpose is to adjust source code of loaded files. The first real use case is removing final keyword from classes and class methods. Usage is simple. Call the Tester\Environment::bypassFinals() on the very top of a test file, it will setup FileMutator for you.

require 'bootstrap.php';
Tester\Environment::bypassFinals();
# Now, you can extend final classes or overload final class methods.

Changes in output formating

When other than expected exception is thrown in the Assert::exception(), its stack trace will be printed. It helps to find out, what is wrong.

If you are using Assert::match() with modifiers (%a%, %d% and so on), you will be pleasured by a new feature. In case of fail, modifiers are replaced by their matching values in output and dumps. An example will say more. Imagine test such:

$tested = "My name is Milo and I'm -65535 years old.";
Assert::match("My name is %S% and I'm %d% years old.", $tested);
# An output before the Tester 2
    Failed: 'My name is Milo and I'm -65535 years old.' should match
        ... 'My name is %S% and I'm %d% years old.'

# An output with new Tester
    Failed: 'My name is Milo and I'm -65535 years old.' should match
        ... 'My name is Milo and I'm %d% years old.'

Do you see that small difference? The %S% modifier is now replaced by a matching value and you don't need to decrypt where mismatch occurred. Very handy when matching long strings.

If you use code coverage functionality with a PHPDBG, some tests may fail due to a memory exhaustion. Coverage data collecting is a memory consuming operation. You can increase memory limits, or you can call Tester\CodeCoverage\Collector::flush(). It will write collected data into file and release occupied memory. Calling has no effect with Xdebug or when collecting is not running at all.

HTML output of code coverage has been facelifted. Check out an example.

And here we come with the biggest BC break in the Tester 2. Changes in the OutputHandler interface. Before I'll describe it, I have to make a short introduction how tests are handled internally by the Tester.

Inside the Tester, single test is represented by Test class instance. In a short, it contains test file path, test title, result code, captured standard output and so on. Check out the source code for details. These instances are passed to output handlers in the Tester 2. Refactored OutputHandler now looks as follows:

namespace Tester\Runner;

interface OutputHandler
{
    # Before testing, something like init phase.
    function begin();

    # When new test has been found. Called for every test before a testing starts.
    function prepare(Test $test);

    # Test achives some result.
    function finish(Test $test);

    # Called on a testing end. Not necessarily all tests have been run.
    function end();
}

Some other minor improvements have been done, like printing code coverage percent value into console, or dumping arrays in a short syntax.

In conclusion

The Tester's internal code went through more changes, but they are not important for end users. For example, Tester's code base now uses new Nette coding style with small true, false, null constants and use new code checker. Details can be found in commits history.

Documentation is up to date and is still on the same place as usual.

Many thanks to all contributors! Enjoy the new release!


Changes in Nette Tester 1.7.0

by Miloslav Hůla about a year ago in Tester

Nette Tester 1.7.0 is out. I'll summarize the changes.

HtmlGenerator, which generates HTML code coverage report, now counts lines from not evaluated files as not covered. The only impact is, that you can see lower percentual coverage in report. Imagine the situation, that you have two source files (same lenght for simplification), but only one of them is loaded by tests. You got 100% coverage before, now you get only 50%.

If you are using TestCase test methods with @dataProvider, data provider method can return Traversable now. As a consequence of this, you can use generator (yield). It is handy for lazy data loading, for example from remote API. Short example:

class RemoteAPITest extends Tester\TestCase
{
    /** @dataProvider getRemoteData */
    public function testMe($arg)
    {
        Assert::same(..., $arg);
    }

    public function getRemoteData()
    {
        foreach ($this->remoteService->cases() as $case) {
            yield [$case];
        }
    }
}

(new RemoteAPITest)->run();

Few changes happend in Dumper class. There is a object hash among the dumps. It helps to recognize a dumped object identity.

# Before
(object) array()
(object) array()
(object) array()

# Now
(object) /* #f231 */ array()
(object) /* #1550 */ array()
(object) /* #f231 */ array()

Dumper::dumpException() shows line of source code beside an Assert call in a stack trace.

# Before
Failed: 27 should be 25

in tests/Utils/Strings.phpt(23) Tester\Assert::same()
in Utils/tests/bootstrap.php(39) {closure}()
in tests/Utils/Strings.phpt(32) test()

# Now
Failed: 27 should be 25

in tests/Utils/Strings.phpt(23) Tester\Assert::same(25, Strings::indexOf($foo, '1'));
in Utils/tests/bootstrap.php(39) {closure}()
in tests/Utils/Strings.phpt(32) test()

Next change is related to @dataProvider. You could use the INI file path before, now you can use PHP script as a data provider. You have to return array or Traversable from it.

# dataprovider.php
return json_decode(__FILE__ . '/data.json');

# or
return new DirectoryIterator(__DIR__ . '/data-files');

# or just
return [
    'one' => ['a', 'b', 'c'],
    'two' => ['d', 'e', 'f'],
];

Next two small changes. Environment variable term = xterm-256color enables color printing. And FileMock now supports unlink()ing.

And one big change as the last. Tester now supports PHPDBG SAPI with PHP 7.0.0 or newer. The biggest impact of it targets code coverage. PHPDBG offers mechanism to collect information about executed lines of source code. By the other words, it can collect code coverage statistics. It is faster alternative to Xdebug which is not needed with PHPDBG. Usage is simple:

vendor/bin/tester -p /path/to/phpdbg --coverage coverage.html --coverage-src src

Some plans on the end. Tester supports PHP 5.3 now. We agreed that we will wait two or three weeks after 1.7.0 release and we will skip to 5.4 at least. I belive, that Tester 1.7.0 is good enough for PHP 5.3 projects. But if you are against, leave us a comment with arguments.

Many thanks to volunteers which contributed! This relase is actually their work. Check out the changes.

Tester's documentation is up to date.


Changes in Nette Tester 1.6.0

by Miloslav Hůla 2 years ago in Tester

Nette Tester 1.6.0 is out. Let me summarize the changes.

New optional parameter $description has been added to almost all Assert methods. It works as an error message prefix when an assertion fails. It is handy in situations where the error message is not self-describing enough. For example, in a loop with assertion:

foreach ($model->getUsers() as $user) {
    Assert::true($user->isActive(), "User $user->name");
}
# Before 1.6.0 you got
Failed: FALSE should be TRUE

# Now you get
Failed: User sandokan: FALSE should be TRUE

Next change takes place in the Assert class too. A new assertion Assert::noError($callable) is suited to test snippets of code when there is no other assertion. It checks that $callable function does not generate any PHP notice/warning/error or exception. For example:

use Tester\Assert;

require __DIR__ . '/bootstrap.php';

Assert::noError(function () {
    # Checks that there is no error or exception
    $o = new Object;
    $o->doSomething();
});

Before 1.6.0 you could not wrap code into noError() and you had to make a workaround to prevent Error: This test forgets to execute an assertion message.

Next changes target the TestCase.

The @throws assertion is applied to testMethods() only, not to setUp() nor tearDown(). For example, if you had a testMethod() annoted by @throws \Exception, the method didn't throw an exception but tearDown() did, the test (probably wrongly) succeeded. Now, any exception thrown by setUp() or tearDown() is considered as test fail.

Another improvement is that the TestCase::tearDown() is called even when error is generated in a test method.

One small thing remains. Closures and anonymous classes dumping has been improved. You can find file and line info in dumps.

If you are interested in details, you can compare 1.5.0 and 1.6.0 version.


Changes in Nette Tester 1.5.0

by Miloslav Hůla 2 years ago in Tester

The Nette Tester 1.5.0 has been released last week. It's a small release and here I bring news.

Compatibility with PHP 7 has been improved. Tester works with PHP 7 well and now it supports Throwable interface. So all PHP 7 exceptions are printed and dumped correctly. No more problems are known, if you hit some, open an issue please.

The test runner (vendor/bin/tester) now prints PHP startup errors. It may disclose problem with PHP extensions loading.

Next new feature is internal thing for now. The Job instance, which represents one PHP process, now captures stderr output. It's internal, because you cannot use it for assertion yet. Stderr capturing has been discussed a long time ago. It's because this part of PHP (capturing more descriptors then one) is very buggy on Windows. With new PHP versions it looks better and better.

And that's all. As I said, a small release. Some tiny bugs in CloverXMLGenerator, Assert::match() and DomQuery::css2xpath() have been fixed. You can check the whole changelog for more information.

I have one bonus for you which I want to write in a bit more detail. It's about using Tester in a non common way.

Tester is available as a single PHAR file. It can be used in a dual way. The first one is as a library code source. You require PHAR file. Let's write two simple test files quick-1.phpt:

require __DIR__ . '/tester.phar';

Tester\Assert::true(TRUE);

and quick-2.phpt:

require __DIR__ . '/tester.phar';

Tester\Assert::true(FALSE);

Of course, you can run them as ordinary PHP scripts,

php quick-1.phpt
php quick-2.phpt

But with the Tester PHAR duality, you can simultaneously use tester.phar as a test runner:

php tester.phar *.phpt

One file, two purposes. It brings new workflow possibilities for a quick prototyping. You don't need a composer.json. You don't need an unpacked Tester code. Just download one file, include it and run…


Changes in Tester 1.4.0

by Miloslav Hůla 3 years ago in Tester

The Nette Tester 1.4.0 has been released one month ago. I would like to describe new features now, they are in the name of XML.

The code coverage functionality (via --coverage parameter) is present in the Tester for a long time. Report is formatted in an HTML, pleasant for programmer eyes. The Tester supports XML format now, perfect for a machine processing. The format is known as a CloverXML format. It is borrowed from the Java world, but it is quite spreaded in the PHP world too. For example, PhpStorm and Netbeans IDEs can load such report and helps you to visualize, how is the code covered by tests. The CloverXML format is supported by some CI tools too, e.g. Scruntinizer and Jenkins CI, or by specialized services like Coveralls. If you are curious how the CloverXML format looks like, you can take a look at Tester's test.

And how to generate such report? Simply use the --coverage coverage.xml. The file extension is important.

Next new feature is related to XML too. Parameter -o (output format) has been extended for the junit value and it means JUnit XML format. So, the Tester prints tests result as an XML into console by -o junit. It is friendly for a machine processing, e.g. for the Circle CI service.

The last change, I would like to describe here, looks like Tester's internal thing. Tester's source files has been moved from the Tester directory into src directory. If you are using the Composer's autoload, do not worry. But if you are loading the Tester manually by require 'tester/Tester/bootstrap.php', change the path. The old path still works, but will be sooner or later removed.

And that's all. A list of all small changes can be found in the release notes.