How to run Concuerror with EUnit

Here are some guidelines on how to use Concuerror as part of an EUnit suite.

First of all, the “order” in which you run the two tools is significant: e.g. invoking Concuerror with the test/0 function generated by EUnit is a bad idea, for at least two reasons:

you probably don’t want to test all of EUnit’s boilerplate code systematically and
the default test function generated by EUnit runs all tests, one after another; as a result, systematic testing will have to explore a number of schedulings that is the product of every individual test’s schedulings! You should use Concuerror on single tests instead.

Suppose you have the following EUnit tests:

-module(eunit_sample).

-include_lib("eunit/include/eunit.hrl").

%%==============================================================================

foo_test() ->
  ?assert(true).

msg_test() ->
  P = self(),
  spawn(fun() -> P ! foo end),
  spawn(fun() -> P ! bar end),
  receive
    Msg -> ?assert(foo, Msg)
  end.

reg_test() ->
  Fun =
    fun() ->
        receive message -> ok
        after 100 -> timeout
        end
    end,
  P = spawn(Fun),
  register(p, P),
  p ! message,
  ?assert(true).

For this suite, running EUnit yields:

$ erlc eunit_sample.erl
$ erl -noinput -s eunit_sample test -s erlang halt
  All 3 tests passed.

It is fairly easy to see, however, that the two latter tests can fail in particular interleavings¹.

Invoking Concuerror on msg_test/0 easily finds the error:

$ concuerror -m eunit_sample -t msg_test
Concuerror v0.17 (...) started at 08 Oct 2017 16:19:48
[...]
Errors were found! (check concuerror_report.txt)
Done at 08 Oct 2017 16:23:49 (Exit status: error)
  Summary: 1 errors, 2/2 interleavings explored

The output file contains the trace:

Concuerror v0.17 (...) started at 08 Oct 2017 16:23:49
 Options:
  [...]
################################################################################
Interleaving #2
--------------------------------------------------------------------------------
Errors found:
* At step 6 process P exited abnormally
    Reason:
      {{assertEqual,[{module,eunit_sample},
                     {line,15},
                     {expression,"Msg"},
                     {expected,foo},
                     {value,bar}]},
       [...]}
    Stacktrace:
      [...]
--------------------------------------------------------------------------------
Event trace:
   1: P: P.1 = erlang:spawn(erlang, apply, [#Fun<eunit_sample.'-msg_test/0-fun-0-'.0>,[]])
    in erlang.erl line 2693
   2: P: P.2 = erlang:spawn(erlang, apply, [#Fun<eunit_sample.'-msg_test/0-fun-1-'.0>,[]])
    in erlang.erl line 2693
   3: P.2: bar = P ! bar
    in eunit_sample.erl line 13
   4: P.2: exits normally
   5: P: receives message (bar)
    in eunit_sample.erl line 14
   6: P: exits abnormally ({{assertEqual,[{module,eunit_sample},{line,15},..]}})
   7: P.1: foo = P ! foo
    in eunit_sample.erl line 12
   8: P.1: exits normally
################################################################################
Exploration completed!
################################################################################
[...]
Info:
--------------------------------------------------------------------------------
* Automatically instrumented module io_lib
* Automatically instrumented module gen_server
* Automatically instrumented module eunit_sample
* Automatically instrumented module erlang
* You can see pairs of racing instructions (in the report and '--graph') with '--show_races true'

################################################################################
Done at 08 Oct 2017 16:23:49 (Exit status: error)
  Summary: 1 errors, 2/2 interleavings explored

Notice that Concuerror instrumented only some very basic modules. In contrast, invoking Concuerror on the test/0 function that is automatically generated by EUnit (and is automatically tested by Concuerror), yields:

$ concuerror -m eunit_sample
Concuerror v0.17 (...) started at 08 Oct 2017 16:28:38

Writing results in concuerror_report.txt

* Info: Automatically instrumented module io_lib
* Info: Automatically instrumented module gen_server
* Info: Automatically instrumented module eunit_sample
* Info: Automatically instrumented module eunit
* Info: Automatically instrumented module proplists
* Info: Automatically instrumented module eunit_tty
* Info: Automatically instrumented module eunit_listener
* Info: Automatically instrumented module erlang
* Info: Automatically instrumented module eunit_serial
* Info: Automatically instrumented module sets
* Info: Automatically instrumented module lists
* Info: Automatically instrumented module eunit_lib
* Info: Automatically instrumented module gb_trees
* Info: Automatically instrumented module dict
* Info: Automatically instrumented module eunit_server
* Warning: Your test seems to try to set up an EUnit server. This is a bad idea, for at least two reasons: 1) you probably don't want to test all of EUnit's boilerplate code systematically and 2) the default test function generated by EUnit runs all tests, one after another; as a result, systematic testing will have to explore a number of schedulings that is the product of every individual test's schedulings! You should use Concuerror on single tests instead.
* Info: Automatically instrumented module queue
* Info: Automatically instrumented module eunit_proc
* Tip: Increase '--print_depth' if output/graph contains "...".
* Error: Stop testing on first error. (Check '-h keep_going').
* Error: Concuerror does not support calls to built-in erlang:statistics/1 (found in eunit_proc.erl line 324).
  If you cannot avoid its use, please contact the developers.
  Stacktrace:
    [{eunit_proc,with_timeout,3,[{file,"eunit_proc.erl"},{line,324}]},
     {eunit_proc,run_group,2,[{file,"eunit_proc.erl"},{line,549}]},
     {eunit_proc,child_process,2,[{file,"eunit_proc.erl"},{line,353}]}]

Done at 08 Oct 2017 16:28:39 (Exit status: fail)
  Summary: 1 errors, 1/1 interleavings explored

Concuerror does not support calls to erlang:statistics/1. Additionally, it emits a Warning that this particular use is problematic.

Fortunately, the EUnit suite can be extended to also run the tests with Concuerror:

%%==============================================================================

-define(concuerror_options, [{module, ?MODULE}, quiet]).

foo_concuerror_test() ->
  ?assertEqual(ok, concuerror:run([{test, foo_test}|?concuerror_options])).

msg_concuerror_test() ->
  ?assertEqual(ok, concuerror:run([{test, msg_test}|?concuerror_options])).

reg_concuerror_test() ->
  ?assertEqual(ok, concuerror:run([{test, reg_test}|?concuerror_options])).

Invoking Concuerror again, yields:

$ erlc eunit_sample.erl
$ erl -noinput -s eunit_sample test -s erlang halt
eunit_sample: msg_concuerror_test...*failed*
in function eunit_sample:'-msg_concuerror_test/0-fun-0-'/0 (/home/stavros/git/Concuerror/tests-real/suites/options/eunit/eunit_sample.erl, line 38)
**error:{assertEqual,[{module,eunit_sample},
              {line,38},
              {expression,"concuerror : run ( [ { test , msg_test } | ? concuerror_options ] )"},
              {expected,ok},
              {value,error}]}
  output:<<"">>

eunit_sample: reg_concuerror_test...*failed*
in function eunit_sample:'-reg_concuerror_test/0-fun-0-'/0 (/home/stavros/git/Concuerror/tests-real/suites/options/eunit/eunit_sample.erl, line 41)
**error:{assertEqual,[{module,eunit_sample},
              {line,41},
              {expression,"concuerror : run ( [ { test , reg_test } | ? concuerror_options ] )"},
              {expected,ok},
              {value,error}]}
  output:<<"">>

=======================================================
  Failed: 2.  Skipped: 0.  Passed: 4.

The second and third tests have failed as expected and if run individually you can debug them from the report.

Have fun and keep testing!

In fact, the second and third tests MAY fail once in a while, but most times they won’t! ↩

Categories