Gallio adapter for Boost.Test

Submitted by NDL on 18 May, 2010 - 23:07

Tagged:

Adapter for Gallio testing platform that allows it to load and run Boost Test library based unit tests is released.

Rationale

Boost.Test framework is natural solution for native C++ code unit testing that already uses boost libraries, and possibly even for the code that doesn’t - though then there are a lot of other alternatives worth considering.

However, as it supports native code only, when using mixed-mode projects consisting of C++ and code in some of the .NET languages, there’s the need to use another testing framework for managed code - and, consequently, run those tests.

I wasn’t able to find any existing solution which would allow uniformly running both Boost.Test tests and any .NET testing framework tests, where by “uniformity” I mean using single interface/configuration to specify what tests to run, using single runner to actually run them and getting single report for both native/managed tests.

Gallio is good solution for running multiple tests in all major .NET testing frameworks so I decided the easiest way to achieve what I want is to add Boost.Test support to Gallio.

Due to Gallio being designed/used exclusively for .NET testing frameworks it appeared adding native testing framework poses some special challenges, but so far it looks like I was able to overcome most of them - and even without changing code of Gallio itself, which seems to indicate their extension mechanism is quite extendable indeed

Installation and usage

Either fetch Gallio.BoostAdapter sources from github and build them (the preferred way) or download compiled Gallio.BoostAdapter binaries (compiled against boost 1.43.0 using VS 2010, see below for pitfalls!)
Put binaries to the Gallio directory (the best is to put it into sub-directory to avoid cluttering the main Gallio directory), on the next run Gallio will detect Gallio.BoostAdapter and start to use it for DLLs in testing solution that are testable using Boost.Test.

Tested compilers are VS 2008/2010, tested boost versions are 1.42.0 and 1.43.0

Boost.Test framework DLL dependency

Gallio.BoostAdapter.Native.dll requires Boost.Test framework DLL to operate.

Provided binaries contain Boost.Test framework version compiled with SECURE_SCL defined to 0 and require testable DLL to use the same Boost.Test framework version and the same SECURE_SCL define value.

If testable DLL uses different Boost.Test framework version and/or SECURE_SCL define value - you should replace provided DLLs with your ones and recompile Gallio.BoostAdapter.Native.dll

Known bugs/limitations

Due to incompatibilities between different VS runtime versions it’s necessary that Boost.Test library, tested DLL and Gallio.BoostAdapter itself be compiled using the same compiler - I was unable to reuse VS 2008 compiled Boost.Test library and Gallio.BoostAdapter with VS 2010 compiled tested DLL.
The information provided by Boost.Test is limited compared to what Gallio could use - for example, there seems to be no way to get source files locations of Boost.Test asserts to show them in Gallio without modifying Boost.Test framework code.
So far I was unable to force VS2008/VS2010 plus Gallio to recognize Boost.Test projects in solution as testable, so it’s not possible to run Boost.Test tests using VS test runner - not sure whose “fault” it is, but it may well appear that it’s VS limitation and then we’re out of options, I’m afraid.
Due to a bunch of reasons, such as: extra efforts required to interact between native/managed code; Boost.Test architecture not mapping exactly to Gallio one; lack of support of native code testing in Gallio - things were done in many places differently to what would be “natural” in Gallio framework, in some places sacrifising beauty/uniformity/performance.

For example, instead of using “common” Gallio approach of Gallio being the controller of tests to run (and reusing existing support classes for that) the information about tests to run is fed to Boost.Test and then Boost.Test performs all selected tests execution and Gallio just receives back status information for each executed test. This distinction might seem minor, but it translates to completely different implementation compared to other test frameworks extensions in Gallio. The same counts for tests exploration and representation: as native tests are not discoverable by reflection and not representable as Gallio ICodeElementInfo objects, I wasn’t able to reuse Gallio support classes for that task either.

Architecture

Gallio.BoostAdapter consists of two components:

Gallio.BoostAdapter.dll - managed assembly that implements interfaces required by Gallio and loads/communicates with Gallio.BoostAdapter.Native.dll
Gallio.BoostAdapter.Native.dll - unmanaged DLL that bridges Gallio.BoostAdapter with Boost.Test and testable DLL.

The solution with single mixed mode DLL written in C++/CLI was considered but determined to be inappropriate in this situation:

It has problems with isolation, most likely related to native C runtime going crazy when performing the same DLL loading in isolated context.
On the other hand, not using isolation is not only less safe (as single failed test would break all tests in the project) but also seems to be not possible at all with current version of Boost.Test as it has problems performing multiple init/clear calls that are required to test multiple testable DLLs in single process.
It would be harder to implement multiple architectures/configurations support as we would have to switch somehow from one architecture/configuration DLL to another one each time isolation starts.

Different processor architectures/configurations support

Gallio.BoostAdapter.Native.dll needs to match the architecture/configuration of the testable DLL so that it can successfully load/test it. This means Gallio.BoostAdapter.dll has to determine architecture/configuration of testable DLL prior to loading it - and then calling the appropriate version of Gallio.BoostAdapter.Native.dll

The required check is implemented by parsing PE header to determine architecture and by parsing import table looking for Boost.Test dll name and inferring configuration from its name.

NDL's blog
• Login to post comments

Thanks .. adapter for cppUnit ?

Submitted by sudarshan (not verified) on 12 July, 2010 - 01:07.

Thanks for the plugin..
Very helpful ..

I am in the stage of deciding the unit test framework for mixed mode projects.
Now, either Boost.Test or cppUnit would be chosen..
Which one do you suggest from your experience ?

Also if cppUnit then do we have a plugin of this kind ?

Thanks a lot for your help..

Thanks .. adapter for cppUnit

Submitted by sudarshan (not verified) on 12 July, 2010 - 01:02.

I was looking for one a plug-in with Gallio and Boost.Test and found it here .. Thanks a lot. I am in the process of choosing a unit test framework. I have bottled down to either Boost.Test or cppUnit.

Which one would you suggest ? Also if cppUnit then do we have a plugin with Gallio ?

I am also "using mixed-mode projects consisting of C++ and code in some of the .NET languages "

Thanks for your help ..

Re: Adapter for cppUnit

Submitted by NDL on 12 July, 2010 - 01:50.

Which one would you suggest ?

I have no real experience with cppUnit, but as far as I know (I may be wrong!) Boost.Test is both less intrusive (i.e. gives more freedom in tests organization and the way they’re implemented, such as free functions support and multiple ways of easy automatic test cases/suites definitions and registrations) and also it supports some nice features (like instantiating the same test with different template parameters) that cppUnit seems to lack.

One more thing is that as I use boost libraries in most of my projects anyway, it’s easier to use Boost.Test than to introduce one more dependency to the project by using cppUnit or whatever else testing framework.

It’s not that Boost.Test is perfect, though:

It lacks proper multi-threading support.
Its mocking capabilities are quite limited and not documented at all.
Poor extensibility of test framework itself: I had some problems when implementing Gallio adapter because of that and result is worse than it could be had Boost.Test been more “cooperative” - for example there’s no way to retrieve details of tests assertions, Boost.Test uses this information when performing its own reporting but does not provide interfaces to retrieve it externally.

Although I’m not sure cppUnit is better at any of these points anyway …

Also if cppUnit then do we have a plugin with Gallio ?

No, I don’t think there’s any. And considering Gallio is targeted at managed testing frameworks, implementing support for native testing framework is not that trivial either.

regarding the adapter

Submitted by sudarshan (not verified) on 12 July, 2010 - 06:25.

Thanks a lot for valuable information ..

Our project does not have Boost libraries, so I was also thinking to go with cppUnit but then I thought it would be relatively easy to manage both managed code and native code under Gallio. With your adapter, it would solve the problem ..

Also, how is Gallio performing in the projects, in the sense that, is it easy to manage in big projects, is it crashing etc ?

One of the reasons we need Gallio is the HTML test reports...

Like even if cppUnit has some kind of plugin to generate the outputs in HTML, it would be great.. do we have one ?

Thanks once again ..

Re: regarding the adapter

Submitted by NDL on 13 July, 2010 - 00:52.

Also, how is Gallio performing in the projects, in the sense that, is it easy to manage in big projects, is it crashing etc?

So far I haven’t tried it on really large projects, but for what I have now it works fine. It has some minor issues with usability & preferences, but in my opinion they’re not really serious.

If you want to use it for continuous integration - check if your CI system supports its reports format, as the XML format Gallio uses seems to be non-trivial, so for example Gallio plugin for Hudson CI system has problems with its reports parsing. On the other hand, if you need only “pass/fail” indication - that is trivial to get as exit status of test runner.

One of the reasons we need Gallio is the HTML test reports…

Like even if cppUnit has some kind of plugin to generate the outputs in HTML, it would be great.. do we have one ?

As I’ve not used cppUnit I cannot give any useful info on that, sorry. I know that there are external runners for cppUnit available, but you will have to check what reports formats they support.

Thanks once again, Now if I

Submitted by sudarshan (not verified) on 15 July, 2010 - 09:53.

Thanks once again,

Now if I want to use this adapter, but in VS2005, can I manually add the files and make a similar solution ? Because I will not be able to open them in VS2005.

Are there any other dependencies which will hinder my method ?

Thanks
Sudarshan

Compiling for VS2005

Submitted by NDL on 15 July, 2010 - 11:00.

In fact project formats between VS2005 and VS2008 are almost the same. I have scripts to convert from VS2008 to VS2005 for one of my older projects, I’ve applied them to project files of Gallio.BoostAdapter and committed resulting projects for VS2005 to the github repository (note I’ve not checked they really work, I do not have VS2005 anymore to test that!)

If that doesn’t work - yes, surely you can just re-create projects from scratch. As for dependencies - for native DLL compilation you’ll obviously need boost, for C# DLL - Gallio. You’ll have to adjust these dependencies in any case though to point to appropriate directories and appropriate versions of packages, even if VS2005 projects open OK.

Thanks a lot.. I will test

Submitted by sudarshan (not verified) on 15 July, 2010 - 18:36.

Thanks a lot..
I will test them and let you know ..

Thanks

errors in VS2005 compilation

Submitted by sudarshan (not verified) on 15 July, 2010 - 20:29.

Hi,

The errors during the compilation were same as the link shows: http://stackoverflow.com/questions/389644/a-new-expression-requires-or-a...

the error in the code -- (Error 23 A new expression requires () or [] after type ndl-Gallio.BoostAdapter-0c72eec\Gallio.BoostAdapter\RunTask.cs 135)

There are other errors too, which I believe is caused by different compiler version.
I am new to C# and I am looking to fix it ..

Do let me know if you know to fix them ..

Thanks
Sudarshan

Re: errors in VS2005 compilation

Submitted by NDL on 15 July, 2010 - 22:53.

Yes, it looks like I’ve used a couple of features that are part of C# 3.0, therefore VS2005 cannot compile it.

Basically it seems to boil down to the following cases: 1. Lambdas. 2. Objects initializers.

For lambdas, the fix is as follows:

change code like

(text) => Logger.Log(LogSeverity.Error, text)

delegate(string text) { Logger.Log(LogSeverity.Error, text); }

Of course in each particular case you have to look up what types delegate arguments have.

For objects initializers, the change is as follows:

MessageSink.Publish(
    new TestStepFinishedMessage
    {
        StepId = testStepsStack_.Pop().Id,
        Result = result
    }
);

TestStepFinishedMessage msg = new TestStepFinishedMessage();
msg.StepId = testStepsStack_.Pop().Id;
msg.Result = result;
MessageSink.Publish(msg);

Just use the same strategies to change remaining code, there seems to be not that many places that require changes.

That worked .. but ..

Submitted by sudarshan (not verified) on 16 July, 2010 - 20:03.

That's so nice .. It worked perfectly and I was able to build the solution.

Also I wanted to just tell that now Gallio does not directly find the plugin folder. I had to execute a command so that it finds the plugin folder. This is just an info to other users. For more details http://blog.bits-in-motion.com/2009/09/gallio-plug-in-model.html

Even though it found the folder, I could not test it through lib. I still get the error "This test is not supported by any available test framework."

I assume the mistake is with the boost.test lib dlls. I did not build them, but downloaded with the help of boostpro.

How do you build and install boost.test in normally ?

Thanks once again for helping me out.

Re: Boost.Test build

Submitted by NDL on 17 July, 2010 - 01:32.

I assume the mistake is with the boost.test lib dlls. I did not build them, but downloaded with the help of boostpro.

If there are no other error/warning messages (like exceptions after attempts to load DLLs to be tested) the problem might be somewhere at earlier stage. Does boostpro distribution use “boost_unit_test_framework*.dll” names for their DLLs? If not, check in adapter code for testable DLL will fail, see BoostTestFileTypeRecognizer.cs, line 46.

Other than that, if you follow the article (i.e. use the same version of boost.test both for adapter building and for your tests, use the same defines and also use the same compiler/runtime version that boost.test was compiled with) you should be fine.

How do you build and install boost.test in normally ?

Mmm … Just following official boost documentation?

Still facing problem

Submitted by sudarshan (not verified) on 20 July, 2010 - 06:56.

Hi,

I am not sure what mistake I am doing. But I get the same error which says "This test is not supported by any available test framework" even though Gallio-Boost adapter is listed in the plugin list.

These are the steps I followed. Please let me know where I am going wrong.

Gallio - Boost integration steps

1. Firstly, I downloaded the source file to build the adaptor.
2. Extracted the zip file and opened the solution in VS2005.
3. Downloaded the boost_1_43_0.7z zip file and extracted to a folder.
4. Used bjam to build the boost libraries. Given below is the command I used to build.
bjam define=SCL_SECURE=0 --build-type=complete
5. Boost.Test alone could have been built, but to follow the correct steps from the documentation, I am building the whole library.
6. I am adding the include paths and the lib under visual studio’s tools menu and by clicking on the options menu.
Added path for include : F:\boost_lib\boost-jam-3.1.18-1-ntx86\boost_1_43_0
Added path for include : F:\boost_lib\boost-jam-3.1.18-1-ntx86\boost_1_43_0\stage\lib

7. Changed the additional include dir and additional lib dir so that it points my boost dir.
8. Gallio adaptor builds perfectly without any error.
9. The bin folder is also formed.
10. DLLs are formed in both debug and release folder.

I then created a folder named Boost in Gallio\bin. Inside the folder Boost, I created resources folder which consits of icon.
Inside the Boost folder, copied the plugin and Gallio.BoostAdapter.dll ( WHICH ONE IS THIS? The one formed in the debug dir of adaptor or the release dir of adaptor ?).
Created a folder x86, in which another folder is created which are Debug and Release
Inside Debug - boost_unit_test_framework-vc80-mt-gd-1_43.dll and Gallio.BoostAdapter.Native.dll (this is copied from debug dir of the adaptor) are copied.
Inside Debug - boost_unit_test_framework-vc80-mt-1_43.dll and Gallio.BoostAdapter.Native.dll (this is copied from release dir of the adaptor)are copied.

----

Can you please give a sample app with Boost.Test. Do I need to add any references in the project ?
with the sample app, I can build and test once again. By this I can rule error in this part.

Thanks a lot. This will be really helpful .. Thanks
Thanks

Sudarshan

Re: Still facing problem

Submitted by NDL on 20 July, 2010 - 14:23.

Hi,

Some notes:

Used bjam to build the boost libraries. Given below is the command I used to build. bjam define=SCL_SECURE=0 —build-type=complete

The proper define name is _SECURE_SCL, though it’s unlikely to be the case of your problems - it might crash in release due to wrong define but that would look differently from what you experience.

Inside the Boost folder, copied the plugin and Gallio.BoostAdapter.dll ( WHICH ONE IS THIS? The one formed in the debug dir of adaptor or the release dir of adaptor ?).

Doesn’t really matter: I use “release” for actual test runs but “debug” is better if you’d like to debug adapter itself.

Created a folder x86, in which another folder is created which are Debug and Release Inside Debug - boost_unit_test_framework-vc80-mt-gd-1_43.dll

The file boost_unit_test_framework-vc80-mt-gd-1_43.dll is the one you’ve got after boost build, right?

Your sequence of actions looks correct though. What are exactly the messages shown when you try to load testable DLLs to Gallio.Icarus? Does it happen right on the loading - or only after you try to run tests?

One more possibility is outdated Gallio plugins cache (probably irrelevant as you say Gallio.BoostAdapter plugin is shown in the list of plugins), you might want to try removing it, located at C:\Users\\AppData\Local\Gallio\Plugin Metadata Cache for Win7, use relevant path starting with “Documents and Settings” for WinXP.

Can you please give a sample app with Boost.Test. Do I need to add any references in the project ? with the sample app, I can build and test once again. By this I can rule error in this part.

I cannot give you the complete test project as I don’t have nor VS2005 anymore, neither boost/Boost.GallioAdapter compiled with it (currently I have it compiled for VS2010 only). However, the sequence to create such project is easy:

Create new Win32 project named TestableDll, set its type to DLL in wizard: Application Settings → Application type → DLL.
Define _SECURE_SCL as 0 in project settings (doesn’t really matter for debug builds though).
Add BOOST_ALL_DYN_LINK define to project settings.
Add boost include/link paths to project settings.
Replace file TestableDll.cpp with the following content:

// TestableDll.cpp : Defines the exported functions for the DLL application.
//
 
#include "stdafx.h"
 
#include <boost/test/unit_test.hpp>
 
// This function is required so that Boost.Test can load this DLL.
extern "C" __declspec(dllexport) bool init_unit_test()
{
    boost::unit_test::framework::master_test_suite().p_name.value = "TestableDll Unit Tests";
    return true;
}
 
BOOST_AUTO_TEST_SUITE(TestableDllSuite)
 
BOOST_AUTO_TEST_CASE(CorrectAddition)
{
    BOOST_CHECK_EQUAL(2 + 2, 4);
}
 
BOOST_AUTO_TEST_CASE(WrongAddition)
{
    BOOST_CHECK_EQUAL(2 + 2, 5);
}
 
BOOST_AUTO_TEST_SUITE_END()

Pay attention to init_unit_test() function.

Then try loading this DLL in Gallio.Icarus.

You can also use console_test_runner example application distributed with boost.test (you have to compile it separately though) to check your testable DLL loads correctly with boost.test framework - Gallio.BoostAdapter uses the same mechanism for loading, so if console_test_runnter loads it OK - so should Gallio.BoostAdapter.

If console_test_runner loads your testable DLL OK, yet Gallio.BoostAdapter cannot test it - you’ll have to debug Gallio.BoostAdapter itself. To do that copy its debug version to Gallio folder, attach to the Gallio.Icarus executable with debugger and set breakpoint somewhere near BoostTestFileTypeRecognizer.cs (line 27) to check it gets executed and trace it to check it returns true.

Works in VS2008 but some strange errors in VS2005

Submitted by sudarshan (not verified) on 21 July, 2010 - 10:33.

Hi,

Thanks for the boost test sample. I think I was going wrong there.

Now the great news is that, I can run the tests in VS2008. It runs perfectly.
In VS2005, the tests are loaded which were not loading at all previous cases.
But when I run the tests they don't behave in the right way
This is the screenshot of what happens:
http://www.flickr.com/photos/37773563@N00/4814830410/sizes/l/

So the tests are not skipped, but they also are not shown as passed or fail in the case of debug dll.
In case of release dll, number of failed cases is shown but not the passed ones.
Quite unique.
All these happen in VS2005.
My guess is I need to debug as you have suggested.

Please do let me know if you have any thoughts on this.

I really appreciate the help and the time from your side.

Thanks once again
Sudarshan

Re: Works in VS2008 but some strange errors in VS2005

Submitted by NDL on 21 July, 2010 - 21:45.

Hm, are you using exactly the test DLL code I’ve sent you or have you modified tests somehow?

Because in original code there were two test cases, one of them (CorrectAddition) should succeed, while the other one (WrongAddition) should fail.

On your screenshot it looks like individual test cases are processed correctly for release, but not for debug (both succeed while WrongAddition should fail). Summary is wrong both for debug and release indeed.

As for the VS2005/VS2008 - do you mean that you’ve tried full compilation (boost.test, Gallio.BoostAdapter, your test DLLs) separately with VS2005 - and then with VS2008? Or was there some mixture of both?

Could you upload screenshot when running with VS2008?

Also please post execution logs both for VS2005 and VS2008 (“Runtime” tab, select “Minimum Severity” = “Debug” before loading tests).

These are the changes I made in the code for VS2005 to build

Submitted by sudarshan (not verified) on 21 July, 2010 - 21:40.

Added to the previous reply,

changes made in the code for VS2005 .. I hope these are not affecting. Please check them..

1. Runtask.cs line 104 
 
Changed to:
ErrorReporterDelegate errorReporter =
                new ErrorReporterDelegate(delegate(string text) { Logger.Log(LogSeverity.Error, text); }); 
 
from :
ErrorReporterDelegate errorReporter =
                new ErrorReporterDelegate((text) => Logger.Log(LogSeverity.Error, text));
 
 
------------------------
2. Runtask.cs line 135
 
Changed to:
TestStepFinishedMessage msg = new TestStepFinishedMessage();
                msg.StepId = testStepsStack_.Pop().Id;
                msg.Result = result;
                MessageSink.Publish(msg);
 
from:
 MessageSink.Publish(
new TestStepFinishedMessage
{
 
StepId = testStepsStack_.Pop().Id,
 Result = result
}
);
 
------------------------
3. Runtask.cs line 192
 
Changed to:
TestStepStartedMessage msg_start = new TestStepStartedMessage();
                msg_start.Step = new TestStepData(testStepsStack_.Peek());
                MessageSink.Publish(msg_start);
 
from:
MessageSink.Publish(
new TestStepStartedMessage
 {
 
Step = new TestStepData(testStepsStack_.Peek()) }
 
);
-------------------------
4. Runtask.cs line 217
changed to:
TestStepFinishedMessage msg = new TestStepFinishedMessage();
                msg.StepId = testStepsStack_.Pop().Id;
                msg.Result = result;
                MessageSink.Publish(msg);
 
from:
MessageSink.Publish(new TestStepFinishedMessage
{
 
StepId = testStepsStack_.Pop().Id,Result = result
 }
);
------------------------
5. Runtask.cs line 266
changed to: 
TestStepStartedMessage msg_start = new TestStepStartedMessage();
                msg_start.Step = new TestStepData(step);
                MessageSink.Publish(msg_start);
from:
MessageSink.Publish(
new TestStepStartedMessage
{
 
 Step = new TestStepData(step)
  }
 
 );
-------------------------
6. Runtask.cs line 307
changed to:
TestStepFinishedMessage msg = new TestStepFinishedMessage();
                msg.StepId = testStepsStack_.Pop().Id;
                msg.Result = result;
                MessageSink.Publish(msg);
 
from:
MessageSink.Publish(
                    new TestStepFinishedMessage
                    {
                        StepId = testStepsStack_.Pop().Id,
                        Result = result
                    }
                );
-------------------------
7. Explore.cs line 62
To:
ErrorReporterDelegate errorReporter =
                    new ErrorReporterDelegate(delegate(string text) { Logger.Log(LogSeverity.Error, text); });
From:
ErrorReporterDelegate errorReporter =
 new ErrorReporterDelegate((text) => Logger.Log(LogSeverity.Error, text));
 
------------------------
8. BoostTestDriver.cs line 171
To: 
progressMonitor.Canceled += delegate(object sender, EventArgs e) { remoteMonitor.Cancel(); };
From:
progressMonitor.Canceled += (sender, e) => { remoteMonitor.Cancel(); };
------------------------
9. BoostTestDriver.cs line 180
 
To: 
delegate(string statusMessage) { progressMonitor.SetStatus(statusMessage); },
 
From:
(statusMessage) => progressMonitor.SetStatus(statusMessage),

Thanks

replies

Submitted by sudarshan (not verified) on 21 July, 2010 - 22:16.

Hm, are you using exactly the test DLL code I’ve sent you or have you modified tests somehow?
-- I am using the same Test DLL code which you have sent.

Because in original code there were two test cases, one of them (CorrectAddition) should succeed, while the other one (WrongAddition) should fail.

-- yes.

-- yes that is the strange thing happening. Debug its not at all showing the failed cases.

--
yes I did the whole compilation of all separately in my personal laptop which VS2005 is in development laptop. Since the our source code is in VS2005, I need to integrate this in VS2005.

Could you upload screenshot when running with VS2008?
-- Sure, I will upload when I get back home.

Also please post execution logs both for VS2005 and VS2008 (“Runtime” tab, select “Minimum Severity” = “Debug” before loading tests).
-- Sure, I will do this too.

Thanks
sudarshan

log in VS2005

Submitted by sudarshan (not verified) on 22 July, 2010 - 02:01.

--------------------------------------------
Host started at 7/21/2010 3:55:54 PM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.793a907477256b62'
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Running 2 test cases...
f:/testproj/testabledll/testabledll/testabledll.cpp(24): error in "WrongAddition": check 2 + 2 == 5 failed [4 != 5]
Host stopped at 7/21/2010 3:55:55 PM.
Host process exited with code: 0
Host started at 7/21/2010 3:55:55 PM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.92835052ff6bdb66'
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Running 2 test cases...
./testabledll.cpp(24): error in "WrongAddition": check 2 + 2 == 5 failed [4 != 5]
Host stopped at 7/21/2010 3:55:57 PM.
Host process exited with code: 0
--------------------------------------------

The first section is debug dll and second part is the release dll.
I even changed the test case so that both are failing, I could see that failed cases are listed in logs. But in the execution log, gallio shows failed cases of release alone.
I does not show fail or success case for debug. And does not show the sucess case in release.

I will paste the log from VS2008 soon.

VS2008 screenshots and log

Submitted by sudarshan (not verified) on 22 July, 2010 - 09:31.

Hi,
Here is the screenshot from VS2008. It works perfectly here
http://www.flickr.com/photos/37773563@N00/4817665998/sizes/l/

Log from VS2008 during the tests:
------------------------

Host started at 7/22/2010 1:59:52 AM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.24a72b823886e72e'
Disabled plugin 'Gallio.VisualStudio.Shell': The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or ${process:VSTESTHOST.EXE} or ${process:QTAGENT.EXE} or ${process:QTAGENT32.EXE} or ${process:QTDCAGENT.EXE} or ${process:QTDCAGENT32.EXE}'.
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.
Host stopped at 7/22/2010 1:59:55 AM.
Host process exited with code: 0
Host started at 7/22/2010 2:00:13 AM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.780f4edd5d88dd2d'
Disabled plugin 'Gallio.VisualStudio.Shell': The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or ${process:VSTESTHOST.EXE} or ${process:QTAGENT.EXE} or ${process:QTAGENT32.EXE} or ${process:QTDCAGENT.EXE} or ${process:QTDCAGENT32.EXE}'.
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.
Host stopped at 7/22/2010 2:00:16 AM.
Host process exited with code: 0
Host started at 7/22/2010 2:00:16 AM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.2b6e6e18627e2b80'
Disabled plugin 'Gallio.VisualStudio.Shell': The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or ${process:VSTESTHOST.EXE} or ${process:QTAGENT.EXE} or ${process:QTAGENT32.EXE} or ${process:QTDCAGENT.EXE} or ${process:QTDCAGENT32.EXE}'.
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.
Host stopped at 7/22/2010 2:00:18 AM.
Host process exited with code: 0
Host started at 7/22/2010 2:00:21 AM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.b1d5957bec018814'
Disabled plugin 'Gallio.VisualStudio.Shell': The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or ${process:VSTESTHOST.EXE} or ${process:QTAGENT.EXE} or ${process:QTAGENT32.EXE} or ${process:QTDCAGENT.EXE} or ${process:QTDCAGENT32.EXE}'.
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.
Running 2 test cases...
c:/documents and settings/sprasad4/my documents/visual studio 2008/projects/testabledll/testabledll/testabledll.cpp(24): error in "WrongAddition": check 2 + 2 == 5 failed [4 != 5]
Host stopped at 7/22/2010 2:00:23 AM.
Host process exited with code: 0
Host started at 7/22/2010 2:00:24 AM.
Running under CLR v2.0.50727 runtime.
Listening for connections on IPC port: 'IsolatedProcessHost.126140b727016405'
Disabled plugin 'Gallio.VisualStudio.Shell': The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or ${process:VSTESTHOST.EXE} or ${process:QTAGENT.EXE} or ${process:QTAGENT32.EXE} or ${process:QTDCAGENT.EXE} or ${process:QTDCAGENT32.EXE}'.
Disabled plugin 'Gallio40': The plugin enable condition was not satisfied: '${minFramework:NET40}'.
Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell'.
Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.
Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.
Running 2 test cases...
./TestableDll.cpp(24): error in "WrongAddition": check 2 + 2 == 5 failed [4 != 5]
Host stopped at 7/22/2010 2:00:25 AM.
Host process exited with code: 0
--------------------------------------------------------------------------

If I need to debug, how shall I proceed doing it ? I feel the results are not populated correctly in VS2005. Somewhere it is missed.

Thanks once again

Re: VS2008 screenshots and log

Submitted by NDL on 22 July, 2010 - 14:14.

Hi,

changes made in the code for VS2005 .. I hope these are not affecting. Please check them..

If you use the same code for Gallio.BoostAdapter both for VS2005 and VS2008, then if it works for VS2008 but not for VS2005 it’s unlikely the problem is in the changes you’ve made.

Logs both for VS2005 and VS2008 look OK, unfortunately as far as I see there are no hints as to what might be wrong …

If I need to debug, how shall I proceed doing it ? I feel the results are not populated correctly in VS2005. Somewhere it is missed.

I’ll describe one approach that can be used for debugging - there are several others, but this one, while a bit hacky, should do the job.

Tests execution happens in separate process - either Gallio.Host.exe or Gallio.Host.x86.exe, depending on the platform. This means you cannot simply attach debugger to Gallio.Icarus.exe to debug Gallio.BoostAdapter. This separate process starts tests execution by calling BoostTestTask::RunImpl. Therefore you can put either delay at the start of this function or MessageBox.Show call so that when tests are started, you have enough time to attach debugger to Gallio.Host.exe process and set breakpoint while the delay/MessageBox call is in progress.

After attaching debugger I suggest to set breakpoints in all functions in RunTask class used as delegates for tests feedback: TestStart, TestFinish and all the like. You should see them called in the meaningful sequence, also check appropriate “Finish” functions are called both for unit tests and for test suites.

One more thing you might want to check is the content of XML reports Gallio.Icarus generates (typically in the directory Reports). Try comparing versions obtained using VS2005 and VS2008, that might give you a clue as to what is wrong.

Re

Submitted by sudarshan (not verified) on 22 July, 2010 - 18:40.

Hi,

Not really, since in VS2008, the original code would work because the complier is matched. So I did not change the lambdas to delegates. etc.
Well, I will change them and see if its working in VS2008. But still seems unlikely.

Will debug and let you know what happens.

Thanks again

Re

Submitted by sudarshan (not verified) on 25 July, 2010 - 03:49.

Hi,

Can you please let me know how I can build the adaptor with the boost library built in VS2005 ? I get a linker error which looks for the boost library built in msvc9.0 when I try to build the adaptor in VS 2008.

I have seen the limitation that all the compilers must be same, but I was trying to overcome that.

Is it because of the BOOST_TEST_DYN_LINK ?

I am trying to build testdll in VS2005 and adaptor in 2008. Just trying to fix that limitation. Gallio icarus loads the testdll now, but the name it gives under the root is "Master Test Suite". When I try to run that, it gives me a error "Boost Test error: test tree is empty"

I feel its due to the boost dll built in two versions now. Can I link manually with the boost library built in msvc-8.0 ?

Any suggestion ?

Thanks
Sudarshan

Enforcing boost toolset

Submitted by NDL on 25 July, 2010 - 10:17.

Can you please let me know how I can build the adaptor with the boost library built in VS2005 ? I get a linker error which looks for the boost library built in msvc9.0 when I try to build the adaptor in VS 2008.

I wouldn’t recommend that as you may have problems due to mixing two different runtimes, but if nonetheless you’d like to try it out - the define you need is BOOST_LIB_TOOLSET=”vc80”

Is it because of the BOOST_TEST_DYN_LINK ?

No, it would be the same if static linking is used too, here the issue is with auto-linking: boost determines what DLL/lib name to use for linking based on toolset used to compile the program that includes boost headers. The alternative to BOOST_LIB_TOOLSET define is to disable auto-linking and specify required libraries manually, but it’s quite painful to manage in the long run.

Re

Submitted by sudarshan (not verified) on 25 July, 2010 - 11:25.

Hi,
Will try them..

Kindly let me know if there is a problem in the value of the bridge in the Execute function of the Runtask.cs. they seem to be different in VS 2008 http://www.flickr.com/photos/37773563@N00/4825722903/sizes/l/

and vs 2005 http://www.flickr.com/photos/37773563@N00/4825722901/sizes/l/

and when I compared both of them in some functions of Rantask.cs, I found that for VS2005, AssertionResult parameter is true if the test case failing. Though we get in the runtime log in gallio that the case failed. I am guessing that the native to managed code didnt work correctly for this DLL?

Thanks
Sudarshan

Linker errors

Submitted by sudarshan (not verified) on 26 July, 2010 - 03:48.

Hello,

I tried to link boost libraries built with vc8.0 and gallio adapter, but got linker errors.
In preprocessor definitions I have the following
WIN32
_DEBUG
BOOST_TEST_DYN_LINK
_SECURE_SCL=0
BOOST_LIB_TOOLSET=”vc80”

In the additional dependencies for Linker, I added
boost_unit_test_framework-vc80-mt-gd-1_43.lib

Additional library directories is also correctly pointing the folder.

The linker error I receive :
Error 1 error LNK2019: unresolved external symbol "__declspec(dllimport) protected: __thiscall boost::unit_test::test_tree_visitor::~test_tree_visitor(void)" (__imp_??1test_tree_visitor@unit_test@boost@@IAE@XZ) referenced in function "public: __thiscall TestsVisitor::~TestsVisitor(void)" (??1TestsVisitor@@QAE@XZ) BoostTestBridge.obj

Error 2 error LNK2019: unresolved external symbol "__declspec(dllimport) protected: __thiscall boost::unit_test::test_observer::~test_observer(void)" (__imp_??1test_observer@unit_test@boost@@IAE@XZ) referenced in function "public: __thiscall TestObserver::~TestObserver(void)" (??1TestObserver@@QAE@XZ) BoostTestBridge.obj

Am I missing anything ?

Thanks a lot

Re: linker errors

Submitted by NDL on 26 July, 2010 - 09:42.

Kindly let me know if there is a problem in the value of the bridge in the Execute function of the Runtask.cs

There’s no sense in comparing function pointer values, they do not provide any meaningful information and can perfectly be different when compiled with different versions of compilers - in fact considering this is managed code and pointers may point to some kind of stubs/trampolines the addresses may be different even between successive runs of the same code.

and when I compared both of them in some functions of Rantask.cs, I found that for VS2005, AssertionResult parameter is true if the test case failing. Though we get in the runtime log in gallio that the case failed. I am guessing that the native to managed code didnt work correctly for this DLL?

You can also try debugging unmanaged code in BoostTestBridge.cpp to check that values are reported correctly from boost.test.

In the additional dependencies for Linker, I added boost_unit_test_framework-vc80-mt-gd-1_43.lib

You shouldn’t need that if BOOST_LIB_TOOLSET is specified correctly, if it does not work without explicit library specification - smth else is wrong.

Am I missing anything ?

As I said already, I don’t think mixed compilation with VS2005 and VS2008 is going to work, especially when boost is involved.

See, for example, this question - the exact problem is different there, but the answer is the same.

few clarifications

Submitted by sudarshan (not verified) on 13 August, 2010 - 22:14.

Hi,

Wanted to update that the issue got fixed after finding that there was problem in finding boost test cases as a test suite or a test case. Due to some reason, it was wrongly reading a test cases as a suite. Now the report from Gallio works fine. I will post the changes so that you can see them.

But I had few general clarifications:

1. The actual test dll has the following settings in the debug mode :
Runtime library :Multi-threaded Debug (/MTd)
Use of MFCUse : MFC in a Static Library

2. Due to this, I removed BOOST_ALL_DYN_LINK, otherwise this error would come "Mixing a dll boost library with a static runtime is a really bad". So, I did static linking.

3. Now the library used for this would be the libboost_unit_test_framework-vc80-mt-sgd-1_43.lib

4. Both the test project and Gallio-Boost adapter correctly compiles without any error.

5. Now what other changes should I make so that the Gallio adapter correctly recogonizes the test dll ( that it needs Boost.test)

6. This is because, the plugin file points to boost_unit_test_framework-vc80-mt-gd-1_43.dll. This would not work.. Correct me if I am wrong.

7. I guess due to this we need make changes in BoostTestFileTypeRecognizer.cs

8. In general, why do we have libs forming for boost_unit_test_framework-vc80-mt-gd-1_43 and libboost_unit_test_framework-vc80-mt-sgd-1_43 ?

Thanks for your help.

ndl@home

Gallio adapter for Boost.Test

Rationale

Installation and usage

Boost.Test framework DLL dependency

Known bugs/limitations

Architecture

Different processor architectures/configurations support

Comment viewing options

Thanks .. adapter for cppUnit ?

Thanks .. adapter for cppUnit

Re: Adapter for cppUnit

regarding the adapter

Re: regarding the adapter

Thanks once again, Now if I

Compiling for VS2005

Thanks a lot.. I will test

errors in VS2005 compilation

Re: errors in VS2005 compilation

That worked .. but ..

Re: Boost.Test build

Still facing problem

Re: Still facing problem

Works in VS2008 but some strange errors in VS2005

Re: Works in VS2008 but some strange errors in VS2005

These are the changes I made in the code for VS2005 to build

replies

log in VS2005

VS2008 screenshots and log

Re: VS2008 screenshots and log

Re

Re

Enforcing boost toolset

Re

Linker errors

Re: linker errors

few clarifications

Navigation