Verifysoft Technology GmbH Testwell CTC++ Version 9.1 03 April 2020 Microsoft Visual Studio Integration CTC++/Microsoft Visual Studio Integration ========================================= This readme.txt file describes how to use Testwell CTC++ with Microsoft Visual Studio 2003 and later (when writing this: upto Visual Studio 2019), both from the IDE and from the command line. With "from the IDE" we mean that first the graphical IDE has been started normally and then CTC++ is used graphically in that context. With "from the command line" we mean "ctc-builds" that are invoked from command line something like "ctcwrap ctc-options devenv devenv-arguments-and-options". With VS2010 or later one additional step is sometimes needed to complete the install. That is typically if the basic CTC++ was installed before VS installation or if PATH information gets invalid due to an VS update. See below chapter 1. Install, point (*). Usage may be possible also with VC++ 7.0/Visual Studio 2002, but this is untested. There is a remark on CTC++ usage with the old Visual Studio 6.0 projects. CONTENTS: 1. Install 2. Use 2.1. CTC++ use from command line with Visual Studio 2.2. CTC++ use via the Visual Studio IDE integration 3. Advanced information 3.1. Treatment of precompiled headers, options /Yc, /Yu, /YX 3.2. Treatment of stdafx.cpp file 3.3. Using instrumented static libraries, i.e. .LIBs 3.4. Parallel builds by running many 'devenv's from command line 3.5. Parallel builds by Visual Studio internal means 3.6. Issue in Visual Studio 2010 (and later) rebuild 3.7. Invoking the IDE for CTC++ 3.8. Remark of the used auxiliary directories 3.9. Possible issue with 64-bit msbuild.exe and ctcwrap 4. Uninstall 5. Use with Visual Studio 6.0 projects 1. Install ---------- Here we are in a situation where the basic CTC++ for Windows has already been installed by its setup.exe InstallShield script. That installation has required administrator rights. The CTC++ files have been copied to the CTC++ installation directory. CTCHOME environment variable has been set to point that directory and it has been added to PATH. The additional files needed in this CTC++/Microsoft Visual Studio integration reside at %CTCHOME%\Vs_integ. Now you are assumed to be logged on to your machine with your normal userid, i.e. necessarily not with the userid (that had admin rights), who did the basic CTC++ installation. This installation step makes some recordings to your system that are user-specific, and that's why it is important that you are now logged to your machine with the same userid as you will be when using CTC++ in the Visual Studio. This CTC++ integration is installed with following command: %CTCHOME%\Vs_integ\vs_integ.bat -install If you have many Visual Studio variants ("Visual Studio", "Visual Studio Express", ...) and versions within the variants ("8.0", "9.0",...) in your machine, the installation script asks on which specific variant and version you want the CTC++ integration to be installed. Note that the installation script looks these names from registry, and so, for exmaple, - "Visual Studio"/"8.0" stands for Visual Studio 2005 - "Visual Studio"/"9.0" stands for Visual Studio 2008 - etc. Once the selections are done (if any) the installation script asks you permission to do the installation on the selected variant and version. Into Visual Studio IDE Tools menu two external commands "CTC++ Set/Clean..." and "CTC++ Report..." are added. Environment variable CTCWRAPDIR is set to %TEMP%\ctc. When Visual Studio 2008 or earlier the setting '$(CTCWRAPDIR)' is added to the Visual Studio IDE's internal "Executable Directories" setting for all platforms (Win32, X64, ... as you have them in Visual Studio). When Visual Studio 2010 or later the installation script does not add the setting '$(CTCWRAPDIR)' to Visual Studio's internal PATH. Instead it is added to MSBuild PATH setting already during the basic CTC++ for Windows installation by its InstallShield script. (*) You can for the rare cases manually set MSBuild PATH with following command (open command prompt as administrator): %CTCHOME%\Vs_integ\modify_msbuild_path.bat -install This is needed if you have installed the basic CTC++ before Visual Studio 2010 or later. Running this script will require administrator rights (asks userid and password if needed). Running of this script is not required e.g. if a VS2010 Express was installed before the basic CTC++ and you install VS2010 Professional later. The CTC++ integration does not work, if '$(CTCWRAPDIR)' is not properly set in Visual Studio or MSBuild internal PATH or the environment variable itself is unset. After having executed the vs_integ installation script, you should not need to log out and log in so that the CTCWRAPDIR environment variable would come into effect. Installation refreshes the environment variables for you. 2. Use ------ Generally, CTC++ can be used at Windows with VC++ in various ways. The code instrumenting+compiling+linking can be done basically as follows: a) Directly from command line, e.g. ctc -i m cl -c prime.c ctc -i m cl -c io.c ctc -i m cl -c calc.c ctc cl -Feprime.exe prime.obj io.obj calc.obj or as a one-liner... ctc -i m cl -Feprime.exe prime.c io.c calc.c b) From command line, using a makefile, e.g. nmake -f mymakefile.mak "CC=ctc -i m cl" "LNK=ctc link" c) From command line, applying 'ctcwrap' on a makefile, e.g. ctcwrap -i m nmake -f mymakefile.mak d) From command line, invoking devenv/vcbuild/msbuild, using 'ctcwrap', e.g. ctcwrap -i m devenv mysolution.sln /build debug or ctcwrap -i d vcbuild /rebuild myproject.vcproj Debug or ctcwrap -i fti msbuild myproj.vcproj /t:Rebuild /p:Configuration=Debug e) Graphically from the Visual Studio IDE Here we do not talk more of the a) - c) ways mentioned above. To CTC++ they are not Visual Studio IDE use. They are covered in the general CTC++ documentation (ctcug.pdf) and in the %CTCHOME%\readme.txt. Here we talk of the d) - e) ways of use. The CTC++ integration must be installed, before they would work. In this connection terms like "MSBuild" and "Visual Studio Platform Builder" have also been used. The normal CTC++/Windows (host-only) is usable for these kind of usages, when the build target is normal Win32 or x64 platform. The MSBuild can be used also for developing code for some other platforms, for example for PowerPC. CTC++ can be used in those connections, too, but then CTC++ Host-Target add-on (HOTA) is additionally needed. In such cases the CTC++ run-time library is worked up of the HOTA files and linked to the target executable. Here we talk only of normal 32-bit or 64-bit Windows programs, where HOTA is not needed. 2.1. CTC++ use from command line with Visual Studio There are three commands by which a project (or the whole solution) can be built from command line using Visual Studio - devenv - vcbuild - msbuild Instrumenting (building): - - - - - - - - - - - - - If you would normally (without CTC++) build your program from command line in one of the following ways devenv arguments_and_options_to_devenv vcbuild arguments_and_options_to_vcbuild msbuild arguments_and_options_to_msbuild you simply do the "CTC++-build" as follows ctcwrap options_to_ctc devenv arguments_and_options_to_devenv ctcwrap options_to_ctc vcbuild arguments_and_options_to_vcbuild ctcwrap options_to_ctc msbuild arguments_and_options_to_msbuild And that's all! Examples: ctcwrap -i m devenv mysol.sln /build debug ctcwrap -i d -v devenv myproj.vcproj /rebuild release ctcwrap -i d vcbuild /rebuild myproj.vcproj Debug ctcwrap -i fti msbuild myproj.vcproj /t:Rebuild /p:Configuration=Debug The building takes place with CTC++ according to the dependency rules as enforced by Visual Studio. Into each project directory (if you are building a solution that has many projects) there comes MON.sym file. If you want to get only one symbolfile (and corollary one datafile) over all projects, you can use ctc option e.g. -n abspath\myMON. The build target files (.exe, .dll, ...) get instrumented, overwriting the original non-instrumented targets. The source files remain intact. Test runs: - - - - - - Run your instrumented version of your program normally. When the program ends, CTC++ automatically writes a MON.dat file (to the same directory as the MON.sym resides). The MON.dat file contains the coverage data in binary form. Read the general CTC++ documentation for further advise, - if you want the MON.sym and MON.dat files to have other name or other directory where they will reside (-n option at instrumentation time) - if you want to have the instrumented source files to be recorded in the "CTC++ machinery" (in MON.sym etc.) with full path. Issue here is that Visual Studio often specifies the source file names with their base name only, which is relative to the build directory. You may want to change in the ctc.ini configuration file in 'cl' command block the setting SOURCE_IDENTIFICATION=absolute. It works better in coverage reports, when the source files can be found better and directory specific groupings and TER% summaries can be be calculated. - if you want the MON.dat file write-out to happen at some other time, - if you run tests in some other machine that has not similar directory structure that the build machine had Getting reports: - - - - - - - - - Coverage reports can be obtained in normal manner from the command line using the ctcpost and ctc2html tools. Examples ctcpost MON.sym MON.dat -p profile.txt ctc2html -i profile.txt ctcpost MON.sym MON.dat ..\prj2\MONsym ..\prj2\MON.dat \ -f "*\XMGR_*" -p - | ctc2html 2.2. CTC++ use via the Visual Studio IDE integration Instrumenting (building): - - - - - - - - - - - - - Usage is started with Tools > CTC++ Set/Clean... dialog. In it you set CTC++ mode ON and select the instrumentation options you want to apply in building the instrumented code. This setting applies on all projects and solutions that you build until you set CTC++ mode OFF. Once the CTC++ mode has been set ON, building of the instrumented code is done in normal manner, via the Visual Studio Build menu commands. Whenever Visual Studio internally emits a cl compiler command or a link linkage command, they are catched by the CTC++ integration and made to be "ctc-compile" and "ctc-link". In the beginning suggesting that you use -v (verbose) ctc option for seeing what happens and for understanding the overall CTC++ behavior. Once things work smoothly you can take the -v option away and you get less screen output. The building phase results in instrumented .obj files and instrumented target .exe, .dll, .lib files, as the case may be. They come to the same name and place as the original non-CTC++ build created them. Your original C/C++ source files remain intact. Building with CTC++ also creates or updates symbolfile MON.sym if no -n symbolfile CTC++ instrumentation option is used. It is project-specific and resides in the project directory. (So, if a solution contains many projects in different directories, they all get their own MON.sym files). The symbolfile contains descriptions of the instrumented files. The MON.sym file is needed in generating the CTC++ reports. With -n instrumentation option the symbolfile can be given other directory/name. For example you might want that the symbolfile and datafile are common to all projects in the solution. If you use -n option at CTC++ instrumentation time, you need to specify the .sym and .dat files in the CTC++ Report dialog, because these files are not found from their default places. The CTC++ Set/Clean... dialog has subdialog "CTC++ Clean...". It is used to delete CTC++ files (when default naming) from the selected project. Test runs: - - - - - - Test runs are done in the same way as you would do test runs with the original code. A test run with the instrumented code creates or updates automatically datafile MON.dat at the end of the program execution. The datafile contains in binary form the cumulative code coverage (and timing) data of the program execution. The datafile will reside in the same project-specific directory as the corresponding symbolfile MON.sym. If you are using one machine for building and another machine for running the code, you need to arrange to the other machine similar directory structure as there was in the build machine. Alternatively you have to use CTC_DATA_PATH environment variable for advising CTC++ the directory where the datafile is written. See the CTC++ User's Guide for more of this matter. Getting reports: - - - - - - - - - Use Tools > CTC++ Report... menu command for getting the reports of the currently selected project. If the project directory contains the symbolfile and datafile with their default names MON.sym and MON.dat (no -n option used at instrumentation time), they will be used. If they are not found or if you wish to add symbol and datafiles from other projects, you need to give their names explicitly in the "ctcpost additional options" field. In "ctcpost additional options" field you can also give many symbolfiles and datafiles. You need to use that way when you want to get one combined coverage report of all projects of a solution (if the projects have their own project-specific .sym/.dat files). The report is generated to a ctc-named file into the project directory. Normally, additionally, a viewer (e.g. notepad when a textual report or your machine's default browser when HTML report) is also started on the generated report. When you ask for a HTML report and also start it, for smooth behavior, your browser should be already open. Getting rid of CTC++ instrumentations and of CTC++ generated files: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - For getting back to normal, uninstrumented level of code, just force a rebuild on your code when CTC++ mode is OFF. For getting rid of the files that have been created along with the use of CTC++, you can delete them explicitly from the project directory, or you can use the CTC++ Set/Clean... > Clean... subdialog for deleting the CTC++ generated files (when default names used) from the currently selected project. 3. Advanced information ----------------------- Some technical details are listed here. 3.1. Treatment of precompiled headers, options /Yc, /Yu, /YX Visual Studio projects are often built so that precompiled headers are used. You can keep those settings in your project, but note that CTC++ anyway makes its building (C-preprocessing of the source file, instrumenting the C-preprocessed file, compiling the instrumented code) so that precompiled headers are not used. This comes via the setting in %CTCHOME%\ctc.ini configuration file (in 'cl' block) OPT_NO_COMPILE = ...,/Yc,/Yu,/YX,... See also the upper level readme.txt for issues with precompiled header files. 3.2. Treatment of stdafx.cpp file Visual Studio projects often have this file. And the use of precompiled headers is connected to this file in certain way. It also seems to be the primary role of this file, because the file does not actually have executable code. So, there's no use instrumenting the file. This comes via the setting in %CTCHOME%\ctc.ini configuration file (in 'cl' block) EXCLUDE + stdafx.cpp 3.3. Using instrumented static libraries, i.e. .LIBs If your project target is a static library (a .lib file), it can be instrumented by no means. But note that it contains calls to CTC++ run- time library, which you need to explicitly add to the linkage of your final executable. The library that you need to add is %CTCHOME%\lib\ctcmsnt.lib or %CTCHOME%\lib\ctcmsnt64.lib. There is not similar issue with instrumented dynamic libraries (.dll files). Into them the CTC++ run-time library has already been linked in. 3.4. Parallel builds by running many devenv's from command line Parallel builds can be running in two or more command shells. For example in first command window there could be running cd solu1dir ctcwrap -i m devenv mysol1.sln /rebuild debug and in another command window cd solu2dir ctcwrap -i d devenv mysol2.slm /build release Running two IDEs in parallel and using CTC++ builds in both of them does not generally work. CTC++ usage in one IDE has influence to the builds of the other IDE as well, and the net result is easily a mess. 3.5. Parallel builds by Visual Studio internal means In newer 'cl' compilers there is option /MP for compiling the argument code files in parallel. It is not an issue to ctc, because ctc anyway instruments and compiles the code files from the command line one at a time. In visual studio there is another kind of parallelism: in a multiproject solution build the projects in parallel (normal in multi-core machines). Also this is ok for ctc. 3.6. Issue in Visual Studio 2010 (and later) rebuild Visual Studio 2010 (and later) has started to keep per-project bookkeeping what files were written during the previous build. Somehow magically Visual Studio knows, if the build was with ctc, that ctc.exe was also invoked and it had (created if needed and) written to symbolfile, normally named MON.sym. The Visual Studio bookkeeping is in file ctc.write.1.tlog. In the beginning of each project rebuild, Visual Studio deletes the files that according to its bookeeping were written during the last build. This is a problem to ctc. To overcome this problem, you need to command 'clean' in Visual Studio before commanding 'rebuild'. In 'clean' Visual Studio deletes what it normally deletes, here also ctc's symbolfile and the bookkeeping file ctc.write.1.tlog. In next 'rebuild' things go ok from ctc's perspective. The above 'clean' is a must-to-do in the following situation: Assume you have a multiproject solution and you use one common ctc symbolfile on all of them (not the default project-specific .\MON.sym). If the 'clean' is not done, Visual Studio deletes ctc's symbolfile in the beginning of each subproject rebuild. At the end of the solution 'rebuild' in that common symbolfile there would be only descriptions of the code files that were instrumented after the last symbolfile delete. This Visual Studio behavior has also the following not-so-nice-effect to ctc: In 'rebuild' (or if you have done explicit 'clean') the previous symbolfile is deleted. It means that in the reinstrumentation the code files get new timestamps in the reconstructed symbolfile. (If there had remained the old symbolfile, ctc would have consulted it, and if the reinstrumentation would mean same file description as there was before in symbolfile, ctc would have kept the old timestamp in the new instrumentation of the file). This has the harmful consequence that coverage data of the tests of the previous build cannot be summed up to the tests of this build. In automated production use this thing (coverage results cannot be summed up with previous build) may not be a problem, if the usage model is alway to do a full test cycle with the then-latest build. If you anyway want to keep the previous coverage data, you can before the clean or rebuild copy the previous MON.sym and MON.dat aside, say, to names MON-old1.sym and MON-old1.dat. Then at ctcpost time you give as input ctcpost MON.sym MON-old1 MON.dat MON-old1.dat -p profile.txt and you get the summed you coverage report. The summing up happens ok on those files that have not changed in reinstrumentation. Of the changed files there is coverage data only of the now latest file level. 3.7. Invoking the IDE for CTC++ You should invoke the Visual Studio IDE "normally/without CTC++". Notably, you should not start the IDE from the command line with ctcwrap possible_options_to_ctc devenv Well, this works reasonably, but you start getting mess-ups, if you have parallel builds. 3.8. Remark of the used auxiliary directories The CTC++ Visual Studio IDE usage uses auxiliary directory %TEMP%\ctc. The CTC++ command line integration to Visual Studio uses directories %TEMP%\ctc1, %TEMP%\ctc2\, ... etc. Each parallel "ctc-build" uses its own ...\ctcn auxiliary directory. If for some reason, e.g. abnormal terminations of the ctc-builds, there seems to remain extra/unused ctc1, ctc2, etc. directories, you can simply delete them regardless of their contents. 3.9. Possible issue with 64-bit msbuild.exe and ctcwrap The following behavior (problem) has been verified in a 64-bit Windows 7 machine, which had Visual Studio 2012 installed, and both the 32-bit and 64-bit compilers (cl.exe, msbuild.exe,...) were installed. The project that was compiled generated 64-bit code. When using a ctc-free build: msbuild project.vcxproj ... the build succeeded. In the above the msbuild.exe, cl.exe, link.exe were all 64-bit executables. The following was the problem, e.g.: ctcwrap -v -i m msbuild project.vcxproj ... Some VS layer said "error MSB6006: "CL.exe" exited with code -1073741701." The root reason here is still unclear. It may be related that the ctcwrap machinery inserts to the programs-calling-one-another-chain ctcagent.ex_, which is a 32-bit executable, and the otherwise all 64-bit VS machinery does not tolerate it. The following workaround exists on this issue: - change your context (PATH) so that the msbuild.exe to be called is a 32-bit executable - invoke the ctc-build as follows: ctcwrap -v -i m msbuild project.pcxproj ... /p:Platform=x64 - now the 32-bit msbuild invokes a 32-bit cl.exe cross-compiler, which generates 64-bit code, and 32-bit/64-bit exe conflict is avoided 4. Uninstall ------------ Before uninstalling you may want to ensure that you have turned the CTC++ mode OFF. You may also want to polish your project directories from the CTC++ files that were created there along with the tool use. The following files are "CTC++ files", which you may consider to delete: MON.sym, MON.dat, the ctcpost output listings as you may have generated them (execution profile, untested, summary, timing, XML listings), and the CTCHTML subdirectory. These files can be cleaned with Visual Studio IDE Tools menu command CTC++ Set/Clean... > Clean... Removing (uninstalling) the CTC++/Microsoft Visual Studio integration can be done with following command: %CTCHOME%\Vs_integ\vs_integ.bat -remove The CTC++ commands are removed from Tools menu. Environment variable CTCWRAPDIR is removed from registry. When Visual Studio 2008 or earlier the setting '$(CTCWRAPDIR)' is removed from Visual Studio IDE's internal "Executable Directories" PATH. When Visual Studio 2010 or later the setting '$(CTCWRAPDIR)' is removed from MSBuild PATH when the basic CTC++ is uninstalled. You can manually remove the setting '$(CTCWRAPDIR)' from MSBuild PATH with following command: %CTCHOME%\Vs_integ\modify_msbuild_path.bat -remove The associated files of the integration are removed along with uninstalling the whole CTC++ from your machine. You can also delete the CTC++ commands from your Visual Studio Tools menu manually, e.g. if you have already uninstalled the whole CTC++ from your machine. Select Tools > External Tools > select the CTC++ command one by one and click Delete. 5. Use with Visual Studio 6.0 projects -------------------------------------- The CTC++ Visual Studio 2xxx IDE integration cannot be used with Visual Studio 6.0 IDE, because these IDEs are architecturally different in certain critical respects where the CTC++'s integration relies to. The usage is possible with Visual Studio 6.0 projects as follows: - In the Visual Studio 6.0 IDE do 'Export makefile' on the projects you want to instrument with CTC++. You get projectname.mak files. - Build the project by 'nmake' and prepend the command with 'ctcwrap', something like the following ctcwrap -i d -v nmake -f projectname.mak clean all - Instrumented object files and target executable are born. Symbolfile MON.sym is born/updated. - Run the tests normally with the instrumented executable. At the test run end the datafile MON.dat gets born. - Get the reports by ctcpost and ctc2html utilities, e.g. ctcpost MON.sym MON.dat -p profile.txt ctc2html -i profile.txt -nsb start CTCHTML\index.html - For getting back to the un-instrumented version of your code, rebuild it without CTC++. --- end of readme.txt ---