TestWorks FAQ Frequently Asked Questions about TestWorks

TestWorks is a family of software test tools which, combined or in subsets, represents the state of the art in modern software quality control and testing. This FAQ answers commonly asked questions about TestWorks, about the "solutions" that TestWorks supports, and how to properly apply TestWorks to your situation.

Topics in this FAQ are listed below in categories that you can click to:

General Software Quality Topics

• About Software Quality General comments to help you figure out how to test your software.
• About Testing General comments to help you figure out how to test your software.

TestWorks Specific Topics

• About TestWorks Where TestWorks fits into the picture.
• Regression Testing What regression testing really means, and is and isn't good for.
• Coverage Testing Why coverage testing (completeness checking) is an essential part of a test/quality process.
• Static Testing What you can, and can't expect, from static test techniques.
• TestWorks Internal Architecture Technical details about how TestWorks' architecture helps you build a strong test process.
• About TestWorks Licensing How the TestWorks products are licensed.
• Technical Support What means and forms of technical support you can expect with TestWorks.

Applications of Testworks (Solutions)

• Integration Testing What best to do after unit testing and before functional testing.
• Embedded and Cross-Testing More technical details about two very important sub-areas of testing technology.
• Web Site Testing How to use eValid and Testworks to test your internet website.
• Remote Testing Option Benefits of TestWorks' Remote Testing Option.

About Software Quality (BACK TO TOP)

• Why worry about software Quality?

  Software quality is concerned with how well an application "meets its requirements" and also with how well it works for its users.

  Quality can arise from good design, from good software development process, from good testing, or most commonly, from a combination of all three.

  Good quality software often sells better and lasts longer and makes for better business than "not so good quality" software.

About Testing (BACK TO TOP)

• Why do people bother to test their software? Shouldn't it have been built so it doesn't have any errors?

  The main reason software test methods are used is that they are effective in reducing the defect content of software products. People ought to build high-quality software, but to make sure, you have to test the final product. And, programmers aren't perfect.

  Testing saves lots of money -- some say as much as 100:1 over field-disovered defects.

• Why do we need automated tools at all?

  Because the size and complexity of applications has grown so rapidly, "old style" methods of testing just don't work any more.

  In the late 1990s you really don't have the economic choice any longer NOT to consider test automation -- unless you don't want to test at all!

• Can't we do manual testing like we always did?

  You can, but you'll find that it is difficult and costly and not terribly reliable or effective. In fact, if you are dealing with a complicated program at all, you'll virtually be compelled to do some kind of automated testing.

  Then, too, there are some kinds of analyses that you simply can't consider doing manually, e.g. code coverage analysis.

About TestWorks (BACK TO TOP)

• What is TestWorks?

  Testworks is a collection of software test tools that supports all of the major functionalities of most software test projects, including static analysis, metrics, test file generation, GUI testing, test management, test validation, branch and call-pair coverage analysis, etc.

• Why is TestWorks organized into bundles?

  We put like-function products into a bundle to save you a bundle (lower license fees) and because we know from our own experience that use of one tool, like CAPBAK/X, generally leads to use of a companion tool, like SMARTS.

• Why is it important to purchase your automated testing tools from the long-established technical leader of the industry?

  Like fine wines, the older the better.

  Experience is the best teacher, and SR has learned a lot over the years and has put as much of its real-world test experience as it can into TestWorks.

• How will TestWorks save us time and money?

  You can save 10:1 to 50:1 when you can find a defect and repair it before your product goes to the field. Admittedly, automated testing has some setup costs. Most organizations get into the positive pay-back period after two or three months of TestWorks use.

• How do TestWorks products fit together?

  Very nicely, we think!

  They all have the same look and feel and they work well together because they share many common features and approaches.

• Do the TestWorks tools work across platforms?

  Yes, but you have to be very cautious: nobody's tools work perfectly across platforms, regardless of what everyone thinks!

Regression Testing (BACK TO TOP)

• What is regression testing really good for?

  Regression tests confirm that an application under test works on this run the same way it worked on the last run. If it doesn't, then you *do* have a problem.

• Can I expect to move tests from one platform to another? How much work is it going to be?

  Mostly, the answer is "Yes" if you are very, very careful about how you plan and organize your tests and if the platforms don't differ too wildly!

• Why does everyone make such a big thing about GUI testing?

  GUI testing has two big advantages: it is easy for a user to understand, and a GUI makes an excellent testbed for running tests that have a high likelihood of revealing defects!

• What are the main modes of GUI test capture/playback?

  TestWorks' CAPBAK test capture/playback engine has three main operating modes:
  • TrueTime Mode: The test is played back EXACTLY the way it was recorded. TestWorks' TrueTime Mode includes the powerful Automatic Output Synchronization capability so tests are very reliable without a lot of extra work.
    • PROS: You keep the user's real timing the way it was recorded.
    • CONS: Tests may be excessively sensitive to changes in the GUI.
  • Character Mode: The recorded test uses the built-in OCR engine to synchronize or to capture essential test validation data.
    • PROS: You can base tests on what is written on the screen, independent of font and typesize.
    • CONS: There is a bit of local testware that has to be written to get the most out of the scripts.
  • Object Mode: The test is recorded in such a way that playback is less sensitive to the locations of buttons and other "widgets" on your GUI.
    • PROS: Tests are insensitive to GUI layouts, other "non-critical" formatting details.
    • CONS: Tests may miss out on catching missing or invisible buttons, and they don't preserve the any actual-user timing information.

• How many defects are actually detected by regression testing?

  Nobody knows for sure, but expect that a 5% functionality change in a 1000 suite test will reveal 5-25 new defects.

• I've heard that capture/playback tools are dangerously invasive? Exactly how do they really work?

  CAPBAK/X works with X windows on UNIX by using the XtestExtension1 or Xtrap extension to the X11 server (display driver). You can tell which extension(s) you have using the "xdpyinfo" command on your UNIX machine. CAPBAK/X also uses a special version of the Xt toolkit for ObjectMode recordings from X/Motif GUIs.

  On Windows machines CAPBAK/MSW uses built-in features of the Windows 3.x, Windows '95 or Windows NT operating systems.

  Every kind of recording mechanism is to some extent "invasive," but most users don't consider this very low-level of interaction with the underlying operating system software much of a risk.

Coverage Testing (BACK TO TOP)

• What is test coverage good for?

  Test coverage tells you if your tests missed something. So good test coverage is good to have if you want to be extra-thorough about your testing.

  Manual testing seldom exercises more than one-third or so of the overall structure of code. Your applications go to the field with major sections never executed -- unless you use coverage to make sure you don't have this problem.

• What is the difference between "BlackBox" and "WhiteBox" testing?

  "BlackBox" testing means you don't get to see what's going on inside the application under test. "WhiteBox" testing means you do.

  Most of the time, BlackBox testing means you're doing functional testing. And, WhiteBox testing means you're probably doing coverage analysis.

• What happens if I don't do any WhiteBox coverage testing?

  Sadly, without a really good metric to tell you what you have done and haven't done, you'll probably be badly undertested. You may want to watch out!

  For example, even the best functional testers generally only hit about 30%-50% of the structure of the application they're testing. So, without coverage analysis it may be that upwards of 70% of your application has never been exercised at all when your product goes into field service.

• What is a segment? What is a [logical] branch?

  A segment, sometimes called a [logical] branch, is a piece of code that is always executed as a unit after some piece of program logic happens. For example, an "IF" statement has two segments, the "true" and the "false" outcomes.

  Programs always have at least one segment (the "entry segment), but they may have hundreds and hundreds. Normal-sized programs have 1-50 segments; if the program is much bigger than 50 segments programmers tend to break the program down into smaller pieces.

• Why not just use "statement coverage"?

  Statement coverage turns out to be 100% identical to branch coverage if every piece of logic in your program starts on a new line. Obviously, this never happens, so statement coverage tends to overstate the actual coverage by 50% or more. You can 100% statement coverage and only 50% branch coverage. CAUTION: This overstatement may be dangerous to the health of your software!

• Why do we need automated tools at all?

  Because the size and complexity of applications have grown so rapidly, "old style" methods of testing just don't work anymore.

  In the late 1990s you really don't have the economic choice any longer NOT to consider test automation -- unless you don't want to test at all!

• What is a call-pair?

  When two functions (programs) call one another they make a call-pair.

  The caller function could call the callee a lot of times, and TestWorks' view of coverage analysis holds that you have to check ALL of them.

• Do all these coverage measures have names?

  Sure do. Helps to keep track of the facts.

  Statement coverage is called C0 (we don't support C0; it's too misleading); segment/branch coverage is called C1.

  Module coverage is S0; call-pair coverage is S1.

• Why not just do module coverage?

  You can and we recommend it as a minimum step, but remember: errors in interfaces happen most often when the calling sequence doesn't match up with the called-function's definition. Call-pair coverage overcomes this by requiring every caller-callee pair to be tested at least once.

• What is a test path?

  TestWorks treats test paths as sequences of segments, counted up to a repetition count for a loop. (This stuff can get pretty technical.)

• How many defects are actually detected by coverage testing?

  Nobody knows for sure, but you ought to figure that increasing branch coverage from 50% to around 90% will expose 5-10 defects per 1000 lines of code (KLOC).

• Can I set up my makefiles to do coverage analysis automatically?

  You sure can! You make a one or two line modification to the makefile to tell it how to call the TestWorks instrumenter when you want to "make" an instrumented target. Then you "make instrumented version" rather than "make normal version."

  Some people make instrumentation the normal and only take out the instrumentation prior to ship!

Static Testing (BACK TO TOP)

• Why should I do static analysis of my source code?

  If you can use a simple mechanical check that finds an error inexpensively then you can have a BIG savings over finding it the hard way: from the field!

• How many defects are actually detected by advisor/static testing?

  Nobody knows for sure, but if you use static and metric analyses wisely, you can yield as many as 2-8 defects per 1000 lines of code (KLOC).

• About how many defects are there in code?

  Nobody knows for sure, but you're on firm technical ground if you figure 30-50/KLOC (1000's of Lines of Code) when it is brand-new software. Smart QA folks aim to get defects down close to 1/KLOC; some critical aerospace applications have to get below 0.1/KLOC.

  NOTE: All these numbers are highly approximate!

• Why are software metrics so important?

  Experience shows that the most complicated pieces of a software product often contribute most of the errors. If you know how to distinguish the most-complex pieces and can then concentrate your effort there, you make the best use of your resources.

TestWorks Internal Architecture (BACK TO TOP)

• Why does TestWorks use "C" as its basic command language?

  Because that is the language most nearly universally understood by programmers and testers alike, and because it is very compact and actually quite convenient to use.

• Do I have to be a "C" programmer to use TestWorks?

  Not really. Most of the time you don't even need to edit scripts, and when you do the syntax rules you have to follow are very simple.

• Why are TestWorks license prices so high?

  License fees for TestWorks products, bundles, and the entire TestWorks Suite are very low on a value per function basis, compared with other products that offer less functionality for more money.

  You'll find TestWorks the most cost-effective way to automate your testing.

About TestWorks Licensing (BACK TO TOP)

• What platforms does TestWorks run on?

  TestWorks products run on UNIX platforms (SUN SPARC Solaris, HP-9000, SGI, DEC-Alpha, etc.) and on Windows 3.1/95/98/NT/2000/XP.

• How is TestWorks licensed?

  On UNIX products, we offer LAN-based floating licenses.

  On Windows products, we have user licenses, group licenses, department licenses, and site licenses.

• What is the TestWorks warranty agreement?

  We have a standard 30-day warranty on all of our products.

  Check your license agreement for complete details and limitations.

• Why is there a price difference between UNIX and Windows?

  Because of the licensing. A single floating license on UNIX can serve 2-4 testers, but on Windows each tester will need a license for his own machine.

• How hard is it to install TestWorks?

  Should be easy.

  On UNIX versions there is an "install.stw" that does it all. Your system admin will probably do this and will probably need to have "root/superuser" permission.

  On Windows versions you can install the products yourself using the supplied hands-off installation script.

• How long does it take to install TestWorks?

  Installation on UNIX systems should be 20 minutes or less if you use the install script. Upwards of two days or "never quite right" if you don't.

  Installation on Windows platforms is less than 10 minutes work.

• How much memory do TestWorks products use?

  On UNIX the distribution tapes range in size from 20 MB to 50 MB but that includes on-line documentation and oodles of utilities.

  The UNIX products, like almost all X windows based systems, take 0.5MB to 2 MB of RAM to execute.

  The Windows platform products are shipped as self-extracting zip files that run 2.5 MB To 5.5 MB depending on the version. The Windows products execute in 250-750 KB.

Technical Support (BACK TO TOP)

• What kind of technical support can I expect?

  We respond to ALL incoming calls, E-mails, FAXes, etc. within 24 hours.

• Is training on the TestWorks products available?

  Standard 2-day, 3-day, 4-day, and 4-1/2-day trainings exist for TestWorks. The length varies with how many products you're learning.

• Can we see a demo?

  We will be pleased to arrange for a trial or evaluation of the fully functioning TestWorks products you are interested in. We can even do a "phone demo" for you!

  Contact our sales group to arrange a demo/trial/evaluation.

• Can we evaluate the software in our own environment?

  This really is better than a demo.

  We strongly advocate in situ trials/evaluations so you can see TestWorks working in YOUR environment on YOUR product.

• What if we are not satisfied after the purchase?

  If you are dissatisfied for any reason within the 30-day guarantee period, you can return the product no questions asked. For longer periods, you have to contact sales. But our guideline is that we want happy customers so we will do the right thing if we possibly can.

Integration Testing (BACK TO TOP)

• What is integration testing good for?

  Most of the time you put two or more pieces of a big system together after you've made sure they work separately. Doing this is called integration testing.

  A lot of defects that can hurt you later get discovered in integration testing, and TestWorks helps by insisting on high levels of call-pair coverage during the integration test process.

• What do I have to do to make TestWorks integrate into my software process smoothly?

  First, make sure your software process runs smoothly in the first place. If it does, then TestWorks will add to the existing processes quite nicely.

• Why do I have to keep my software maintenance current?

  You gain two main things: continuous technical support, and you have the option of getting upgraded products at no cost.

  Hint: Once your maintenance lapses it is more expensive to restore it than it would be if you maintained continuity. Normally you're in a "lapsed maintenance" state 60 days after the end of your prior maintenance.

• What if I have problems with TestWorks after 30 days?

  If you are not on maintenance, we will do our best to help. If you are on maintenance, we WILL help! This is a good reason to be on maintenance.

Embedded and Cross-Testing (BACK TO TOP)

• What does cross-testing mean?

  If you're developing on one machine -- a host -- and you're product runs on another machine, that's called cross-development.

  Cross-testing is TestWorks' way of supporting cross-development.

• Does TestWorks test embedded code?

  Indeed, for cross-development host/target type environments we have special kits you can obtain to help you test your embedded code. The kit contains the source of a compatible version of the runtime that you cross-compile to your target so that you can collect coverage data from the application as it runs there.

• How many changes to my code do I have to make for TestWorks to be effective?

  None.

  The coverage analysis tools work with source code modification, but they make "throw away" versions of your code, then feed that directly to your compiler. You never [have to] see the intermediates.

• My frequently asked question isn't answered above? What should I do?

  Send your question to info@sr-corp.com and we'll answer it immediately, and then add it to this page as soon as we can!

Web Site Testing (BACK TO TOP)

Testing a website for functionality and content is an important application of the eValid website suite.

Remote Testing Option (BACK TO TOP)

The Remote Testing Option (RTO) of TestWorks/Coverage gives you the ability to record test coverage data "from the field". This is an ideal way to gain details about how well your beta-test activity is proceeding.

RTO gives you the possibility of collecting coverage data on your product from a wide variety of users at very low overheads.

• For What Languages is the RTO Option Available?

  We can arrange for RTO support for C, C++, Java and COBOL.

• How does RTO work for Java??

  TestWorks RTO support is available for C/C++ and Java applications. You can try out the Java RTO capability on this website. The technology demonstrated collects coverage data from an instrumented Java application that runs on Java-enabled clients and reports coverage to our website. (We E-mail you the coverage report after the fact.)

• How does RTO work for C/C++ and COBOL?

  For C/C++ and Ada and COBOL applications, you provide your beta-site customers with an instrumented version of your application (the overheads are very low, e.g. 2%-5%) and after each use of the product the intermediate tracefile data is Emailed to your site for analysis.