sss ssss rrrrrrrrrrr ssss ss rrrr rrrr sssss s rrrr rrrr ssssss rrrr rrrr ssssssss rrrr rrrr ssssss rrrrrrrrr s ssssss rrrr rrrr ss sssss rrrr rrrr sss sssss rrrr rrrr s sssssss rrrrr rrrrr +===================================================+ +======= Quality Techniques Newsletter =======+ +======= July 2000 =======+ +===================================================+ QUALITY TECHNIQUES NEWSLETTER (QTN) (Previously Testing Techniques Newsletter) is E-mailed monthly to subscribers worldwide to support the Software Research, Inc. (SR), TestWorks, QualityLabs, and eValid WebTest Services user community and to provide information of general use to the worldwide software and internet quality and testing community. Permission to copy and/or re-distribute is granted, and secondary circulation is encouraged by recipients of QTN provided that the entire document/file is kept intact and this complete copyright notice appears with it in all copies. (c) Copyright 2003 by Software Research, Inc. ======================================================================== o Quality Week 2000 - "Ask the Quality Experts!" Panel Summary (Part 2 of 3) o eValid: Changing the Way You Think About WebSite Testing o A Cost-Benefit Model for Software Testing o New TCAT/Java Download Available o Comments on Dekkers' Article (QTN, June 2000) by Linda Miller, with Response by Dekkers o Testing, Brains, Process, and Tools, by David L. Moore o Word Play (Contributed by Ann Schadt) o Advance Program and Call For Attendance, International Symposium on Software Testing and Analysis (ISSTA'00), Portland, Oregon, 22-24 August 2000 o QTN Article Submittal, Subscription Information ======================================================================== Quality Week 2000 - "Ask the Quality Experts!" Panel Summary (Part 2 of 3) Note: This discussion is the approximate transcript of the "Ask the Quality Experts!" panel session at QW2000. The questions were posed by guests to a web page sponsored by Microsoft (courtesy of Panel Chair, Nick Borelli) who voted on each topic. The top-voted questions were answered first. Ask the Quality Experts! Panel Members: Mr. Nick Borelli, Microsoft, USA Dr. John D. Musa, Independent Consultant, USA Prof. Lee Osterweil, UMASS, USA Mr. Thomas Drake, ICCI, USA Mr. Robert Binder, RBSC Corporation, USA *** What is the most important lesson about achieving software quality? After 6 years it's become apparent to me that in order to test well, you have to do a lot of other things well. Requirements analysis, requirements gathering, etc. Most important lesson is to look at the entire process of the system, pay attention to the lessons that have been learned. Understand what you want to build, and why. Have some way of deferring technical decisions until the last possible moment. There are basic strategies that are known and known to work well. Master and deploy all your high-quality systems. As a society, we know how to develop very high quality systems. Be oriented to the user. Consistently meet customer expectations. I've got 1 week before the announced release date, and they just handed me a new build! What do I do? If you feel confident in the testing effort that you've done up until this build, then you're probably okay executing your automation suite and focusing on the specific parts that have changed. If not, you're not okay, and you're not going to make the release date. If you do go ahead and ship, you'll regret it. Rerun your highly automated test regress suite. One week is better than one day. :) I would do a very quick operational profile. Take a couple hours to find out what the customer is going to do most and focus on those. Do the minimum set of tests that run the core functionality. Suggestion from the audience - assess whether you can ship within a week and make that well known. *** Do you think that the internet is secure enough? No. 90% or greater of all users are completely defenseless to hackers on the internet. We are a long way before security outpaces what hackers are able to do. Example: My dad installed a virus checker and didn't enable it, so it was doing him no good. From an engineering and product standpoint we are at a crossroads of starting to support an infrastructure that is both programmable and secure. There is a tremendous amount of e-commerce that goes on without a hitch. The hacker break-ins make the headlines but there aren't enough to flood the news. Do you think that downtown San Francisco is secure enough? Just like the internet, on a cloudy day, the answer is clearly no, on a sunny day, the answer is clearly yes. The internet is a serviceable thing that for a lot of things, works quite well. > Comment from audience - Some folks not using the software and techniques that is out there today to make their systems more secure. Have heard comments like, that won't happen to us. I had an up-to-date virus checker, and scanned the love bug attachment before opening it. It was from someone I knew. Maybe I'll need to wait a week and see what hits the news before opening attachments. Large attachments are scary now. > Question from audience - What about other types of data transmitted on the internet, such as credit card numbers? I have had private information stolen by hackers. Many people will remain unwilling to use it. Some hackers are using the same technology we use to test our systems. > Comment from audience - It's the same at gas stations or restaurants. Yes, but more measures can be taken to ensure that the card user is the card owner when it's in person. *** Does coverage analysis really work? Is it worth the fuss and bother? Depends. A lot of coverage analysis probably isn't worth it. Static, dynamic, etc... a lot of the classical coverage analysis basically takes the source code and trigger test cases, then look back and find out what was covered. When you're doing that type of analysis you really have to be careful what you are looking for. On the dynamic side, you work directly at the executable level, then you're looking for the risk-type results (dynamic run-time errors). Sometimes it's good to find out that you're only covering a small percentage of the code-coverage analysis can be a check to find out how good your test cases are! In my opinion, it works reasonably well for units, but for operational systems it's much more cost effective. A coverage model is not much more than a guess where we think the bugs might be. It tells us that we've turned over the rocks that we think are good rocks to turn over. Two well-known systems (Oc and Tec) were submitted, their source code was available, no coverage analysis had been run. Large, expert development teams with lots of time looked at these rather large and sophisticated test suites. Very knowledgeable users never reached 25% of the code. Coverage analysis guarantees nothing, but it is a necessary minimum criteria! > Question from audience - Is there a specific goal that's good to shoot for percentage-wise? 85% is a reasonable expectation In some cases you should have much higher expectations than 85%. > Question from audience - In looking back to the original question, I suspect the person asking the question isn't doing any code coverage at all. What should he be doing first? When my team did some coverage analysis and saw that it was low, we were able to talk to the developers and show them what we weren't covering, and they were able to guide us to what were critical things that we needed to be testing. Good point. Talking to developers we can figure out what to do with that 25%-50% of code that isn't covered. Yes, it's probably useful to know what the code that isn't being covered is. > Comment from audience - Three things: 1. do a lot of unit testing 2.it can help to prioritize testing 3. as we started to use the static analyzer, the developers' quality went up Static analysis can help you target areas for inspection/review too. (To Be Continued) ======================================================================== eValid: Changing the Way You Think About WebSite Testing Our new eValid(tm) family of WebSite test products and WebSite test services aims to change the way you think about testing a WebSite. This new consolidated offering -- which integrates our unique testing, tuning, loading, and monitoring technology under a single brand name -- is based on eValid's unique Test Enabled Web Browser(tm). The eValid engine runs on Windows 98/NT/2000. eValid is a user-oriented test engine, our eValid monitoring service platform, and the internal technology basis for our WebSite Quality consulting work: producing load experiments, building complete WebSite Test Suites, and doing WebSite page tuning. eValid as a test engine performs essentially all functions needed for detailed WebSite static and dynamic testing, QA/Validation, and load generation. eValid has native capabilities that handle WebSite features that are difficult, awkward, or even impossible with other methods such as those based on viewing a WebSite from the Windows OS level. eValid has a very rich feature set: * Intuitive on-browser GUI and on-web documentation. * Recording and playback of sessions in combined true-time and object mode. * Fully editable recordings/scripts. * Pause/SingleStep/Resume control for script checkout. * Performance timings to 1 msec resolution. * Content validation, HTML document features, URLs, selected text fragments, selected images, and all images and applets. * JavaScript and VBScript fully supported. * Advanced Recording feature for Java applets and ActiveX controls. * Event, timing, performance, tuning, and history charts that display current performance data. * Wizards to create scripts that exercise links on a page, push all buttons on a FORM, and manipulate a FORM's complete contents, etc. * The LoadTest feature to chain scripts into realistic load testing scenarios. * Log files are all spread-sheet ready. * Cache management (play back tests with no cache or an initially empty cache). Try out a DEMO Version of eValid (it has limited functionality but no key is required!) by downloading from: <http://www.soft.com/eValid/Products/Download/down.evalid.html> Or, download the FULL version and request an EVAL key from: <http://www.soft.com/eValid/Products/Download/send.license.html> The powerful eValid LoadTest feature is described at: <http://www.soft.com/eValid/Products/Documentation/eV.load.html> Take a quick look at the eValid GUI and other material about the product at: <http://www.soft.com/eValid/Products/Documentation/eV.GUI.unlinked.html> A detailed feature/benefit analysis of eValid can be found at: <http://www.soft.com/eValid/Products/features.benefits.html> ======================================================================== A Cost-Benefit Model for Software Testing by Rudolf GoeldnerIT Centre of North-Rhine Westphalia's Finance Department Duesseldorf, Germany Summary: This article attempts to illustrate the economic viability of quality assurance measures such as software testing by comparing the costs of such measures with the costs that the user incurs when errors occur at the workplace. Since there are hardly any empirical studies on these costs, with the exception of Compaq's "Rage against the machine" study, this article seeks to estimate these error-induced costs. Obviously, the more users there are, the greater the benefit provided by quality assurance. In other words, quality assurance is worthwhile, irrespective of all other beneficial aspects, as soon as the number of users exceeds a critical figure. Finally, the formula for estimating the error-induced costs incurred by the user is applied to the data from two projects. 1 The status of quality assurance in the software development process Quality assurance measures, such as testing during the software development process, are recognized software engineering methods and are included in every software process model. Software development is based on four conflicting dimensions: functionality, quality, deadlines and costs. The scope for solutions on any given project is restricted with regard to these dimensions. Thus, if the costs and deadline are specified in advance, the functionality and/or quality assurance have/has to be limited. So, quality assurance is restricted by the dimension of costs on the one hand and time-to-market requirements on the other. For software producers, quality assurance especially in the form of software testing entails effort which, in turn, affects the release date for the software product. Moreover, the effort required to detect and correct errors does not really depend on whether these activities have to be performed before the product is used or when it is already being used - with the exception of analysis errors. This is why it is often doubted that the effort invested in quality assurance is necessary. However, the economic viability of quality assurance has to be measured using other criteria, e.g. at the user level, where the software is in productive use. There is no doubt that system crashes and malfunctions cost more effort on the part of the user. Unfortunately, there are hardly any representative studies on this issue. One of the few that do exist is Compaq's "Rage against the machine" study. An extract from this study was available on Compaq's website <http://www.compaq.uk> from 27.05.1999 but has since been removed. 2 Compaq's "Rage against the machine" study from the point of view of quality assurance. 2.1 Technology-Related Anger In March and April 1999 a representative survey of 1,255 employees who use computers at work was carried out in Great Britain. The amazing and worrying result was the "technology-related anger" (TRA) detected by the survey. This type of anger is much more common than the much better known phenomenon of traffic-related anger and results in outbursts of rage that can go as far as actual physical attacks against the computer. The survey questions referred to IT as a whole, i.e. hardware, software and networks. The responses showed: * there is a high level of stress, anger and frustration caused by computer problems; * this leads to substantial costs for the company; * 23 per cent of those questioned said that their work was interrupted at least once a day; and * 20 per cent of those questioned said that they spent up to three hours per day solving IT problems. The damage for the company is not only caused by the fact that staff spend time on computer problems themselves, but also by the psychological stress exerted for the individual employees. Although they may react differently according to their disposition, their reactions as a whole have a negative influence on the atmosphere and productivity within the company. 2.2 CBI Statistics Statistics compiled by the Confederation of British Industry (CBI) show that, on average, it takes one hour per day and user to solve IT problems . (These figures are quoted in the Compaq study, too.) This figure is more or less the same as the result of the Compaq study with regard to the 20 per cent who said that they spent up to three hours per day solving IT problems. Thus, if this figure is transferred to all of the users (i.e. 100 per cent), an average of 0.6 hours per day is used for this purpose. The CBI concludes that these breakdowns cost =A325,000 per person per year. Since 9 million people in Great Britain have a computer at their workplace, the macroeconomic damage caused by the breakdowns comes to almost =A3 225 billion per year. The Compaq study focuses on the psychological effects rather than the costs of IT breakdowns. It also makes no distinction between hardware, software and network failures. But our experience shows that the computer hardware (even the desktop printer) and the network hardware rarely break down in a stable system and certainly not on a daily basis. This means that the main cause is the software. 3 The Cost-Benefit Model 3.1 Error-Induced Costs Incurred by the User As software testing is one of the most important measures in quality assurance I will restrict my reflections to software testing, now. The costs incurred during use are defined as downtimes in person days (PD), with one PD equaling eight hours. Only serious errors, i.e. those errors that cause the user effort, are taken into account. There is a link between the costs (C) during use, the number of errors (ER) in the software and the number of users (U). If the average effort during use per error and user is defined as B(m) the following is true: C =3D B(m) * U * ER (PD) 3.2 Optimistic estimate of the error-induced costs incurred by the user The next step is to attempt to produce an optimistic estimate of the average effort B(m) per error and user. This process does not cover follow-up effort of the type necessary in the case of data being lost due to software bugs. Though this follow-up effort can take on huge dimensions, it can hardly be calculated without empirical studies. The average effort per error and user is determined as follows: B(m) =3D e * p (PD per error and user) where e is the average effort in PD per error and p is the proportion of users that is actually affected by the error. For e, i.e. the effort invested by the user per error, empirical values can be quoted. Users hardly ever spend more than half a day dealing with an error. However, the minimum time required is normally not less than half an hour. In this case, the following is true: 0.0625 < e < 0.5 (PD per error and user) The proportion (p) of users affected by an error should really be 100 per cent but this is not the case because: 30 per cent of users may be absent or may perform another task 10 per cent may hear about the error from other users and do not use the erroneous function 10 per cent may get an error message from the hotline in time and do not use the erroneous function Thus, p could be said to equal 50 per cent or 0.5. The result would be 0.03125 < B(m) < 0.25 (PD per /error and user) and the minimum costs would be C(min) =3D 0.03125 * U * ER (PD) and the maximum costs C(max) =3D 0.25 * U * ER (PD) The average value is EX(B(m)) =3D 0.14 (PD per error and user) and the average costs C(m) =3D 0.14 * U * ER (PD) If a software system that has 20 serious errors is delivered and then used by 400 people, the minimum costs will be 250 PD, average costs 1,120 PD and maximum costs 2,000 PD. If there were 2,000 users, the costs would be 5 times as high. The average costs would then come to 5,600 PD or 28 person years and, as said, this model calculation doesn't even cover the considerably larger effort required when data is lost due to software bugs. 3.3 Financial Benefits of Software Testing It is now possible to compare the costs incurred by the user with the costs of software testing in order to assess the efficiency. Obviously, the software testing costs C(ST) must be lower than the costs incurred by the user if no software testing is done. ER(ST) stands for the errors detected through software testing. C - C(ST) =3D B(m) * U * ER(ST) - C(ST) > 0 In other words, software testing is also worthwhile if the number of users exceeds a certain, critical level U(crit), as follows: U > C(ST) / B(m) * 1 / ER(ST) =3D U(crit) We shall now use data regarding two projects in North-Rhine Westphalia's finance department to conduct a model calculation. The developers' test (software developers' white box test) was followed by user acceptance testing, during which serious errors were discovered ER(ST). The effort was defined as C(ST). 386 errors were detected on Project A and the effort involved came to 1.820 PD. The product is used by 200 people. The average costs incurred by the users due to this error would have been 10.808 PD. Thus, the benefit from using user acceptance testing totaled 8.988 PD. Or, to put it another way, user acceptance testing would have even been worthwhile if only 34 people had been using the product. On Project B, the results were even more drastic. 604 errors were discovered and the effort required came to 1.362 PD. This product is also used by 200 people. The average costs incurred by the users due to this error would have totaled 16.912 PD. This means that the benefit from using user acceptance testing was 15.550 PD. Or, in other words, user acceptance testing would have even been worthwhile if only 17 people had been using the product. On both projects the costs of user acceptance testing were less than 20 per cent of the total costs. The method used to perform the user acceptance testing was "SQS-TEST" by Software Quality Systems AG, Cologne. 4 Outlook It is clear that the U (user) value is the significant factor. The higher U is, the greater the benefit and the justifiable effort that can be invested in software testing. If the average value is used for B(m), the justifiable effort for software testing can be calculated as follows: C(ST) =3D 0.14 * ER * U (PD) It is not as easy to estimate the effort incurred by users when software bugs lead to data loss or corruption. Such estimates necessitate company-specific, empirical studies, which, in view of the rationalization potential that could be tapped using error-free software, are really crucial. If, for whatever reason, empirical studies cannot be carried out, the only real alternative is the "minimum marginal costing" method presented here. ======================================================================== New TCAT/Java Download Available We are pleased to announce that we have completed a new build of TCAT/Java, Ver. 1.3, for Windows 98/NT/2000 build, and have made this build available on our website. All current TCAT/Java users should download this new version; all existing license keys will work with the new release. This new TCAT/Java version includes: o Runtime. There is an improved java runtime that includes more buffer size options and more efficient operation. Typically the slowdown for fully-instrumented Java applications that use the "infinite buffering" option is less than 5% additional execution time compared with that of the non-instrumented Java application. The new build provides buffer sizes of 1, 10, 100, 1000, 10,000, 100,000 in addition to the "jruninf" (infinity) option. You use the non-infinite buffering in case you have an application that terminates so abnormally that the last buffer load is lost. The internal buffers in all of the other options are dumped automatically after the indicated number of "hits", and are also flush automatically at the end of the execution. o Examples. The new release includes updated example programs that show how TCAT/Java operates. The examples include: o "Fib.java": A small, 2 module, 6-segment Java application that calculates Fibonacci numbers. Though very small, this Java application can generate a lot of segment hits; for example, Fib(20) generates 40,588 hits. o "Metalworks.java": A medium sized, 115 segment, 117 module Java application. A typical use of this application generated 7,684 hits. o "SwingSet.java": A large sized, 709 segment, 232 module Java application that provides a wide rage of GUI elements. A minimal exercise of this collection of GUI elements yielded 169,605 hits. o Other Improvements. A range of other minor changes and repairs to improve operation and simplify processing have been made throughout the product. ======================================================================== Comments on Dekkers' Article (QTN June 2000) by Linda Miller with Response by Dekkers. From: "Linda Miller" To: QTN Subject: Re: Quality Techniques Newsletter (QTN) -- June 2000 Issue Date: Tue, 27 Jun 2000 I totally disagree with the article Real World Strategies for Improving the Test Process presented by Marco Dekkers. I suppose Mr. Marco thinks the CMM is also a waste of time since it has maturity levels and questionnaires? The TMMsm (Testing Maturity Model) developed by Illinois Institute of Technology by highly recognized technology experts with more credentials and degrees than anyone I have encountered in testing. Everyone has their opinion and has the right to voice it, however they should stick to the facts. My company uses the TMM and it has made a significant impact on how we perform testing. It establishes repeatable measurable standards and procedures that can be integrated throughout the development life-cycle. We work with the IIT and have developed all the necessary documents and an assessment that has proven to be very successful. There are many questions in the assessment, but there is a purpose and theory for each question. Also IIT and MST Lab, recognizes and insist on organization wide support for any process as does the CMM. Thank you, Linda Miller MST Lab, Inc. (314) 487-1178 o o o o o o From: MDekkers@kza.nl To: QTN Date: Wed Jun 28 2000 Subject: Response to Ms. Linda Miller's Comments Thank you for your frank reaction to my article. The purpose of this publication was to stimulate discussion on the subject of testprocess improvement. It is only natural that different people have different perspectives. Nevertheless, I think this means that there is an opportunity to learn from each other. I would like to react in more detail with regard to some point you make. First of all I have never stated that the use of models for test process improvement is "a waste of time". On the contrary my article states that organizations can benefit from using or or more models, if they are used with care. I have however seen countless instances where a model was followed "blindly", without regard for the company's business and IT goals and specific circumstances. As I write in my article I view models as useful tools. Also my criticism of questionnaires is not addressed at the principle of using them, but at the lenght of some of these. This criticism was not specifically targeted at the TMM. However, some other models, including the TPI, have lengthy questionnaires for which no theoretical of practical purpose is given. In fact we ourselves us a compact questionnaire when assessing the testprocess (which I mentioned in the article). Assessments would be very hard to perform without any form of questionnaire. Maturity levels can be useful to assess where an organization stands and to help in establishing goals. However, reaching a certain maturity level can never be a goal in itself. Always there has to be a relationship with the business goals. Therefore I encourage organizations to establish goals for testprocess improvement which are linked to their business goals. If these goals can be satisfied by reaching a certain maturity level there is no reason not to do so. Again, however, the question has to be asked what is accomplished when a higher level is reached. Management would not be doing it's job if such questions where not asked. I am glad to hear that you have had success using the TMM. If it works, I encourage you to use it. You wrote that you totally disagree with my article. Could you please let me know what other point besides my coment on the length of questionnaires you disagree with? Also, are there any aspects in the article that you find to be correct? In closing I would like to thank you again for taking the time to react. I would be very thankful if you could further elaborate on your thoughts regarding the article and the subject of testprocess improvement. Also I am very interested in how you made testprocess improvement work for your company. With kind regards, Marco Dekkers ======================================================================== Testing, Brains, Process, and Tools by David L. Moore Independent Testing Consultant This white paper will discuss the inextricable link between the need for testers to think about what they are doing, the process on which to base this thought, and the use of tools to assist them in these tasks. Why do testers need brains? Today's tester has a job that fully parallels that of any developer. To be effective, testers should be on the project from the start to the finish ie. End-to-End testing. It is widely understood that testing can no longer be dismissed as something that can happen at the end of a project. Certainly not without significant risk of failure. The variety and complexity of tasks that a modern tester must perform certainly means that good testers must be smart people. And, obviously, testers require a high degree of cunning and a willingness to "try something" different just to be effective at their jobs. As they say, "If you do what you've always done, you'll get what you've always got". Testers need to decide what to do and what not to do. Tests need to be designed! Why do testers need process? Complex tasks are often broken into numerous smaller less complex tasks. As a result they need to be part of a process to be performed effectively, analyzed and improved upon. Process provides a focus for communicating and tying together the component tasks into an effective activity. It is simply not possible for the average human to keep it all in their head. This applies to testers and indeed any professional performing a complex job. Because testers are often involved in tasks and outcomes that have an impact on customers, they are required to be professional and beyond reproach more visibly than anyone else in their organization. Processes on which to base testing are an essential and an easy starting point for achieving this credibility. Why do testers need tools? Testers need tools for all the same reasons that developers need tools; to work faster, more efficiently, and to be freed from mundane tasks. Also, there are some things that cannot be done without a tool. If the testing effort is not constantly proportional to the development effort ie. testing is being increasingly left behind, then, more often than not, some tests don't get done. As this gulf increases, the likelihood of important functionality going untested and the project's risk of failure also increases. Bearing in mind that today's testers are smart, they aren't going to be interested in doing the same old thing over and over again. So they need tools to apply to the mundane and time-consuming tasks. Once again, the sophistication of the tester's role demands that some of the complex issues be captured in places other than in their heads. Tools, for example. What happens when you don't have process and brains before getting a tool? The effect is somewhat similar to throwing confetti into the wind. The first thing that happens is that the task spreads out and then it starts to go backwards. The initial direction is rapidly lost. The reason for this is simple. Tools are inherently flexible. They always will be. They have to be. The market dictates that they are customizable and it is an economic reality that these tools need to be sold to more than just one customer and no two customers are alike. Without strong direction a tool will be purchased, installed, and all of its features will be investigated and implemented, regardless of whether these features really help to achieve the particular task at hand. Why is it important to have process and brains before getting a tool? * A tool must do the right job for you. If you don't know what that job is, then no tool, of any nature, will provide you with that solution. * Testing is far from just a mechanical task. It is not possible to remove the tester's expertise from testing. * A tool must not waste your time. You must have the time available to implement a tool. Don't expect a tool to make time for you out of thin air when a deadline is tight. * A tool must be cost effective. Ignoring that time costs money, the fact of the matter is that many tools are very expensive and their cost effectiveness must be clearly understood before adoption. * A tool must actually be used to be effective. If it is not clear how to use the tool, through a well defined process, then most people don't have the spare time to figure it out. * A tool consumes resources. From people, to shelf space, to CPU cycles, to hard disk space etc. If the tool cannot live happily with your resources, then it is going to detract from the goal at hand. How do you know if you are ready to buy a tool? If you can answer "yes" to the following questions: * Do you have a documented process that works? * Do you have the metrics to prove the need to stakeholders? * Are your testers complaining about the proportion of mundane to interesting tasks? * Do you have the money to waste? * Are your testers mature enough to focus on applying your process to the tool rather than exploring the tool's capabilities? * Do your staff think critically rather than perform mechanically? * Are you prepared to try again and refine the use of the tool when it doesn't work the first time? * Do you have understanding management? * Have you identified a pilot project, that has not yet started, and have you included tools in the project plan? * Do you have the resources for tool evaluations? How should you implement a tool? * It should be done in addition to an already satisfactory level of testing. Don't do anything critical with your tool on the pilot project/s. Be prepared to exclude its results from your deliverables. * It should be done in parallel with your existing process. Don't discard existing manual routines until you have proof that the automatic ones are at least as good, hopefully better/faster. * You should be prepared to run the tool implementation as a separate project with the usual project planning principles applied to it. * It should be implemented with close monitoring and carefully thought out metrics. You need visibility throughout the project to tweak it along the way and reach concrete conclusions at the end for postmortems. * It should be used where appropriate. Don't expect to do everything with the tool and be disappointed. Expect to do a little with the tool and be pleasantly surprised. * It should be done on a project that has plenty of time, money and is low risk. * The tool should be implemented with appropriate training for the users. Don't go to the trouble of doing the right thing up to now, and then throw a shrink-wrapped box to your testers. Conclusions Modern testing demands that testers are professionals with a significant degree of intelligence. The ever increasing size and complexity of the task means that intelligence alone is not enough to keep pace with what is required of the job. A process is required as a starting point and as a mode of communicating direction throughout an organization. Tools are a useful mechanism for handling complex processes and voluminous tasks. The testing solution for today, and the future, is a carefully thought out balance of brains, process and tools. ======================================================================== Word Play (Contributed by Ann Schadt) *** Part I: There are definitely some creative people out there... The Washington Post recently published a contest for readers in which they were asked to supply alternate meanings for various words. The following were some of the winning entries: - Abdicate (v.), to give up all hope of ever having a flat stomach. - Carcinoma (n.), a valley in California, notable for its heavy smog. - Esplanade (v.), to attempt an explanation while drunk. - Flabbergasted (adj.), appalled over how much weight you have gained. - Negligent (adj.), describes a condition in which you absentmindedly answer the door in your nightie. - Lymph (v.), to walk with a lisp. - Gargoyle (n.), an olive-flavored mouthwash. - Coffee (n.), a person who is coughed upon. - Flatulence (n.) the emergency vehicle that picks you up after you are run over by a steamroller. - Balderdash (n.), a rapidly receding hairline. - Semantics (n.), pranks conducted by young men studying for the priesthood, including such things as gluing the pages of the priest's prayer book together just before vespers. - Rectitude (n.), the formal, dignified demeanor assumed by a proctologist immediately before he examines you. - Marionettes (n.), residents of Washington who have been jerked around by the mayor. - Circumvent (n.), the opening in the front of boxer shorts. - Frisbatarianism (n.), The belief that, when you die, your soul goes up on the roof and gets stuck there. *** Part II: The Washington Post's Style Invitational also asked readers to take any word from the dictionary, alter it by adding, subtracting or changing one letter, and supply a new definition. Here are some recent winners: - Sarchasm: The gulf between the author of sarcastic wit and the reader who doesn't get it. - Reintarnation: Coming back to life as a hillbilly. - Giraffiti: Vandalism spray-painted very high. - Inoculatte: To take coffee intravenously. - Osteopornosis: A degenerate disease. - Karmageddon: It's like, when everybody is sending off all these really bad vibes, right? And then, like, the Earth explodes and it's like a serious bummer. - Glibido: All talk and no action. - Dopeler effect: The tendency of stupid ideas to seem smarter when they come at you rapidly. - Intaxication: Euphoria at getting a refund from the IRS, which lasts until you realize it was your money to start with. ======================================================================== Advance Program and Call For Attendance International Symposium on Software Testing and Analysis (ISSTA'00) Portland, Oregon, 22-24 August 2000 Workshop on Formal Methods in Software Practice (FMSP'00) Portland, Oregon, 24-25 August 2000 Sponsored by ACM SIGSOFT ISSTA is the leading research conference in software testing and analysis, bringing together academics, industrial researchers, and practitioners to exchange new ideas, problems, and experience. FMSP aims to bring together researchers and practitioners from around the world to work on the problems of the practical use of formal techniques. FMSP is a co-located conference to ISSTA. Organizing Committee: General Chair: Debra J. Richardson, University of California, Irvine, USA. ISSTA Program Chair: Mary Jean Harrold, Georgia Institute of Technology, USA FMSP Program Chair: Mats Heimdahl, University of Minnesota, USA Conference Home Page: http://www.ics.uci.edu/issta-fmsp ISSTA Home Page: http://www.cc.gatech.edu/~harrold/issta00 FMSP Home Page: http://www.cs.umn.edu/crisys/fmsp/ On Line Registration: http://regmaster.com/issta2000/ Early Registration Before July 28, 2000. Reduced rate for ACM SIGSoft members and participants who attend both ISSTA and FMSP Hotel Information: Portland Marriott Downtown Reservations should be made before July 28 by calling: Marriott reservations: 1-800-228-9290 or Portland Marriott Downtown: 1-503-226-7600 and identifying "ACM" as the organization and "ISSTA" or "FMSP" as the conference. Group rates are available three days prior/post to conference dates, subject to availability. ISSTA ADVANCE PROGRAM Session I: ISSTA Opening and Keynote Address Welcome: Debra J. Richardson, General Chair (University of California, Irvine) Opening Remarks: Mary Jean Harrold, ISSTA Program Chair (Georgia Institute of Technology) Analysis is necessary, but far from sufficient: Experiences building and deploying successful tools for developers and testers Invited Speaker: Jon Pincus (Software Design Engineer, PPRC, Microsoft Research) Session II: Static Analysis Chair: Frank Tip (IBM TJ Watson Research Center) Formal Analysis of Network Simulations, by: Karthikeyan Bhargavan, Carl A. Gunter, Moonjoo Kim, Insup Lee, Davor Obradovic, Oleg Sokolsky, and Mahesh Viswanathan (University of Pennsylvania) A Relational Method for Finding Bugs in Code, by: Daniel Jackson and Mandana Vazir (Massachusetts Institute of Technology) Putting Static Analysis to Work for Verification: A Case Study, by: Tal Lev-Ami (Tel-Aviv University, Israel), Thomas Reps (University of Wisconsin), Mooly Sagiv Tel-Aviv University, Israel), Reinhard Wilhelm (University of Saarlandes) Session III: Testing Object-Oriented Software and Components Chair: Thomas Ball (Microsoft Research) Automated Testing of Classes, by: Ugo Buy (University of Illinois, Chicago), Alessandro Orso and Mauro Pezze' (Politecnico di Milano, Italia) OMEN: A Strategy for Testing Object-Oriented Software. Amie L. Souter and Lori L. Pollock (University of Delaware) UML-Based Integration Testing. Jean Hartmann and Claudio Imoberdorf (Siemens Corporate Research), Michael Meisinger (Technical University, Munich, Germany) On Subdomain, Testing, Profiles, and Components. Dick Hamlet (Portland State University) Session IV: Real Time and Process Chair: Pascale Thevenod-Fosse (LAAS- CNRS, France) Requirements-based Monitors for Real-Time Systems. Dennis K. Peters (Memorial University of Newfoundland) David L. Parnas (McMaster University) Classification Schemes to Aid in the Analysis of Real-Time Systems. Paul Z. Kolano (Lockheed Martin), Richard A. Kemmerer (University of California, Santa Barbara) Verifying Properties of Process Definitions. Jamieson M. Cobleigh; Lori A. Clarke; and Leon J. Osterweil (University of Massachusetts, Amherst) Session V: Empirical Studies Chair: Antonia Bertolino (CNR Italia) Prioritizing Test Cases for Regression Testing, by: Sebastian Elbaum (University of Nebraska, Lincoln) Alexey G. Malishevsky & Gregg Rothermel (Oregon State University) Which Pointer Analysis Should I Use? Michael Hind (IBM TJ Watson Research Center), Anthony Pioli (Register.com) Comparison of Delivered Reliability of Branch, Data Flow and Operational Testing: A case study. Phyllis G. Frankl and Deng Yuetang (Polytechnic University) Minimizing Failure-Inducing Input. Ralf Hildebrandt and Andreas Zeller (University of Passau, Germany) Session VI: State of the Art and Future Directions Report Chair: Mary Lou Soffa (University of Pittsburgh) Finite State Verification: A New Approach for Validating Software Systems. Invited Speaker: Lori Clarke (Professor, Computer Science Department, University of Massachusetts, Amherst) Session VII: Testing Chair: Istvan Forgacs (Balthazar Ltd., Hungary) A Framework for Testing Database Applications. David Chays, Saikat Dan, and Phyllis G. Frankl (Polytechnic University), Filippos Vokolos (Lucent Technologies), Elaine J. Weyuker (AT&T Labs Research) jRapture: A Capture/Replay Tool for Observation-Based Testing. John Steven, Pravir Chandra, and Andy Podgurski (Case Western Reserve University) Testability, Fault Size and the Domain-to-Range Ratio:An Eternal Triangle. Martin R. Woodward (University of Liverpool), Zuhoor A. Al- Khanjari (Sultan Qaboos University, Sultanate of Oman) Black-Box Test Reduction Using Input-Output Analysis. Patrick J. Schroeder and Bogdan Korel (Illinois Institute of Technology) Session VIII: State of the Art and Future Directions Report Chair: Michal Young (University of Oregon) Testing Component-Based Software Invited Speaker: Craig H. Wittenberg (Development Manager, Component Applications Group, Microsoft Research) Session IX / FMSP Session I: FMSP Opening and Keynote Address Welcome: Debra J. Richardson, General Chair Opening Remarks: Mats Heimdahl: FMSP Program Chair (University of Minnesota) Model Checking Java Programs Invited Speaker: David Dill (Associate Professor, Computer Science Department, Stanford University) Session X: Concurrency Analysis (FMSP Attendees welcome) Chair: Joanne Atlee (University of Waterloo, Canada) Slicing Concurrent Programs. Mangala Gowri Nanda (IBM Research Laboratory, India), S. Ramesh (Indian Institute of Technology, Bombay, India) Improving the Precision of INCA by Preventing Spurious Cycles. Stephen F. Siegel and George S. Avrunin (University of Massachusetts, Amherst) A Thread-Aware Debugger with an Open Interface. Daniel Schulz and Frank Mueller (Humboldt University, Berlin, Germany) ISSTA Closing Symposium Wrap-up: Program Chair: Mary Jean Harrold (Georgia Institute of Technology); General Chair: Debra J. Richardson (University of California, Irvine) ======================================================================== ------------>>> QTN ARTICLE SUBMITTAL POLICY <<<------------ ======================================================================== QTN is E-mailed around the middle of each month to over 9000 subscribers worldwide. To have your event listed in an upcoming issue E-mail a complete description and full details of your Call for Papers or Call for Participation to "ttn@sr-corp.com". QTN's submittal policy is: o Submission deadlines indicated in "Calls for Papers" should provide at least a 1-month lead time from the QTN issue date. For example, submission deadlines for "Calls for Papers" in the January issue of QTN On-Line should be for February and beyond. o Length of submitted non-calendar items should not exceed 350 lines (about four pages). Longer articles are OK but may be serialized. o Length of submitted calendar items should not exceed 60 lines. o Publication of submitted items is determined by Software Research, Inc., and may be edited for style and content as necessary. DISCLAIMER: Articles and items are the opinions of their authors or submitters; QTN disclaims any responsibility for their content. TRADEMARKS: eValid, STW, TestWorks, CAPBAK, SMARTS, EXDIFF, STW/Regression, STW/Coverage, STW/Advisor, TCAT, and the SR logo are trademarks or registered trademarks of Software Research, Inc. All other systems are either trademarks or registered trademarks of their respective companies. ======================================================================== -------->>> QTN SUBSCRIPTION INFORMATION <<<-------- ======================================================================== To SUBSCRIBE to QTN, to CANCEL a current subscription, to CHANGE an address (a CANCEL and a SUBSCRIBE combined) or to submit or propose an article, use the convenient Subscribe/Unsubscribe facility at: <http://www.soft.com/News/QTN-Online/subscribe.html>. Or, send Email to "qtn@sr-corp.com" as follows: TO SUBSCRIBE: Include this phrase in the body of your message: subscribe Email-address TO UNSUBSCRIBE: Include this phrase in the body of your message: unsubscribe Email-address NOTE: Please, when subscribing or unsubscribing, type YOUR email address exactly as you want it to be used. Unsubscribes that don't match an email address on the subscriber list are ignored. QUALITY TECHNIQUES NEWSLETTER Software Research, Inc. 1663 Mission Street, Suite 400 San Francisco, CA 94103 USA Phone: +1 (415) 861-2800 Toll Free: +1 (800) 942-SOFT (USA Only) Fax: +1 (415) 861-9801 Email: qtn@sr-corp.com Web: <http://www.soft.com/News/QTN-Online> ## End ##