So, today we will discuss about the acceptance strategies and acceptance stress methodsand how do we plan for the acceptance test and what are the important steps involved inthe acceptance testing, what are the things to be tested and how do we plan for its; wewill discuss about these aspects in today's lecture.
(Refer Slide Time: 01:42)
So, acceptance test is basically the final test in qualification. As I mentioned this isseparated from the validation as it is conducted by stakeholders and not system engineersand if this fails the system must be changed. So, if the acceptance test fails thendefinitely then this system has to be changed except for a few exceptions; most of thetimes we will have to change the system, because this is not acceptable to the customer.And the threshold of acceptance need to be defined in the beginning itself; so, we cannothave a threshold identified after the system design. So, what is the threshold values forvarious system design or the system aspects or the system performance need to bedefined well before the test and or even before the beginning of the system design, weneed to ensure that the thresholds are define.So, that there is no ambiguity of the thresholds and there is no changes in the thresholdafter system has been designed or based on the actual system design, we cannot changethe threshold; so, it has to be predefined. And acceptance test basically will focus on theuse of a system by a small group of people or a representative group because this is notdone by the system engineers, we need to have a group of people who will be the userssince it is not possible to have a very large group of people testing the system, we need toidentify a small group of people who can actually do the testing and then find outwhether it actually meets a requirements.
So, various aspects of the system used will be tested by these set of people and then if itmeets the threshold values, then it will be accepted. The two main questions asked hereis basically what to test, and how to test? So, what are the things to be tested in theacceptance test, and how to do this testing? So, these are the two important aspects in theacceptance testing.(Refer Slide Time: 03:49)
The question of what to test is the first one; basically the answer is everything possible.So, there is no restrictions on this, so whatever the things we can test; we can do thetesting that is everything possible can be tested. And the key question should be asked orwhat should be test? And what did we forget to test? So, this is also very important whatshould we test is an aspect which we should take care of in the testing, but moreimportantly what did we forget to test is also important.Sometimes we may forget to test some important parameters and that may lead to afailure in the system. There are many case studies which actually showed that has someof the aspects were tested and exactly these parameters became the cause of failure in thesystem. Therefore, we need to see that it is not only important that what are the things weare testing? But what are the things we forget to test? Also is very important.So, in this case the acceptance test they replace the development testers with users. So,this I already mentioned; so, the development in testers are different from the users and
therefore, and the acceptance test need to be conducted by the users only, this is not doneby the developmental testers.So, that part need to be taken care of and it is not possible to have a exhaustive repetitionof the test; mainly because there are lot of test to be conducted and there are many subsystem, components and interfaces. So, it is not possible to have an exhaustive test of thesystem at the acceptance level. So most of the time, we will have to rely on the past testalso. So, some of the data from the past test basically in the qualification stage and othersub system integration stages, some of the test data will be used here and that lot ofreliance on these data will be there during the acceptance test.So, depending on the type of the system and depending on the nature of the testrequirement, the designers can actually decide; what are the important test to beconducted during acceptance? And what are the data to be used from the previous test?So, that is important here because there are lot of things to be tested. So, this is may notbe possible always during the acceptance test.(Refer Slide Time: 06:07)
And the results of the acceptance testing; so when we will do an acceptance test basicallythe idea is to see, whether it can actually meets the requirement of the stakeholders.So, what kind of results we can obtain from here, after the end of the test stakeholderscan actually accept the system as it is. So, that is one option the system can be accepted
as it is without any major changes. So, most of the parameters are tested and theyactually meet the acceptance threshold, the stakeholders can accept the system as it is.But then another option is that accept it subject to certain changes. So, there are fewchanges needed in the system based on the testing acceptance testing the stakeholderscan suggest some changes and then accept it based on these changes. So, then in this casethe system will be accepted, but they may request for some changes and the stakeholderscan start using the system, but changes will be implemented later based on theacceptance test results; that is a second option.The third option is that accept; it only after changes this, the system cannot be acceptedin the present status and deem to be some changes and the system can be accepted onlyafter making the changes. So, in this case compared to the previous one here the systemwill not be used by the stakeholders, they will wait for the changes to be made and thenonly it will be accepted and the last option is not too except it.So, the stakeholders can simply say that it is not meeting the requirements and therefore,we cannot accept the system and it needs to be redesigned. So, there is another optionhere the system need to be redesigned and then the acceptance test to be repeated and theonce that is qualified after the acceptance test, then only it will be accepted by thestakeholders, so these are the four options for stakeholders. So, based on the acceptancetesting they will decide either to accept the system as it is because no need of any morechanges.So, in this case the stakeholders can start using the system without any changes, but thesecond option is that they will ask for some changes or they found out there are somesmall changes to be made either in the interface or in the functional performance, whichare minor and does not really affect the performance of the system.So, in this case the stakeholders will accept the system as it is with the minor changes.So, the minor changes will be intimated to the designers and designers will make themodifications and add to the existing system. So, the stakeholders will start using thesystem with even though the minor changes are needed and these minor changes; willcome as a next version or a updation of the existing system. The third one is the systemmeets most of the requirements almost it qualifies in most of the cases, but require some
changes which are very important and the system can be accepted only after thesechanges are made.So, in this case the stakeholders dont know use the system they will ask the designers tomake the changes and then give it to them for using. So, that is the third option and thelast one is reject the system, so it is not qualifying any of the or most of the requirementsare not met by the system. So, in this case they will reject the system and ask thedesigners to redo the system or redesign the system.And then it has to go through the integration and qualification stage and the qualificationtest as well as the acceptance test to be repeated and then it will be accepted. So, theseare the various options for the stakeholders after the acceptance test, but there are somecases where there are few acceptance, where the system will be accepted even if thereare some problem with the system.(Refer Slide Time: 10:08)
So, here these are the special cases the stakeholders accept the system without completepositive acceptance test. So, here even though the system is not completely satisfying therequirements, the stakeholders will accept the system. This is because the existing variantis causing too many problems, so whatever they are having at present; it is causing toomany problems and hence a partially improved product accepted and released with a noteto engineering team to complete the given task quickly.
So, here they will accept it because the previous version or whatever is existing atpresent is causing too much of problems. So, they want to have many improvement uponthat one is acceptable to them. So, they will start using it, but they will give a note to thedesign team to ask him to change the design or asking to improve it. And system isunacceptable in some cases though it is meeting the most of the requirement; it is notacceptable because it causes more problems than existing system.So in that situation they may not accept it, though it actually meets the many of therequirement. So, if it causing more problem than the present system then it will not beaccepted and in most of these test, they will make two assumptions and then go aheadwith the testing. Basically one is that; they assume that it is totally acceptable orunacceptable. So, if this is assume it is totally acceptable then most of the test will bedesigned in such a way that to prove that it is not acceptable.So, you assume that is unacceptable and then conduct test to prove that, it is notacceptable and if this test fails; then it is acceptable. Similarly, you assume that it isunacceptable and then carry out the test to prove that it is acceptable and if the testsucceeds then the system will be accepted. So, this way we can have two kinds ofassumptions and test to carry out the acceptance test.(Refer Slide Time: 12:04)
The next one is how to test? So, that there are various ways of testing and various thingsto be tested and there are different options for the stakeholders after the test. But how to
do the testing is again an important aspect because there are various aspects to be testedand how do we carry out these test is important. So, here we look at one aspect as theusability of the system and then see how to do the testing of for usability.So, usability testing is to determine the success in meeting requirements of stakeholders.So basically the stakeholder requirements; how much it is satisfied, how the usability ofthe system by the stakeholders is satisfied by the system is tested. There are five aspectsin the usability test; basically you look at the learnability of the system that is time tomaster a defined efficiency level.So, we look at the system and then see how first can be learned by the or mastered by auser. We will have a predefined efficiency level and we will see how much time it takesfor a user to reach that efficiency level; that is the learnability test. The next one is theease of use; now how ease to use the system, so here time for a frequent user to completea define task.So, you find out the time required by a frequent user to complete a defined task. So, thereare users with various capabilities; some of them are very frequent user, some of themrarely use the system, some of the experts in the use of the system. So, we take a mediumlevel user, a frequent user to complete a define task and find out how much time he takesto complete a particular task.So, that actually gives us the use of the system and the third one is memorability of thesystem. So, time for a casual user to achieve previously achieved rate of production. So,here it is a casual user is not a every frequent user, so he will be asked to use the systemand to see how much time it takes to achieve a previously achieved rate of production.So, he may be having some level of production using a previous version of a system or asimilar system. So, this user will be asked to use the system and to see how fast heactually reaches the normal rate of production and that actually gives us the value of thememorability of the system. And the fourth one is error rate, the number of arrays in agiven period for a given task. So, depending on the number of errors we can find outwhat is the error rate of the system? And then for a given period, for a given task we willtry to find out how many errors are happening in the system and that actually gives anerror rate of the system.
Then the satisfaction level, the satisfaction level is the again it is a little bit subject you,but then it is the stress or fun level associated with the user. So, basically a person who isusing the system; how happy he is or how much troubled he is or how much frustrated heis; on using the system. So, that is a measure of the satisfaction of the system; so, theusability of the system can be tested using this five measures.And the last one; that satisfaction level can be tested only when the complete system isavailable. So, when we have the complete system; we can do the satisfaction test, but theother, the test can be completed even when the complete system is not available also. So,these are the five levels of usability testing and we can actually take these parameters andmeasure their values and see how usable the system is and what are the obstaclesbasically there in the usability testing.(Refer Slide Time: 15:34)
Though there are factor emitted, there are some problem in carrying out the usabilitytesting. So, these are basically the first one is of course, getting the test participants; whorepresent the real user? So, we need to have a set of users to do the testing, but then itmay not be always easy to find the users or the test users the sufficient, number of testusers who can actually participate in the testing of the system. So, that actually gives aproblem and we need to ensure that who are we selecting for the testing of the system;they represent the real users or there is a chance for them to use or there is a potential forthese people using the system.
So, then only there will be some validity for the test. So, getting these people is bitdifficult because finding them for the testing or getting them for that testing to come tothe test facility and do the test may not be always possible. So, that is one of theproblems in doing the usability test and then getting a representative sample of howpopulation will evaluate the system; that is we need to have a group of people, whoactually can evaluate the system using one person or a few numbers may not besufficient because we need to get a representative sample of how population willevaluate the system.There may be various sections of people using the system; there may be a system, maybe used by adults or may be children or physically challenged people or those who areelderly. So, we need to get a representative sample of all these people to evaluate thesystem that also might be a difficult task. Then selecting tasks most critical to usabilityneeds, so as I mentioned we need to see the task; which are needed to be tested by theusers but then this selecting the task most critical to usability needs also is important.So, the designers need to analyze all the task and then see which are the most critical taskand accordingly they need to do the testing. And then writing the test scenarios theaccurately represent the real situation. So, the scenarios need to be properly defined andwritten down so that all the test can be done based on these scenario. So, writing thisscenarios also; a big challenging and predicting which user interface is more critical. Sothis again difficult to predict initially because depending on the use and that dependingon the people who are using, so some of the interface maybe critical then just many notbe critical.So, identifying this critical interface also a challenge, so these are the main obstacles indoing the testing. So, we need to ensure that whenever we do a usability testing ; youneed to make sure that we have a representative, a sample of the population who use thesystem and we have the scenarios identified properly and we have the critical userinterfaces identified and we write down the scenarios and to test and what are theimportant task to be tested.So, all these need to be analyzed and recorded properly to make sure that we do a propertesting and that those testings are really important. And these test values really representthe acceptance of the system by the user. So, though these challenges in the usability
testing need to be identified early stages and measures to be taken to overcome thesedifficulties; so that is about the obstacles in usability testing.(Refer Slide Time: 19:12)
So, as I mentioned these are the various stages in the acceptance testing. So, we have toidentify the test scenarios, we need to identify the test scenarios as well as we need toidentify the task to be tested and so many of the test decayed out that usability testing.How do we do usability testing? So, here I will take two case studies to show that howimportant is the testing and acceptance test in the system qualification. We discussedabout the case of THERAC 25 failure, but this was a medical, a device; a computercontrolled radiation therapy machine and developed for giving controlled radiation to thepatients and then in the period of 1985 to 87; 3 patients were killed by radiationoverdoses and though machine was supposed to protect these patients, they were killedbecause of the overdoses in radiation.So, the reason for this was the four different operators entered an acceptable, but infrequently used sequence of commands. So, the reason for the overdose was basically theuse of the system by various operators. So, four different operators entered an acceptable,but in frequently used sequence of commands and that actually led to the overdose andthese errors can be traced to requirement analysis and design flaws.So, when we start the designing of the system it was; as I mentioned earlier we need toidentify the requirement properly and then record them. So, there was an error in
identifying the requirement, the proper requirement were not identified. What is thecondition under which these parameters to be entered? And what is the sequence inwhich need to be entered? Those things were not properly identified and the properlydesigned qualification system could have detected these flaws.So, though there were errors in the beginning in the analysis stage or the requirementanalysis stage, if we had a properly designed qualification system then these flows couldhave been awarded or it can be could have been detected in the initial stage or beforeimplementing the machine or before deployment of the machine, we could haveidentified these flaws. So, the proper testings were not carried out; where we can identifythese requirement or these scenarios.So, the operating scenarios like the; in frequently used scenario could have beensimulated and the testing could have been carried out to find out what actually, what willbe the output and then the errors could have been identified and then could have beenrectified. So, that was the problem really what actually caused the tragic death of manypatients because of overdose.So, this was a clear case where all possible data entry sequence should have tested. So,this actually shows the importance of acceptance testing. So, if all the possible data entrysequence should have been tested to make sure that there cannot be error in data entrysequence and which may cause the system to view some errors. So, this actually showsthe importance of acceptance testing and identifying all the scenarios of testing and thisis one case study that is the another one again from the industrial accidents or thefailures.
(Refer Slide Time: 22:39)
The ARIANE 5 failure, we discussed about this in one of the lectures. So, ARIANE 5was the launch vehicle developed by the European space agency, it causing around 500million US dollar and 37 seconds into its flight ARIANE 5 veered off course anddisintegrated shortly.Again the failure was traced to two inertial reference systems, one of which was in hotstandby, we lend about the standby system which basically it is a fall tolerant system. So,there were two initial reference systems; one of which was in hot standby, so even if onefails, the other one can be used to get the output from the system and then can find outthe location of the system the vehicle.Both SRI’s failed when their software converted 64 bit floating point number to a 16 bitsigned integer. So, this was the basic cause for the failure there was a conversion of a 64bit floating point number to a 16 bit signed integer which was not planned in the systemor they never expected this kind of a conversation. And since it was converted into thisand the system could not accept, the interface could not accept the data and therefore,there was a failure.Now, during the design stage seven variables were identified that could cause an operanderror. So, in the initial stages of the design; the design engineers have identified sevenvariables which can actually cause this kind of a problem and they identified four ofthem. So, a deliberate decision was taken to protect SRIs from four these variables; so,
out of the seven, they identified four variables and they made necessary measures toprotect the system or the SRI’s from this problem. And four of them have been protectedand the other three were assumed to have a large margin of safety.So, again they to get decision here saying that there is a possibility of these three gettinginto this there are resulting into a operand error is very remote and then they are decidednot to protect them against this three variables and no testing was done on SRI toexamine the operational scenario associated with the countdown and trajectory. So, theSRI’s is the basically used during the initial stage of the flight. So, they did not do anyacceptance test to see whether this kind of a scenario will be generated or not.So, since they have identified three parameters or three variables which can actuallycause an opponent error, they could have tested these conditions to ensure that this willnot cause a problem. But unfortunately exactly one of these variables resulted into a anoperand error and that actually caused the problem in the whole launch, vehicle wasdestroyed by that simple issue and later on when they changed that one and then theyidentified the reason for the failure, they could rectify it very easily and then solved theproblem.But if there was a proper testing which actually examined the particular scenario, wherethese three variables may result into an operand error; then this failure could have beeneliminated. So, this again shows the importance of a acceptance testing in the system andidentifying all the possible scenarios of developing an error. And then making sure thatthe acceptance test actually ensured that such errors will not happened and it actuallyqualifies through this acceptance test. So, once it is qualified then the system is ready fordeployment.
(Refer Slide Time: 26:20)
So, in the last three lectures we discussed about the system integration and qualificationand here we mentioned that, we have various stages in integrating the system. So, westart with the subsystems and then components and then subsystem. Then interfaces, westart integrating them and there are different methods of integrating, the system that isbasically, the bottom up approach, top down approach, then big bang approach; so wecan use any of these methods to integrate the system.Similarly, we discussed about the qualification strategies basically the validation,verification and acceptance. So validation and verification are basically carried out bythe system designers and the acceptance test is basically carried out by the users with thehelp from the designers. So, designers will identify the task to be tested and the scenariosto be tested and then the users will be asked to test them and then based on the; forparticular threshold value, the system will be accepted.So, the users have different options; after testing they can actually accept the system orthey can accept it with some changes or they can reject the system. In some cases andsome special cases, the users will be forced to accept the system even if there are someerrors because the existing system is giving lot of troubles. So, they want avoid thesetroubles and then you continue to; they want to new system, just to escape from theerrors of the previous system. These are the options for the users and then we discussed
about the acceptance testing procedures at the usability testing procedures. And then wesaw few case studies, where failures in acceptance test caused the system failure.So, these were the topics we discussed in the last three lectures and as I mentioned in theprevious lectures, so this is the last step in the design process that the six functions of thedesign process. So, if we complete these then actually for each life cycle, we have tocomplete these six functions of the design process. In the next few lectures, we willdiscuss about few supplemental topics which are actually useful in the system design. So,before going for the supplemental topics; I will take few case studies or few systemdesign examples to show you; how do we actually use the principles, whatever we learntin a real scenario or real system design scenario and how to go in a systematic way ofdesigning the system.So, I will take three case studies and explain the procedure and we will not be going intothe details of each and every stage, but I will be giving an overall view of how theprinciples can be used in the system design.(Refer Slide Time: 29:04)
And after that, we will be going to the other supplemental topics like graphical modellingand modelling methods, decision analysis; decision making and decision analysis tools.Then we will look the reliability of the system; how do they design the system forreliability. Similarly, how to avoid the faults in the system? How do we incorporate thefault analysis into the system design? And similarly we look at some of the statistical
tools which can be used in the system design like design of experiments and othermethods.To start with, we will go for the system design examples; now I will take few exampleshere and then explain the procedure of a system design using the standard systemengineering principles.(Refer Slide Time: 29:53)
So, this is the first example for the system design.(Refer Slide Time: 29:57)
We will take the case of an auto link system, I mentioned about this in one of the earlierlectures while explaining one the functional decomposition. I will explain about the autolink system, so we will look into the detail of this system and then how do we actuallystart with the system design, with the procedures or the methods we already discussed.The auto link system is an information system for drivers to assist them in navigation andemergency situations. So, we are actually trying to design a system which can actually beused by the drivers or passengers in a car or any other vehicle in an emergency situationor to get some other information about the navigation aspects.So, if somebody is driving on a highway or traveling on a long
Log in to save your progress and obtain a certificate in Alison’s free Engineering System Designs: Qualification Planning & Design Process online course
Sign up to save your progress and obtain a certificate in Alison’s free Engineering System Designs: Qualification Planning & Design Process online course
Please enter you email address and we will mail you a link to reset your password.