today we will look at thequalification aspects we briefly explained about qualification what do you mean byqualification and what are the various stages in the qualification like verification validationand acceptance and we discussed about those top level cycle and the bottom level cycle or thehigher and lower level cycles of qualification. Today, we will go into the details of thisqualification procedure what kind of procedure to be adopted and what are the different kindsof acceptance tests and how do we actually ensure that the customer accepts the system andwhat are the process by which we can do the testing in order to make sure that the systemperforms as per the requirements of the customer.
(Refer Slide Time: 02:12)
So, first of all let us recap the qualification procedure and what is qualification. So, as wediscussed, it is the process of verifying and validating the system design and then obtainingthe stakeholders acceptance of the design. So, basically we need to verify and validate thedesign and to ensure that it actually meets the requirements of the customer and it involvesthe verification validation and acceptance.So, as we discussed in the previous class verification is basically to verify the developedsystem with respect to the specifications we identified at the design stage validation isensuring that it meets the requirements of the customer in terms of the concepts and validityor the design validity and then acceptance is basically we give the system the completesystem to the user and the user conducts acceptance test and then accept the system.So, these are the 3 stages through which the qualification process goes through and then weneed to design the qualification system as we design the engineering system. So, coqualification system need cannot be designed at the end of the process though we discussedas the last step the system need to be designed at the basic level, I mean when we start thedesign of engineering system the qualification system also need to be parallely developed thatis when we design a subsystem, we need to look at what is the requirement for that suchsystem and then how do we test that system and how do we validate that system and thenhow do we do an acceptance test for that particular system. So, we need to develop theprocedure, we need to identify the resources, we need to identify the schedules.
Then along with the design of the system and then this need to be used when we do thevalidation or verification or acceptance. So, the design of the qualification system need to bedone at the beginning of the design or as we designed the system the qualification system alsoneed to be designed along with that and the exit criterion for integration and qualification isacceptance of the designed by the stakeholders. So, that is the exit criterion. So, we can saythat the system design is complete or we can actually come out of that particular designprocess only when the stakeholder accepts the system.(Refer Slide Time: 04:28)
And this one will discussed actually and again in the last class that these are the variousvalidation procedures and this is the verification and this is the acceptance test.So, we do the acceptance test at this stage when the integration is complete and we do thevalidation test at this stage and parallely we develop the qualification systems in this at thisstage itself we develop the qualification system and then verification is basically verifying theelements against the specifications designed and the operation validity conceptual validityand requirements validity and design validity all these need to be validated as we go aheadwith the qualification process.
(Refer Slide Time: 05:10)
So, before going to the qualification process we will define few terms which will be interestto the qualification engineers or those who do the qualification test. So, the purpose ofqualification is not only to find faults and failures, but also to prevent them and to providecomprehensive diagnosis about their location and cause.So, it is not only that we just identify the faults and failures we need to ensure that weidentify the source of these failures as well as we provide some kind of inputs to thedesigners to ensure that such faults one occur in the design. So, as we do the testing weensure that we can identify the faults and failures and we can identify the location of thefailures as well we can actually suggest methods to prevent these failures also. So, if youidentify the location we can actually see from whether the what is the source of that error andthen once we know the source of the error we can do a redesign of the system. So, that thesuch failures can be eliminated, the qualification engineers or those who are doing theverification or validation. So, they have the responsibility to identify the sources as well as tohelp the designers.To prevent such faults, we just recap the term normally we use when we discussed about thefailures that is the faults in the system. So, failure is basically a deviation in behaviourbetween the system and its requirement. So, we have some behaviour requirements in thesystem and when the system is not providing that behaviour then we call that is a failure erroris a subset of the system state which may lead to a failure. So, the a system state whether it is
the temperature the pressure or its processing time or the processing capability any of theseparameters which is a subset of the system state. So, this may actually cause to a failure. So,that is an error and fault is the defects in the system that can cause an error.So, it is a basically a fault in the system causes an error and this error leads a failure of thesystem. So, to have a successful qualification system a number of complementary proceduresto be employed; so, if you use 1 or 2 methods that alone will not that help you to identify allthe faults. So, one procedure may identify the are faults in the operational scenarios oneanother method may identify the faults in the internal systems or the interfaces. So, there isno single test which can actually identify all the faults in the system and therefore, we need toidentify different procedures or different testing procedures or verification procedures whichare complimentary. So, that most of the faults can be identified. So, that is the requirementhere we need to have many tests; so, many procedures which are complementary to eachother which can identify most of the faults in the system which can actually prevent thefailures of the system.(Refer Slide Time: 08:00)
So, in order to do this there are many methods employed by various system engineers andvarious system development methods, but it is the software community which actually put themore comprehensive procedures and rules for testing of engineering systems softwaresystems, but some of these rules can actually be implemented or can be adopted for the
engineering systems also basically there are 3 laws in software testing or softwarequalification procedures.If this can actually be used in the engineering system analysis also because the many of theprocedures are common to the software engineering as well as the system engineering thefirst law is known as the pesticide paradox which actually states that every method you use toprevent a bug fault in the case of engineering system. So, every method you use to preventtheir fault leaves a residue of subtler faults; that means, whenever you use a particular methodto prevent a fault that actually brings in another fault in the system which cannot be identifiedusing the present method. So, whenever you introduce a new method to prevent a fault thefaults associated with that new method cannot be identified at that stage. So, this is known asthe pesticide paradox. So, you cannot actually ensure that by simply.By removing one bug you are actually eliminating a all the bugs because this the fault maycome because of the new method also that is the first law of the first door software testingwhich is known as the pesticide paradox the second law is known as the complexity barrier;this actually states that the complexity of bugs or the faults grows to the limits of the ourability to manage that complexity. So, as we are more and more capable of solving acomplexity then the complexity of the fault also will keep on increasing. So, that is known asthe complexity barrier the third law is that the code migrates to data these actually states thatthe hardware and people migrate to software which eventually migrate to data.So, initially if something is done by the hardware or something by the human; this actuallywill slowly com migrate to the software. So, we will try to replace it with software and thenthe software will replace it with the actual data. So, that is the code migrates to data. So,whenever it will be planned for the engineering system qualification or testing we need toensure that these are the facts which actually limits our capability to do the testing. So,whenever we try to employ a new method of avoiding fault that actually can bring anotherfault which cannot be identified by that method. So, we need to be careful while introducing amethod to remove your fault. So, we should look at the importance of that fault it is afrequency of the fault.Then we decide whether to introduce a another method to prevent that fault because this newmethod can bring another fault which cannot be identified by that particular method similarlythe complexity of the fault also will keep on increasing as we are more capable of solving the
problem the complexity also keep on increasing and then there is a migration of hardwaresoftware extra to data. So, our algorithms or the methods should be capable of taking this intoaccount when we do the testing of the engineering systems, then again when we do theverification of the system. So, as we know verification is one of the easiest stages in thequalification. So, we have verification validation and acceptance. So, verification is one ofthe easiest method or easiest part of the qualification procedure. So, here again there are afew barriers.Because verification is basically we look at the specification of the design and then seewhether the actual system or the actual component meets that specification when the problemhere is that we can never be sure that the specifications are correct. So, the specification canbe wrong. So, the specification whatever we made the specification need not be alwayscorrect, but we always we have to go assuming that the verifications are correct and do thespecific testing verification test, but they not be correct always.(Refer Slide Time: 12:07)
And now verification system can verify every correct program. So, there is no verificationsystem which can verify every correct program.You need to have different procedures or different methods to do the verification of acomplete program or one method may not ensure that it is completely correct and then we cannever be certain that a verification system is correct. So, again there is no guarantee that theverification system is always correct. So, these are the barriers in verification that is never we
do a verification we actually have some assumptions and have some limitations. So, withinthat limitation only we are doing the verification. So, this shows that the confidence level ofthe verification engineer depending on to what extent we can assure that specifications arecorrect to the action possible and the system the most of the there are differentcomplementary methods he uses to ensure that all the aspects of the system is tested orverified.And similarly, whether the verification system how good is the verification system is. So,based on this confidence level only we can say that the verification is complete or theverification is up to the expectations of the design engineers. So, that is the barriers inverification, but then before we discuss about the methods.(Refer Slide Time: 13:27)
We need to define a few terms about the fault categorization also. So, there are different kindsof faults happening in the system. So, some of the faults are very sever some of them are notvery sever, but some of them needs attention some of them can be left as such because it maynot cause any other problem in the system. So, the fault categorization is the first step indefining the importance of faults. So, if you have if you want to categorize the fault on the interms of the importance.So, how important that particular fault is we need to categorize them into various categoriesand these categories defined distinctions among the consequences of faults. So, the categoriescan actually give the distinction about the consequences of these faults. So, some of the
categorization or the taxonomy of the faults here; so, a mild fault is something which canactually be discarded which actually say that we do not really need to look at that particularfault because it is a very mild one like colour is not proper or there is not uniform surfacefinish or there is a minor scratch on the system. So, these are known as the minor fault whichcan actually be discarded because they are create any further problem in the system, but thenwe have a moderate faults which are unclear or misleading output or a wrong menu.So, once you give a input it is giving a different output which is not related to the input weare giving or the menus are not properly given. So, one menu is not leading to another one orthe; whatever recurred functions are not providing in the menu. So, these are known asmoderate faults, this is actually may be a problem, but again you do not need to invest toomuch on this kind of moderate errors moderate faults there are annoying kind of faults whichactually give annoy the users basically, he is looking for something some data and he isgetting some other data or it is taking too long to process or that menu is not properly acoming or which is not visible properly. So, these are annoying kind of faults which need tobe avoided because the users may not like to have that kind of faults.But again these are not going to create any major problem in the system performance becausethe system may be performing, but it may not be performing as the convenience of the user orthe user may not like to have that kind of faults because that actually annoys him and he isusing the system. So, again this; this need to be eliminated as much as possible the next one isknown as the disturbing kind of faults. So, this actually we will refuse legitimate transaction.So, if you have a legitimate transaction you have a proper authority to enter to that systemand then carry out some transaction then system is not allowing you it is giving some errors.So, this is a kind of a disturbing fault which again is not acceptable.Next level is serious faults this is actually loses the track of input output. So, it actually doesnot know what actually transacted in the system. So, what kind of inputs was given and whatoutput was given and that is something like in an ATM, you deposited some money it is notaccounted into your account or it does not know where the money has gone. So, that kind of asituation is a serious fault in the system then again the next level is a very serious transactiona very serious fault which actually says that there can be mixing up of inputs and outputs. So,a person deposits money into one account and that the money goes to some other account andthe money is withdrawn from his account because of someone else withdraw the money. So,this kind of mismatches of input output and this is a very serious fault.
Which need to be addressed at any cost then again there are extreme faults where actually thisvery serious faults are happening frequently that is that kind of situation is an extremecondition whether frequent faults of very serious category. So, very serious category faultsare happening very frequently, then it is an extreme fault situation, then there are intolerablefaults which actually causes long term unrecoverable data corruption. So, this actuallyintolerable because that actually you can you can cause the whole system on or a period oftime the whole data may be corrupted or they lose the track of what actually happened. So,that kind of long term errors should not be there and this intolerable kind of faults.(Refer Slide Time: 17:55)
And again considering the seriousness of the issue there are catastrophic; catastrophic faultswhich actually the system shutdown causing data loss.So, it actually causes a data loss to the complete system because it goes to shut down thewhole system and then lot of data is lost without any trace of what happened. So, that kind ofsituation is the; a catastrophic fault and the last one is the infectious fault infectious fault isthat it is not only the that system all the system is associated with that system is also gettingcorrupted. So, this is a very serious situation its known as infectious faults. So, based on thesecurity of these faults we can actually classify them under various categories they startingfrom mild to catastrophic and infectious situations. So, it is the duty of the qualificationengineers to identify these faults and categorize them what kind of a fault it is whether it is avery mild one or it is a an infectious one.
So, accordingly they need to look at the importance of these faults and then identify therequired actions to be taken though need to look at the source of these errors and then find outthe or suggest to the design engineers the procedures or how to actually a eliminate them. So,if you can develop the methods and procedures to identify all these kinds of faults from mildto the infectious depending on the severity of the fault and then try to eliminate them that isthe job of the qualification engineers basically they will qualification engineers will developthe procedures to identify all these faults and categorize them and then report them to thedesign engineers. So, that the design engineers can actually look at the system and then go fora redesign of the system.Again for looking at the importance of the faults we can actually find out the measure ofimportance depending on the frequency of the fault happening as well as the severity of thesituation or the environment at which these particular faults are happening. So, a method of aquantity measure of the importance of the fault type is given here,
Ii=∑j=1JV j PijCijwhere I is the importance of the fault type and for the I th fault in j th scenario. So,Pij the probability of the fault i to happen in a scenario j and Cij is the cost offault in fault i in j th scenario in terms of rupees.So, if you can actually convert that cost of that particular fault if it is a mild for the cost willbe less, but if it is a very sever fault then the cost will be very high. So, if we can actuallyconvert that into rupees then we can say the Cij is the cost of fault i in j th scenarioand V jis the relative measure of importance of that scenario.So, if you have multiple scenarios in which the fault is happening. So, we can actually givethe relative measure of importance of scenario at some scenarios may be very important somemay not be important. So, V j
gives the relative value of this importance. So, the actually
importance of the fault type can be obtained as
∑j=1JV jPijCijSo,
Pij is the probability Cij is the cost and V j
is the relative measure. So, if you haveone particular fault happening then various scenarios and if you know the cost of thatparticular fault as well as the probability of the that fault as well as the measure of theimportance of that scenario we can find out what is the importance of that fault type and thisalong with the tax for taxonomy discussed earlier will be able to categorize these fault interms of whether it is very severe and then we need to look at procedures to eliminate them orwe can actually just leave it like that because it is not causing to cause any problem.So, this is the way how the faults are categorized and then how do we do the analysis of thefaults based on the probability of fault as well as the cost of fault and the importance of thescenarios now let us look at the methods and procedures to do the qualification planning.(Refer Slide Time: 21:59)
So, as I mentioned the qualification strategy need to be planned during the design stage itself.So, there are 4 major levels of qualification planning one is the qualification process you planthe qualification process plan the qualification approaches and you plan the qualificationactivities and then plan specific test.So, these are the 4 activities to be carried out in the qualification planning as you know thefirst one is the qualification process this is the first level of design of the qualificationplanning where.
(Refer Slide Time: 22:32)
We look at the various aspects of the qualification basically we need to do the acceptance testvalidation testing and verification test and all these 4 stages we look at the these 3 tests to becarried out in the qualification and we plan for these 3 tests in all the 4 stages what wediscussed. So, this is the planning the qualification process in planning the qualificationprocess, we look at the system objectives we identify what are the requirement identified inthe system objectives that is one of the earlier lectures we discussed how do we identify thesystem objectives and then use these objectives to define the system requirements as well welook at the hierarchy of the objectives.So, in the qualification planning also we need to start with the system objectives. So, whatactually the system supposed to achieve and based on that only we will look at the objectivesto be a needed and then these objectives become again a base line for the qualificationprocess also. So, we look at the system objectives and then we identify the qualificationsystem objective system objectives more like a performance objective. So, based on theperformance objective we look at the qualification system objective.So, they will be directly related because the qualification system should actually ensure thatthe performance objectives are met and therefore the qualification objectives need to bedeveloped from the system objectives, then we need to look at the pass fail threshold for eachtest because we need to conduct many tests like acceptance test validation test andverification test and we need to look at what are the pass fail threshold for these tests. So, we
cannot have a very low value for the threshold. So, if you put a very low value for thethresholds then that test becomes not really useful, but if you put a very high threshold thenwhat happens is that you may find that most of the tests the system will not be able to passthe test also.So, one of the main requirement here is to look at the objectives the system objectives and thequalification objectives and then decide about a value for this threshold the pass fail thresholdfor each and every test to be connected and then based on that we go for the qualificationrequirements that is like any other system design process we look at the requirements of thequalification and then the functional architecture for the qualification what are the functionsneed to be provided what are the top level functions and what subdural functions are neededthen we go for the physical architecture development and then identify risks and mitigationstrategies . So, what are the risks involved in these processes and how do we actuallyovercome these risks and then create a master qualification plan.So, at the end of the qualification planning process we will be getting a qualification planwhich can actually be employed for the throughout the design. So, initially by looking at thesystem objectives we will develop the qualification objectives and then based on thequalification objectives.We decide about the threshold values for different test and then we identify the qualificationrequirements and from there we look at the functional and physical architecture and finally,we will be getting the qualification plan that is a master qualification plan document and thisdocument will be the master document for all kinds of qualification activities. So, that is thefirst function in the qualification planning plan the qualification process.
(Refer Slide Time: 25:59)
The next one is planning the qualification approaches and again this has been done for all the3 kinds of a test acceptance validation and verification. So, here in that approach we will tryto identify the resources and organizations.Then the assign the qualification activities to organizations assign qualification activities toresources and develop qualification schedules consistent with the development schedules. So,this is the planning the qualification approach. So, here we try to identify all the resourcesneeded for qualification. So, we need to carry out various tests and for this various tests weneed to have various resources and need to identify the infrastructure available and the peoplewho are capable of doing these tests if the test can be done in house we can actually identifyif some equipment to be procured that be planned or there are some standard testing facilitiesavailable whether the test can be carried out. So, that also need to be planned.So, that is the first stage you identify the organizations and activities and assign theseactivities to various resources and then develop the schedules consistent with thedevelopment schedule. So, here will be having various schedule for the design activity as wea planned for the design we have the component level design and then some system leveldesign and. So, along with that in consistent with those schedules we need to have thequalification schedule also; for example, if you want to verify the components. So, when aparticular component is made available as per the schedule then the qualification schedulealso should match with that component similarly when that component is going for an
integration the verification of that integrated subsystem need to be carried out and thesubsystem is ready.So, the schedule whatever we developed for qualification should inconsistent with the designactivities and this is to be ensured in the qualification approach developments or planning thequalification approach. So, these are the various activities carried out in the qualificationapproach.(Refer Slide Time: 27:59)
The next one is planning the activities actually. So, how do we actually do carry out theseactivities. So, develop detailed and derived qualification requirements write actively levelqualification plans and assign qualification responsibilities. So, these are actually planningthe real activities. So, we find out the detailed and derived qualification requirements. So,there are maybe different requirements from the main requirement there may be some derivedrequirements.So, we go into the details. So, in the third level we go into the details if we look at theindividual items configuration items and components and then identify the detailed plans foreach component. So, if you have a particular component identified in the design you have tolook at what is that particular component and what kind of requirements are there for thiscomponent whether we need to have any specific or specialized fixtures to assemble it andthen do the testing or any specialized features are needed or specialized facilities are neededfor that particular component. So, that kind of detailed planning will be done in the third
function which is the qualification activities. So, for each component and subsystem, we lookfor the details.(Refer Slide Time: 29:07)
And then develop the qualification activities and the fourth one is basically the specific test.