Loading

Module 1: Herramientas de evaluación

Study Reminders
Support
Text Version

Set your study reminders

We will email you at these times to remind you to study.
  • Monday

    -

    7am

    +

    Tuesday

    -

    7am

    +

    Wednesday

    -

    7am

    +

    Thursday

    -

    7am

    +

    Friday

    -

    7am

    +

    Saturday

    -

    7am

    +

    Sunday

    -

    7am

    +

In today's lesson, we will study about monitoring and evaluation methods in development research methods.Now monitoring and evaluation is a very important and integral part of the entire research process so much so that today there are various kinds of monitoring and evaluation frameworks that are inbuilt into the research proposals that we make to various kinds of donor agencies sponsoring agencies.And you would see that monitoring and evaluation frameworks are also provided by various sponsoring agencies for the researcher to keep in mind about the things to be thought about very clearly at the beginning of the research itself such that the research does not face any hurdles midway.So, in today's class, I will give you some of the very basics about monitoring and evaluation.I have not taken up any frameworks a part of today's lesson because this will in a sense give you some of the very basic ideas about how to build a monitoring and evaluation framework as a part of the research that you are trying to undertake.So, what we will cover in today’s lecture is as follows; managing and supervising research for development work.We will be looking at the ongoing supervision of research work with suggestions for issues which require attention in supervising research for development.We will also look at the issues of research planning.So basically, how to estimate costs and timescales, some possible tools for timetabling of research work and the timing and costs of various aspects of the research process as well.Also, some particular needs of participatory research and we will end with this entire summarisation of monitoring and evaluation research.Now when we are starting with the research process there need to be two very important kinds of supervision that are integral to research supervision.One is management supervision and the other is research or professional supervision.So now coaching is basically a very excellent way for people to learn where possible a team should include people with more and more experience.So those who are learning may be good at data collection and while the more experienced ones can help in setting a framework, formulating a framework for the researchers in organising the material so that they can learn from each other with time.Now as I said, research projects need two types of supervision, management supervision and research or professional supervision.So overall monitoring of progress towards the aims and objectives of the research is what falls under management supervision and non-managerial research supervision advising a methodology and ensuring that quality standards of research are met is what falls under the purview of research or professional supervision.Now this table 1, that is showing on your slide makes a distinction between these two different kinds.Now, these two different kinds of supervisions may or may not be given by the same person.Where managerial and professional supervision is provided by different people, it will be useful for them to meet up together from time to time with the lead researcher to consolidate forward plans and ensure that all are working towards the same end.So, research is basically a specialised area of work and if researchers are to learn from their work and do it most effectively they need contact with others in their profession as well as those they work with on a day to day basis.So, let us look at some of the very distinct points that management supervision and professional supervision must keep in mind.So, management supervision looks after clarifying aims and objectives.Professional supervision ensures professional standards are met.Management supervision ensures proper communication with senior managers facilitating wider use of the research.Professional supervision assists in identifying relevant literature.Similarly, management supervision insurance progress towards aims and objectives whereas professional supervision assists in making links beyond the immediate area with relevant researchers and others.Management supervision, they set time scales and targets whereas research supervision draws on experience to assist in planning timescales.Management supervision deal with resources issues.Research supervision advises on research methods.Managing research advisory group says what falls under the purview of management supervision ensuring people with appropriate knowledge are invited into advisory groups falls into the research domain.Facilitating links with partner agencies versus considering emotional responses to the process of the research and their consequences for the work.Supporting participatory processes versus advising methods of analysis.Supporting the process of working out recommendations from findings versus assisting in dealing with ethical dilemmas.Ensuring feedback to policymakers or community members in an appropriate format versus assisting with writing for academic audiences.So, these are the few domains that need to be kept in mind for management supervision and professional supervision.They may or may not be the same people or persons who are carrying out both of these activities; however, when you have a research project of a bigger scale, it is often appropriate to have two teams who are looking after management supervision and another team that is looking after mostly research and professional supervision.Also, it is important to understand that the research supervisor can assess the researcher in fulfilling the important functions of keeping the project in touch with related research work elsewhere.Professional non-managerial supervision is a good way of ensuring that such learning takes place.Now one role of a research supervisor is to encourage self-reflection about the researcher's own role in the research.For example, gender, background or manner may have on the research process and you should be possible to talk about the emotional impact research can have and to unpack what this may mean for work.Now also understand that supervision should not be strongly directive and how objectives are fulfilled but should keep an eye on the larger pictures.The supervisor needs to support the researcher in sorting out a sound methodology for the project.Now it is in this context, it is very appropriate to remind oneself about checklists for managing research for development work effectively.These are a few checklists that have been worked out, that can come in handy for anybody who is planning on a research project of a bigger scale or of a medium scale on issues of development that is showing on your slides now.One is progress towards aims and objectives is going well or not.A set of milestones along the way that are planned for and then achieved.Written material should be clearly expressed and appropriate to their audience.Process of negotiating access to participants is handled tactfully.Appropriate partner agencies have been informed of or involved in the research and advisory group is convened as appropriate.People are being kept informed of progress.If the research is intended to be participatory, there is evidence that people feel that they are able to participate and value this process.Disadvantage sections of the population and question are being included or not.Formalities have been properly observed or not.Reports include an appropriate balance between description and analysis.Ethical standards are being upheld or not and finally, we must remember to give positive feedback to the researcher whenever possible because research can be a very lonely business and many of us who have completed PhD researches or those of you who are undergoing dissertation for PhD researchers would very well know that the entire duration of research can be a very lonely process.Therefore, it is very important for supervisors to give positive feedback to the researchers whenever possible.Perhaps one of the most common practical mistakes that we make during research proposals is to underestimate the importance of budget, time and finances.Now, this is an example here that I have put up on research planning timetable.Research is a messy and chaotic business even for the experienced researcher and will be even more chaotic for the researcher with little practical experience.So, research budgets that allowed no room for manoeuvre or any mistakes can create significant problems.So, a crucial part of the planning process is obviously to sort out the time scale for the research and the costs which will be incurred during the entire time scale.As with so many things in life it is a safe bet that when you think it through you will find that the piece of research will cost you more than you had actually hoped for.So, time and money are usually tied up together.More of one can be used to balance less of the other and vice versa.Now this time table that is showing on your slide now shows how to plan time for research.Research project basically involves the series of tasks which are quite different from each other, but which may overlap in time and interrelate in a complex way.So, in planning a piece of research it is worth working out a time table in some detail at the start, you may not be able to stick to it exactly, but at least it will give youa way of looking ahead at the consequences of things start to slip.So, in this example here that is showing on your slide, this is a timeline here of 12months and it is an annual plan of a research project that has been taken up.And if you read through the items that are mentioned in this matrix here, you see that all of those stages that we have been covering is part of a research design have been included.So, planning, discussion with partners, search for existing information, data research tools, selecting and contacting respondents, pilot tools and revise, fieldwork, transcribing, analysis, writing, editing, publication and dissemination.So, planning, discussion with a partner, searching for existing information basically falls under the purview of finalising on a conceptual framework or looking up literature, doing a literature survey, doing a literature review and so on.Data research tools also become a part of the literature survey.Then planning for the fieldwork, so that is the data collection stage, selection of respondents, pilot tools, revising them.Methods that you want to use for your fieldwork, focus group discussions and interviews.Transcribing analysis, writing is part of the data analysis process.Publication and dissemination are part of the communication process.Now you would see that often various items cut across time and many of these things have to be carried out simultaneously.For example, if you see here in this timeline between January and December, while we're planning with partners, the search for existing information has already started.Somewhere after about a month and a half data research tools reviewing has also started.So, this is something that is carried out simultaneously, more or less simultaneously for us to be able to enter the data collection stage.Similarly, if you look at this stage here when the analysis part begins although you start your analysis, somewhere from in between the analysis you begin writing up your research process also and while you are completing the writing of the research process you can begin editing and think about publications and final communication of your research that you are undertaking.Therefore, this kind of planning of your research, a timetable of your research what we often call is a timeline of your research becomes very important for us to plan about things that need to come next with respect to time as well as with regard to the money that we want to spend on our entire research process.Now this table 2 here shows a way of planning research in real-time that is it estimates how long the different stages of the process will actually take to carry out.Another way of looking at time is to estimate how many days work each stage will take.This may be a very useful way of working out costs.So, for example, the items clarifying the focus, setting questions, literature survey, you might want to mention with regard to the time required, whether you require 1 month of time or you need 2 months of time for the literature survey part and so on.You need to mention the time required for this and accordingly, the manpower that you require or the personnel that we require for each of these tasks and then accordingly the costs can be estimated.Now some of the issues to consider while timetabling your research are as follows.One is you can think of the search for existing information.Some of the most valuable information may only be available with a lot of effort.You need to meet a lot of people, you need to enter into various organisations, you need to establish networks and contacts at various organisations to be able to have access to a very small part of the information that you require and you not having access to that very small part of your information that become a big hurdle in the way of on moving on with your project.So, this may involve writing off to distant organisations, going to libraries, both of which takes time and sometimes people will need to be talked to for assisting you in the research process.So, it is important to set deadlines for these kinds of works.It can take on a life of its own and it needs to be concluded within a reasonable time.Another important thing to remember is about planning and drafting research tools.Now, this takes time as all questionnaires and other tools need to be thought about very carefully and we need to go through various drafts of those questionnaires and tool before they can be finalized.Therefore, this is also something that really needs to, this bit about drafting research tools, takes a lot of our time in annual planned time table with regard to how we want to move onto the next stage.Selecting and contacting respondents is another very important area, there can be various kinds of hurdles to overcome usually in terms of persuading gatekeepers who are in control of people who we want to access to for getting information.For example, village leaders or various officials, functionaries within a certain location ora locality, who we need to interview, we may not have access to them because of various reasons.And that may take a lot of our time and that needs to be integrated into the budget timeline that we want to make.Participatory research is an important component that if you are trying to bring in participatory research methods as a part of your research, it is extremely common for reports and participatory research projects to emphasize the need for time to be allowed to build trust with your participants.Because often as a researcher when the required amount of trust is not established with the participants, it is very difficult for information flow to take place in the case of participatory research methods.Because it becomes almost impossible for participants to actively assert themselves during the research process and if participatory research methods practitioners or academicians must take note of these issues before planning on the timeline if you are taking a participatory research method.Fieldwork is a usually multifaceted process.Now you might require just about half an hour for interviewing your respondents, but preparation for the field, preparing yourself to go to the field and the travel time needs to be taken care of because the entire duration if you think of the travel time that is required to reach your correspondent and then the time you require for cleaning of the responses that you have got from your respondents might take you well above 2 hours.And this is something that a researcher must very meticulously plan when you are planning fieldwork.Similarly, recording data.Time must be allowed for this whatever method of recording you are using because often early researchers make the mistake of not recording the data that they want to transcribe.For example, focus group discussions is a method which must definitely be recorded.Permissions must be sort from the participants regarding whether the data can be recorded, because often when you are not making notes of your data while you are carrying out the process or you are not recording the information immediately after, the information that you have collected is often irretrievably lost.Therefore, all efforts should move towards the recording of the information that you want to transcribe later on in your research.Finally, data entry.Large surveys usually take a lot of time in cleaning of the information, cleaning of the data, which can then be put to the data analysis stage.Whether you are doing it by hand or by computer, this is something that takes time and the data entry phase must also be accounted for while planning a research project.Even experienced researchers make the mistake of not integrating the timeline required for data entry as far as the research process is concerned.Now one of the common pitfalls of the research process is thinking that writing is something that can take place over the weekend.It does not happen so.Thinking and writing are two different processes.While you can write when you are thinking often it becomes impossible for you to write without having ever thought about what you want to write.So, thinking and writing are two definitive processes that may take place simultaneously or may not take place simultaneously.And therefore, a sufficient amount of time needs to be kept aside for writing of your research project.But let us now come to what is showing on your slide is an example of a project budget.So, in the last case when I was talking about allowing time for the research writing process, people often forget about it when planning research and it is a crucial part of the process.It is a separate process from thinking and similarly, planning of the budget is also very separate process and qualitative analysis especially can be very time-consuming.With quantitative research, it is the data coding and entry and sorting out of any problems that emerge which takes the time.The actual analysis can actually be very quick.Now coming to this project budget example.Now working out the costs of a research project is basically never easy.Very often the situation is more that you have to work out what can be done for the money that is available with you or which you think you can obtain.What you need to spend obviously depends upon what you already have in the way of equipment and what you can do without.Staff costs are generally the largest item and if spending on other items can save staff time it may well be worthwhile.This is an example of a project that I have undertaken in which we have mentioned about the personnel requirement, travel and accommodation, project implementation cost with regard to materials, the net project cost, the overheads that we carry out as a part of a certain project and the tax calculations with regard to the project that we do and then you come up with the grand budget.But some of the possible costs of research projects will include obviously research wages of fees, they will include interpreters, translators for research materials or data.You will have expenses on travel and subsistence expenses of researchers, payments to community researchers and respondents.Often when we are carrying out focus group discussions with a group of participants say7 or 8 or 10 in number, since they are giving up their wage day for participating in the focus group discussion, it becomes essential for them to compensate them for the day's wage so that they can enable themselves to be a part of the research process.And these arrangements must be done well in advance so that you do not misuse a lot of your time into conducting a good study.Similarly, training of staff and participants, purchasing books, journals and other reference materials, tape recorder, tapes, batteries, transcribing machines, administrative expenses, administrative costs, advisory group travel and other costs.These are some of the costs that need to be kept in mind.If as part of a research project we want all of these costs to be integrated into a budget, all of these things need to be properly planned out so that the budget can have details of all of these.Now it is likely that 2 sources of staff will be needed to support participatory processes with community members.There will need to be an experienced research advisor to oversee the process from the research point of view and they will need to work alongside staff who are familiar with working with the participating group.And trust needs to be established to give participants the confidence to have their say.If it is possible to use staff who are already known to some of those involved are so much the better.For work with children and young people, for example, they might want to work with youth networks or with child care practitioners and teachers who are usually very well aware of the field and the methods that we want to employ.It will be useful to look up a checklist on managing participatory research which is showing on your slide now.For example, we may consider facilitating participation in different elements of the research process.We may allow enough time for real participation at reach the relevant stage of the process.Similarly, reflect on why you are encouraging participation and what participating community members will gain from their involvement, allocate sufficient staff time to support participating community members, consider and resolve any issues in relation to remuneration for participants, take account of social inequalities amongst the group you hope to involve and make links between the participatory project and other relevant work in your agency to maximize learning relating to funders.A similar checklist can be brought up for effective research management.For example, providing management supervision and research supervision, make a careful estimate of time research.Time estimation is usually underestimated.Even experienced researchers make the mistake of time estimation.And often there is a pressure from the funding agencies or sponsoring agencies to stick to certain deadlines, timelines.However, the quality of research suffers because of such timelines.It is also important to allow enough time for contacting respondents and in participatory research for winning the trust of stakeholders.There is great value in building up organisations research capacity considering this as an element of the objectives of project and workout appropriate ways of involving people.Now participatory research is likely to need both research staff and staff who are familiar with the group you are working with to commit substantial time to a project.So, one must make sure that he or she understands what a funder wants from it is grant-giving and exactly what it is remit is before you write an application.Now by the time you reach the completion of a research project, there is a natural tendency to want to have a sigh of relief and move on to the next thing, but if we have to develop better practice in this area, it is essential to evaluate the work as we go along and therefore, our evaluation effort should be in proportion to the resources spent on the research project and how we focus the evaluation will depend upon the key aims of it.What is showing on your slide now is a format in which we would like to evaluate success against the objectives that we have taken up.So, the most straightforward way to evaluate research projects is to look back at the objectives you set for the project and consider to what extent they have been achieved.So, for example, you have 4 objectives, you can very well list out the achievements that you have had as to whether you have achieved each of these objectives, what are the limitations in each of these objectives and what are the issues that you may want to consider because of the limitations that exist in the objectives that you have achieved.Now one problem with evaluation against objectives is that the original objectives may not have been written in such a way as to express clearly all the purposes which were in fact being pursued through the project.So, in this case, it may be appropriate to add to them or adapt them.So, one should make sure that the evaluation process that we are following is as relevant as possible to get the most learning out of it.Another way of evaluating a project is to compare its performance with the set of predefined standards of good practice.So, you might want to look at your project success in relation to the quality standards proposed in various manuals.For example, this slide here which is showing on your screen does your project meet quality standards for research for development framework.This is a very standard format that is used in various research, which I have taken from the reference showing at the end of your lesson.So, the standards are research planning about an important issue where the idea came from, the research processes, interpretation and analysis, promoting research findings, lookup the strengths and weaknesses and notes for action.So, with regard to research planning some of the things that you want to keep in mind as per standard are concerned is whether the research is about an important issue, what is needed to be done, whether the idea came from the researched, there must be a clear brief with regard to the research plan, it must be planned in cooperation with relevant others, transparent organisational structures, develop participants research skills and soon.Research processes must include questions that taken for granted, respectful interactions with respondents, the researched had a say in the research design.So, the participants who we are taking as a part of a research process really had a say in the research design or not, when we are designing our questions and objectives whether they had a say or not, whether the sample is appropriate or not, whether research tools were piloted or not.Similarly, data collection was systematic or not, social inequalities were considered or not, ethical issues were actively considered or not.So, these are some of the standards that may be kept in mind and strengths and weaknesses may be assessed.With regard to interpretation analysis, some of the standards that need to be kept in mind are whether the interpretation was accurate and honestly reported or not.Whether analysis included consideration of alternative interpretations, what was the broader policy context, the wider research findings and so on?With regard to promoting research findings reports must be concise, would include adequate information to assess the research, appropriate summaries must be included, recommendations much arise from the findings and findings may be fed back to the researched inappropriate ways and so on.Now one of the things to keep in mind when we are developing a monitoring and evaluation design or an evaluation framework is that there are some of the ways of getting feedback as a research project develops.Now the research advisory group should have as part of it is a role to monitor the progress and impact of the project.At the end of the interviews of focus groups et cetera, one must ask respondents how they felt about taking part in the research, one must build in a short evaluation discussion at the end of any group event, employee and expert advisor or external evaluator who offers to listen to your and some ongoing evaluation input throughout the life of the project and meet together at intervals as a research team and step back from the day-to-day work to reflect on progress and any difficulties encountered.So, these are some of the ways of getting feedback as a research project develops.Now, these are some of the checklists for evaluating programme-focused research.Ask community members and colleagues what are their criteria for success for the project, structure your evaluation around the project’s aims and objectives, ask people to assess strengths and weaknesses of the research project’s products and process.For example, you might want to carry out a nutrition intervention for which you have designed research.And you are carrying out your fieldwork, but before questioning or after questioning your respondents you might want to sit down with your respondents and ask them about the intervention that you are planning.Whether there will be advantages or disadvantages, what the respondents feel about the intervention that you want to carry out.That could be your first check with respect to the intervention that you are planning out.We can also include questions which invite people to think ahead to engage with what should happen next, do a small survey on paper in the person of key people's knowledge and views on the project and look at the learning and development of staff and community members which has occurred through the project as well as the other outcomes.With regard to evaluating the impact of advocacy-related research, there are 4 points to keep in mind.Policy or practise change, strengthening participation in policy processes, institutional learning and financial.With regard to policy changes, now there are some of the points to keep in mind are as follows.The direct changes in policy, new legislation, new official guidance, the revision to existing policy documents whether they are carried out or not, direct changes in practice, references are made to your research and policy discussions.This is how you evaluate the impact of advocacy-related research.The issue your research addressed appears on agendas that have previously been excluded from.