Learning-workshop 2 WhatWentWellOrNot

imGoats project: Learning and reflection workshop

---

What went well with Outcome Mapping or not

View the presentation by Ann File:Learning from the ImGoats OM Experience day 2.pptx

Ann: I want to talk a little more generally about OM and how it fits into a project or program. How was imGoats born? The birth mother was ILRI : ILRI identified the need for this project, began to look for partners and designed and planned the program. In our very first meeting, Feb 2011, we all came together to design and plan. And, since then,

So far, to my knowledge, there has been no mid-term evaluation and I don't know if there is one planned. What monitoring does is feed information that helps manage a program and helps with the reporting. You are using OM for planning and management, and there was a mention of using OM for reporting from the Mozambique team too. One of the big questions for me is - "Have you been using OM for progress reporting?"

A lot of organizations don't use M&E becuse they don't see the benefit to the program. It is usually left to the donors to do M&E and they see it as a burden. It's very exciting for me to see the potential in OM and to see you using it.

One of the interesting things is that you have been hybridizing OM - instead of stopping what you're doing and implanting something new from scratch. What you're saying is, well, this is what we're doing now.... how can we use techniques that we already know about and marry them with OM? In the case of the India team, you are fully using existing data to analyze whether or not progress markers are being achieved. And, in Mozambique - your hybrid consists of using existing information along with OM journals. Very innovative - both country groups are using a hybrid of existing methods and OM.

Today, we're going to spend some time on what went well and what went less well in our application of OM? [Methodology: the participants split themselves in country teams to answer five key questions - mentioned below. They elected one person to interview the representative of the other team and the facilitator and other audience members had the option to ask prompting questions].

	India	Mozambique
1. What went well in your application of OM?	We are using the existing information systems in addition to OM. We hope to use the information we're collecting. We are entering the information and using it to monitor field activities and performance. This is not a one-off collection, we are regularly collecting and using data. And then analzying the data, with the help of Ram, so that we can make plans for the villages. Questions from the audience: Q: Did you train your field guides? Is there some reluctance by the field guides, because you're asking so much information? How do you keep them motivated to keep on collecting data? A: Yes. Keep motivating them by preparing them - we tell them "if you do this, it will help in the future." Providing feedback from the meetings and encouraging them. Q: Have you had situations where you've had Paravets that you've selected and trained (27) not want to continue? Have you retained all 27? A: Yes, the field guides belong to the same village and are part of the same community. It's too early, it's only one year now... as they grow and become more professional in their job. Only one left, because she went to get married! Some field guides like the attention that they get now, people come to them for treatment now. Even if they don't have to do work related to treating animals, they remain motivated. Q: Also training paravets, but difficult for them to collect data. How do you ensure that the data is the quality that you want? A: Supervisor goes along with the paravet to the field. He verifies the data. Q: How often? A: Regularly. For 5 field guides, there is 1 supervisor. The field guide does not go to all households on the same day. On one day he goes to 4 households, and collects data from them. So, he does not spend the entire day collecting data. It's both socialising and collecting data: "did you sell the goat?" "are there new kids born?" Q: BAIF has been working with field guides before. What is the main difference of what you are doing now, compared to what you were doing before? A: Quantitative data was used before, so collecting qualitative data is something new.	1. Systematic collection of valuable data. 2. Process allowed to adapt to what we thought would work for us. 3. Monthly meetings allowed us to review activities in a systematic way & adapt where needed. 4. Meetings allowed for staff (ext officers) that were not present in initial workshop to understand outcome mapping and its relevance to the project. 5. Follow up field visits by M&E officer raised awareness about project interventions among the goat keepers. 6. Increases accountability among the staff. Questions from the audience: Q: Was there any diffiiculty discussing with the stakeholders about your OM plan? Was there any resistance from the farmers? A: Working with them, we see what change in the behaviour has taken place. We didn't interview them. Producers have some problem in one area, we go to work with them and we didn't say "this is for our process of M&E monitoring" - we were waiting for them to see what is happening. We didn't interview them. When we were in the process, we saw that the process was not good for them. After that we saw that the people liked the system. Q: Are the producers aware of what you're doing with the information and how it's going to be used in the project? A: They know but our staff - when they work with the producers. The field visits - the imGoats team decided to have an M&E officer going back to the field to verify the data and also to ask additional questions of things we didn't know. People responded well as they were very happy that there were people coming into the community to ask them questions. It helped them to understand what we were doing in the project, and helped us too. The field visits were very positive. Q: Positive feedback, there was a change. If there are 10 people in the community, 6 or 7 may be positive, but 1 or 2 may not be. How did you summarise the differences? A: Some things were not going well, so both positive and negative experiences were reported. However, it was not quantified - the feedback was qualitative. (Majority, minority, consensus etc.)
2. What went less well?	The main challenge was managing the volume of data. There are a large number of households and so large volume of data. We struggled to manage this data in the particular time frame. The qualitative data was a problem for us. Questions from the audience: Q: Some information may be different than what they are used to, so how will they deal with that when it comes to analysis? A: Right now we have too much data, so the management of expected future data is difficult. Q: Written in Hindi. Is the person entering the data able to read all of it? A: Yes.	1. We have a lot of information on producers but nit that much info. on boundary partners. 2. Gender and capacity building were not clearly included. 3. Order of discussing boundary partners in meetings affects the quality of data collected ---> long meetings! 4. Follow up visits have stopped 5. Our team tends to focus on positive changes, not on the absence of change 6. Staff not well prepared for meetings (don't read minutes in Portugese) Questions from the audience: Q: Focus in positive change. Wondering - the way you do OM, does it focus on change or also allow for focus on what is not happening, what is not changing? A: If you want to monitor the system you have to know. if you only focus on positive things, it is possible to go away and not be consistent. Some things are very clear, and yet other things are not clear. We are not looking into why there is no change, where there is none. One of the ways we had to do that was through the follow-up visits. But, now that they've stopped, we find ourselves a little stuck. Ann: "Good practice in OM is to record change. It needs to be built in somehow into collection of data. What monitoring does is tell you that something is happening, but not why. Looking at why is more of an evaluative function." Q. In terms of dissemination and using the information from OM, is it going well or less well? A: Immediately try to adapt - immediate monthly evaluation of staff. It is only used at a very direct level, while we collect it, immediately using what we have found.
3. What did you do to meet the problems/issues that arose?	We faced this problem with quantitative data. We bring it all together to identify the issues. We quote them and put the quotes into an excel sheet. Which group is discussing more issues, and which group is discussing less.Regarding health related issues, marketing problems. How can we capture the narratives that are there? The field guide is noting everything down in a big narrative in local language. We had to organise this information somehow, so we went through the registers, looked for common problems, and ranked them by how frequently it arises. (e.g. health related issues such as diarrhoea, marketing problems.) We number the problems and that is how they are then inserted into the Excel sheet. Questions from the audience: Q: You mentioned having more data than you could analyse. Have you tried sampling? A: We are considering this but have not tried it out yet. Sampling for us does not equal representation. It is a part of the field representative's job to look at hundreds of households, but not the whole village. So he keeps maintaining a record of these hundred households - so the data is coming out without sampling. Managing the data is the problem, putting 2700 entries into the computer each month. Perhaps we need to input data more frequently, each week rather than each month. Fecel sampling is another thing we do, and each field guide is collecting these samples too. This was not in our original work plan and is a new activity. When we say volume, it is not just households but other data collection too being fed into the computer. Q: Why don't we only take 10-20% of data for total analysis? A: Unresolved. Q: Depends what you want to do with the data - random samples. So the question is, do you stick to the same households or take a random household selection which may end up being different every time. A: For market information for example, we would like to have information of all of the houses for the traders and not only a sample.	1. One problem was the order of discussion. So we altered the order. It's a 4 hour meeting and we're all tired. So in order to make sure we include the last boundary partner at the end of the day, we change the order of the partners in the meetings. 2. We include questions on gender and capacity building 3. We felt we had a lot of information on the producers, but not a lot of information on the buyers. It was agreed that Amos would go back for follow-up visits to the buyers. Questions from the audience: Q: Why don't you focus one month on one boundary partner, and the next on another? A: It is an option, but then you would have 4 months between each boundary partner. For these meetings, our extension officers come from the field. They don't do that each week and we don't want them to have to do it too frequently - so we discuss all boundary partners at once.
4. What did you do to take advantage of any positive experiences with OM?	When we worked together with our partners…de-worming, we regulate the sample of worms. We are using the OM data for reviewing the performance of the field guides. For example, group meetings are facilitated by field guides. He gives all of the records and through comparison we can see which field guides are doing better than others.	1. OM meetings gives an opportunity to meet with the entire team. We didn't have those meetings before and it's really good to be able to sit together like that. 2. Due to OM, it helped us to look beyond the farmers/ goat keepers (the producers). The methodology widens your scope. Questions from the audience: Q: Have you changed anything in your system, in the field, based on these kind of discussions? A: One example is that we ask if more animals are treated (we asked in the meeting, but we did not know how many animals are being treated, and on what kind of treatment). Now the extension officers go to the field to find out about treatment of animals and feed this information back.
5. What have you learned about applying OM in imGoats?	OM creates opportunity to interact with all boundary partners. We have something to take with us to the traders, for example. We are more prepared and confident, approaching someone with knowledge from OM activities.We communicate better with the boundary partners through OM. Questions from the audience: Q: So looking back on our analysis and how we formulated the boundary partners, did you feel that you made the right selection? A: Not every farmer is a useful boundary partner. Some were initially identified as a boundary partner, but had a specific interest that was not the same as ours (e.g. bankers, pharmaceutical companies). From the second our third day, you can already tell who is useful and who is not. Attendance at the meeting is a clear indicator of how useful and co-operative the boundary partners are. Q: In India I think there is a lot of interest from the government - they are actively involved now and it is different for them. This is a great thing. How do you keep them motivated? A: We have told them some of the incentives, which motivate them.	A lot of information available that other M&E systems didn't capture. Especially with CARE's own experience - they are already capturing info, but not so much on behavioural change for example. There is one group we did not include specifically. Often women, people who sell meat on the market - we don't have a lot of direct contact with these people. This is a group we may have missed and we might want to think about whether we communicate with them or not.
Summing Up: For the last hour or so we have been discussing how OM went well and not well. Firstly, on what went well: the Moz team used implementation part, systematic collection of info - which they use in meetings, follow up and all. The Inida team - field guides, qualitative and quant data from the field. One important Q was how this data is being captured and how the team keep check on this data coming from the field. Secondly, on what didn't go so well: From both countries one problem was clear - that there is a large volume of data coming from the field, and how should they proceed with that? The Moz team also highlighted the difficulty in assessing change, as opposed to simply acknowledging it. Thirdly, on how to get over these problems and challenges: In India, data coding has been used. And in Moz, discussion is rotated to ensure all boundary partners receive full attention in meetings. Fourthly, on taking advantage of positive experiences: The Moz team take advantage of big group meetings by widening scope throughout the project process (not only focusing on producers). The India team narrated that since the concept of IPs - one of the enabling agencies (dept. of Animal Husbandry) said that fecel samples may be done and it has very much helped in the project. Lastly, on what we have all learned in applying OM: The Moz team said - information of behavioural change. In India, OM has provided an opportunity to interact with all boundary partners.