This paper is hight of our approach contributed to Asian Development Bank COP-Mfdr on-line discussion forum using Dr Ray Rist’s “Ten Steps to a Results-Based Monitoring and Evaluation System” publication and commented by him
“We all have to thank Nimit Leelasorn for the continuing contributions to this discussion on how to actually construct an M&E system. His detailing of the fourteen steps (!!) that are required to construct an M&E system is most helpful. A checklist of this type can be used by all of us in this arena. And again, it is not that the 14 steps are mandatory, but that if you are building such a system, you would want to be sure to consider all these 14 at one time or another….and perhaps most importantly in a sequence that makes sense for your own government or organization. Please note in his introduction that he says there is an assumption here that the generic M&E system has already been constructed and that what follows is how to customize the M&E system as needed. It is my thought that these 14 steps can also help with the initial construction and not just customization. Any new construction would need to consider these points as well. Thanks again!! Ray Rist ”
Our approach for the M&E System data management considers below listed issues that form part of entire M&E Information System strategy for short, medium and long term. Please note our approach assumes that the generic M&E Information System (PACPLAN) has already been created but require customisation for each organisation and/or indicator.
- Data/process flow analysis, to identify data source and institutional capacity of the organisation.
- Data structure and process design (data and process linkage)
- Organisation data collection capacity (M&E Information System, hardware,skill resources)
- Data collection and using external or internal resources
- Data volume (priority should be given to large data volume handling)
- Data collection approach (top down or bottom up)
- Impact of data collection on the organisation
- Data coding
- Data transmitting
- Indicator data (Policy, Input, Activity, Output and Outcome)
- Data Quality
- Data Harmonisation
- Data Analysis/Dissemination
- Tracking Outcome
Above considerations should be treated as a guide only – the real implementation must be done with open mind and readiness to change to suit local situation and environment, as this could be different from one developing country to another and one district to another. M&E Information System and process must be designed with extreme flexibility to allow for adjustment during the implementation.
In IT system development for Local/National government the M&E Information System could incorporate experience gained to create a generic solution which can be applied across various indicators. This would minimise efforts of building the M&E Information System for each indicator and/or organisation.
1. Data and Process Flow Analysis
Thorough analysis should be conducted prior to the design of M&E System to determine suitable data structure, data source design, as well as reporting usage and most effective process flow of M&E System in the organisation. Due to limited capacity in developing countries, the data flow could only be restricted to vertical movement (top to bottom or bottom to top) but data structure of the M&E Information System should support multiple viewing in vertical, horizontal and diagonal, etc., i.e. grouped by country where all regions’ and provinces’ outputs are compared and evaluated, or from the policy level data which should be able to be drilled down to project and indicator data level to analyse the project output and status. From the indicator level it should be drilled up to the project level to identify the project and office not able to achieve the performance target. Indicator data ability to move between levels could be predefined, but due to limited exposure to
information systems in developing countries environment, it would be more realistic to determine it’s needs, based on what data has actually been collected, with M&E Information System able to facilitate these needs via real-time report generator.
2. Data Structure Design
With well structured data design, the M&E Information System should be able to progressively provide more and more complex analysis as the organisation’s data collection grows. A sound and solid data structure would facilitate the on-going need of data by stakeholders. Good data structure will facilitate most data analysis reporting requirements, therefore the M&E Information System should be flexible enough to accommodate the organisation’s data structure design (for current and future organisations) and this should be the major emphasis at the initial stage as this is what will provide the facility for the stakeholders to create real time reports that suit them. The design of data structure should represent organisation, goal, objective, location, periodical reporting, etc. However, M&E Information System should also allow for the growth of that data need as the organisation matures and develops.
3. Organisation data collection capacity
Various strategies must be deployed, based on organisation’s assessment, capacity and data/process design. Strategy used in one organisation may not simply be re-applied in the next one. The M&E Information System design should be flexible enough to handle various organisations’ data collection strategies. M&E System design should facilitate the gradual data collection strategy, as well as provide facility for the organisation to develop more in-depth data collection.
4. Data collection and using external or internal resources
Consideration must be made for external resources that have less understanding of the organisation data environment V/S already very busy internal resources.
5. Data Volume
Data volume would also play major part in consideration of the data collection strategy as this stipulate capacity requirement to handle the volume data. The approach to small data volume would be different to approach used for larger volume. Small data volume could be better handled manually without involvement of the painstaking task to develop the M&E Information System, unless time and resources are plentiful.
6. Data collection approach (top down or bottom up)
M&E Information System data entry with top down approach should commence at the head office,
- This approach would allow system data entry and report to be tested and adjusted before deploying the M&E Information System while the information from the system can be used by management.
- Capacity can be planned and built at the provincial/district offices.
- Allow the provincial/district office to be prepared for the data collection process without having to learn the M&E Information Systemat the same time.
- Minimum training effort would be required when the system is ready to be deployed at the provincial office where local officials could commence their own data entry, as the M&E Information System allows local self monitoring and improving of data quality
both at the provincial and head office.
This is the gradual approach, and will take longer time to complete but it is causing less strain on the resources.
With bottom up approach, data entry is being initially performed at provincial and district offices as data could be monitored and corrected at source. Following issued must be considered:
- There is a sufficient hardware and data entry resource at the district office to handle the system data entry
- The data (input, activity and output) from the input form to be entered to the M&E Information System directly at the district office andsystem report to allow data to be locally monitored and validated at source, then transmitted to the head office for organisation’s andpolicy consolidation.
Impact on training of resources as large number of provincial/district officials require to be trained. - Impact on system resources as specialists need to handle unexpected problems due to local environment and system not havingbeen fully tested.
This approach puts enormous strain on the resources, and would not be recommended, unless there are sufficient resources, experience and funding.
7. Impact of Data Collection on the Organisation
Care must be taken with data collection strategy as most government officials, particularly at the district/project level are already busy and overloaded with work. Requiring extra data collection effort would not be warmly welcome, especially if unable to demonstrate less reporting effort in the short term. Frequency of the data collection (i.e. weekly, Monthly, Quarterly, Yearly) should coincide with the data reporting requirement. This will create the impact.
8. Data Coding
Developing countries have limited capacity of “data-skilled” human resources. M&E Information System should allow codifying to be done by a limited number of officials at the head office and M&E Information System should translate those codes into form and screen for easy entering of data by project or district officials.
9. Data Transmitting
This also depends on organisation’s infrastructure capacity, i.e.: – is there connection available. The M&E Information System should allow the information to be transferred by all possible means (i.e. network or internet, or the diskette, or portable media, etc.) to the head office for country, plan or organisation data consolidation.
10.Indicator data (Policy, Input, Activity, Output and Outcome)
Outcome could be established by various levels of stakeholders – national level politicians with election mandate, where the country development plan was defined in legislative process in line with MDG, and local level politicians concerned over the community expectation. The M&E Information System must be able to support multiple stakeholders’ expected outcomes and facilitate the harmonisation of all outcomes. All outcomes must be harmonised to achieve the country development goal while satisfying all stakeholders’ expected outcome. Most developing countries confront multiple development plans, strategies and outcomes. The M&E Information System must be able to handle these. The outcome indicators are established at the policy level upon agreement of all stakeholders. This agreement process could take very long time in developing countries.
11. DATA Quality
It is important to note that data collection should be performed by the unit in the organisation/originator that is accountable for this data, i.e.: the project officer should fill in activity, output data in the input form and enter into the M&E Information System, where correction and adjustment can be made. “To ensure the quality of the data, it must be used at all levels in the organisation” As the on-going improvement of indicator data quality, M&E Information System must provide reports to assist in harmonisation of the indicator data and improving consistency between different stakeholders.There are various data validation techniques (i.e. check sum…) that could be applied to the information system but most important in data accuracy is usage by stakeholders. Once stakeholders have confidence in the data from the information system then the M&E System would be sustained for the long term.
12. Data Harmonisation
Facilities in M&E Information System allow stakeholders, at the policy and project level, to harmonise the data. All levels of indicators can be harmonised using the M&E Information System reporting tools to illustrate live data to gain stakeholders consensus. The accurate information on input, activity, output and/or outcome data, through M&E Information System reporting would lead to agreement on stakeholders’ expected outcomes.
13. Data Analysis/Dissemination
In developed country, data analysis of vast amount of data using sophisticated reporting and graphic tools is the common approach but in developing countries experience in this area is very limited and would take time to mature. Collected data would assist in this learning process with M&E Information System providing stakeholders with the real time reporting facility.
14. Tracking Outcome to Determine Policy Impact
In establishing the M&E system in developing country, the approach for the government to measure and improve their performance could overshadow tracking the outcome. There are issues that existed before and outside the development plan, i.e.: bird flu crises, excessive number of stray dogs, unrest or conflict situation. Effective handling of such situations would be seen by general population as effective management even though it may not be required to achieve the development outcome.
Various approaches in tracking outcome could be used by M&E Information System to collect the survey data from the provincial offices to form the base line. In the subsequent year sampling would be surveyed to establish the outcome to determine the impact of the previous year’s policy/project thus allowing future policy adjustment.