Activity 8.1: Arrange for monitoring and evaluation

GLOSSARY TERMS

By Admin Eltis / Updated: 12 Nov 2015

Rationaleinfo-icon

Monitoringinfo-icon and evaluationinfo-icon of both the planning process and of the implementation of the measures are crucial to the effectiveness of the plan. A monitoring and evaluation mechanism helps to identify and anticipate difficulties in the preparation and implementation of the SUMP, and, if necessary, to “repackage” measures in order to achieve targets more efficiently and within the available budget. It will also provide proof of the effectiveness of the plan and its measures. This allows those responsible for the actions to justify where money was spent.

The reporting should ensure that the results of the evaluation feed back into the public debate, thus enabling all actors to consider and make the necessary corrections (e.g. if targets are achieved or if measures appear to be in conflict with one another). The monitoring and evaluation mechanisms should be defined early and become an integrated part of the plan.

 

Aims

  • Build a suitable monitoring and evaluation arrangement into the plan to help identify barriers and drivers for measureinfo-icon design and implementation, and to enable timely and effective responses.
  • Determine how the degree of measure implementation and targetinfo-icon achievement will be assessed.
  • Develop suitable mechanisms to assess the quality of the planning process.
  • Make monitoring and evaluation arrangements an integral part of the SUMP document.

 

Tasks

  • Connect indicatorinfo-icon selection for monitoring and evaluation to setting SMART targets (> Activity 5.2). Choose a few easily-measurable indicators and avoid information overload.
  • The ex-post evaluation reviews the sustainable urban mobilityinfo-icon planning and implementation stages, and the overall results of the decision making pro-cess. It should include the following areas:
    • Output (action taken): Newly constructed infrastructure (e.g. x km bicycle lanes) or new transport and mobility services in operation (e.g. x new buses) – using output indicators.
    • Outcome (impact of action): Real and measurable improvements in quality of lifeinfo-icon and the quality of transport services (outcome indicators) should be the main focus. Examples are congestion (vehicle delay) or the number of cycling trips. Include intermediate outcomes if possible; these represent milestones towards key outcome targets. The indicators should measure outcomes directly, or measure how outputs are demonstrably related to outcomes.
    • Planning process of the measure implementation: The efficient use of resources as an investment in measures; the process of implementation: e.g. timing of implementation, quality (process indicators).
  • Include qualitative and quantitative indicators.
  • Anticipate arrangements for ex-ante evaluation (appraisalinfo-icon – a process of checking how well a scheme or strategyinfo-icon will perform, can assist in making efficient choices between options.) and ex-post evaluation of plan preparation process.
  • Perform a datainfo-icon auditinfo-icon (what is available? where are gaps?) and if necessary develop a data collection strategy (quantitative and qualitative indicators). This is linked to the data audit in Activity 3.1 (Prepare an analysis of problems and opportunities). These activities are related and should be coordinated with each other.
  • Determine how monitoring and evaluation will be integrated in the Sustainable Urban Mobility Planinfo-icon. Develop a work plan for monitoring and evaluation activities that is integrated with the project schedule.
  • Define trajectories to measure intermediate outcomes and assess the progress in achieving targets.
  • Define clear responsibilities of well skilled staff members – or an external partner – for monitoring and evaluation. Ideally the responsibility should be with an independent body.
  • Clearly define the available budget and activities for monitoring and evaluation – typically this should be at least 5% of the total available budget.
  • Plan for a minimum stakeholderinfo-icon involvement in monitoring and evaluation.

 

Activities beyond essential requirements

  • Integrate an assessmentinfo-icon of costs and benefits of the Sustainable Urban Mobility Plan development process.
  • Plan for extensive stakeholder involvement in monitoring and evaluation.
  • Involve peers from other cities in the feedback process.
  • Coordinate with relevant local and regional stakeholders on regional indicators.

 

Timing and coordination

  • Consider monitoring and evaluation needs from the outset, especially when developing SMART targets and selecting related indicators (> Activity 5.2).
  • Include ex-ante evaluation (appraisal) in the status analysis (> Activity 3.1), scenarioinfo-icon development (> Activity 3.2) and action and budget plan (> Activity 7.2).
  • Include arrangements for monitoring and evaluation arrangements for ex-post evaluation in Sustainable Urban Mobility Plan document (> Activity 9.1).

 

Checklist

Suitable indicators (based on indicators selected in Activity 5.2 Develop SMART targets) selected.
Suitable monitoring and evaluation tools agreed on.
Work plan and responsibilities for data collection and management agreed.
 

For more information

Guidance tools and sources on monitoring and evaluation.

Source

Topics covered

Hyperlink

DISTILLATE, UK (2008)

Guidance on the development of a monitoring strategy and the selection of indicators. See Project C – Indicators.

www.its.leeds.ac.uk

MAX (2009),

MaxSumo. Guidance on how to plan, monitor and evaluate mobility projects.

MaxSumo offers an opportunity to effectively plan, monitor and evaluate mobility projects and programmes aimed at behavioural change.

Available in EN, DE, ES, FR, NL, PL, PT, SE

www.epomm.eu

GUIDEMAPS (2004) Handbook, Volume 1: Concepts and tools

Measuring indicators, p. 59 f., 61

Evaluation methods (Cost effectiveness, cost-benefit analysis, least cost planning, multiple criteria analysis), p. 79

www.osmose-os.org

GUIDEMAPS (2004)

Handbook, Volume 2: Fact Sheets

Measuring indicators, p. 70 f.

Tools for tracking progress, p. 73

Measuring outcome indicators, p. 76

Post implementation evaluation, p. 78

www.osmose-os.org

PROSPECTS (2003), A Methodological Guidebook

Appraisal and evaluation, p. 25 f., 33 ff.

Implementation and monitoring, p. 27 ff.

Basics of CBA, p. 99 f.

www.ivv.tuwien.ac.at

PROSPECTS (2002), Evaluation tools (Deliverable 2)

Covering a wide range of methods and tools for evaluation. See whole document.

www.ivv.tuwien.ac.at

 

Examples

Toulouse, France: Arranging for monitoring and evaluation

The new transport plan of the agglomeration of Toulouse set up a number of initiatives that should assure an accurate monitoring of the realisation of the plan and regular evaluation of its results. In the framework of the “partnership monitoring commission”, all institutions, associations and mobility-related organisations meet at least once a year to discuss the progress made.

 

West Yorkshire, England: Monitoring of targets and indicators – West Yorkshire Local Transport Plan (LTP2) (excerpt of table)

INDICATOR

RELEVANT TARGET

DATA SOURCE AND COLLECTION TECHNIQUES

TIMESCALE

Accessibilityinfo-icon

Mandatory M1

Use of Accession modelling suite

Updates produced annually and/or during services changes

Bus punctuality

Mandatory M2

Roadside Surveys and RTPI system

Updates produced annually

Satisfaction with local bus services (BVPI 104)

Mandatory M3

Information supplied by ODPM. Supplemented by Metro market research

Data produced every 3 years

Annualised index of cycling trips

Mandatory M4

A representative selection of sites across West Yorkshire have been chosen to reflect a variety of cycling environments. Both on and off road sites are monitored. Data collected both automatically and manually

Automatic sites collect data continuously. Manual counts undertaken in neutral months

Average journey time per person mile on key routes

Mandatory M5

14 routes have been selected across West Yorkshire. Occupancy, flow and journey times undertaken on each route

Annual counts carried out in neutral months

Change in peak period traffic flows to urban centres

Mandatory M6

Automatic Traffic Counts (ATC) on five urban centre cordons

Annual counts carried out in neutral months

Mode share of journeys to school

Mandatory M7

Methodinfo-icon of collection deferred until 2007

 

Satisfaction with LTP funded public transport facilities

Local L1

Market research surveys

Scheme by scheme assessment

Cycling trips to urban centres during the morning peak

Local L2

Mode split surveys into five main urban centres across West Yorkshire

Annual counts carried out in neutral months

AM peak period mode split to urban centres

Local L3

Mode split surveys into five main urban centres across West Yorkshire

Annual counts carried out in neutral months

Peak period rail patronage

Local L4

Peak period surveys at Leeds rail station

Annual counts carried out in neutral months

Patronage on Quality Bus Corridors

Local L5

Electronic ticket machine data on selected routes

Scheme by scheme assessment

Number of pedestrians KSI in road traffic collisions

Local L6

STATS 19 Data

 

More info: 

Toulouse, France: Arranging for monitoring and evaluation

The new transport plan (PDU) of the agglomeration of Toulouse set up a number of initiatives that should assure an accurate monitoringinfo-icon of the plan and regular evaluationinfo-icon of its results. They comprise the following activities:

  • Establishment of a “partnership” monitoring commission
  • Installation of an “urban development/ mobilityinfo-icon commission”
  • Continuation of the PDU observatory
  • Creation of a mobility cost account
  • Development of balanced score cards

The revision of the PDU permitted the agglomeration to engage a large number of public and private stakeholders. In the framework of the “partnership” monitoring commission, all institutions, associations and mobility-related organisations meet at least once a year to discuss the progress made, if possible making use of the intermediate evaluation results provided by the PDU observatory, which follows the progress made in Toulouse’s Urban Mobility plan. In addition, the observatory will investigate whether the impacts of these actions are in accordance with the envisaged effects. It will also observe whether the general objectives are fulfilled as described in the respective PDU laws. Every year, an intermediate evaluation update will take place. A full evaluation of progress and results is obligatory 5 years after the official approval of the PDU.

Some of the indicators used feed directly into or come from the legally required strategic impact evaluation. The indicators that should provide the larger overview of the mobility and transport trends in the agglomeration of Toulouse come from:

1) the household mobility study

2) the surrounding road and ring road study

3) the public transport origin-destination study

The urban development/mobility commission was established to assure coherence between the urban development options within the perimeter of the PDU and the organisation of the transport infrastructure. Both the AOUT (authorityinfo-icon with transport competences responsible for the PDU) and the SMEAT (authority responsible for the SCOT, urban development coherence scheme) are participating in this commission. The mobility cost account is a toolinfo-icon made obligatory by the law SRU, which imposes the creation of a number of tools that assist public and private decision making which has an impact on mobility practices. The mobility cost account is one of these tools. It permits the agglomeration to visualise the costs to the users and to society. A balanced score card will be set up that integrates all actions of the PDU. It will provide periodic updates on the precise content of the measureinfo-icon, the progress made and the envisaged timing of realisation.

Source: Rupprecht Consult based on input from Toulouse (Revised PDU project of Toulouse, January 2011)