Harmonisation... Conferences Belgirate Outcome Summary of meeting...

The 7th International Conference on Harmonisation within Atmospheric Dispersion Modelling for Regulatory Purposes (Belgirate, Italy, May 2001).

Harmonisation and model evaluation

Summary of meeting on Harmonisation and model evaluation held on Wednesday May 30, 2001 in Belgirate.
The meeting was convened by John Irwin and Helge R. Olesen and was one of three simultaneous group meetings.

Who should read this?

This discussion summary is primarily of interest to participants in the Harmonisation conference and for people involved in related work.
If you do not belong to these groups, but are looking for tools that you can use in your work, go to the web page "Outcome of discussions and announcements of common tools".

Overview

As an introduction to debate there were two presentations:

• Helge Olesen: How can we pool experiences?
• John Irwin: The ASTM standard guide - status, plans and additional comments

Interspersed with these two papers and following them, there were comments, questions and suggestions from the group. These points of discussion are listed below in keyword form - some with a bit of accompanying explanation.

My overall conclusions from the meeting ("I" refer to Helge R. Olesen, one of the convenors) is that the harmonisation work can best be carried on by creating a variety of common tools. Of particular importance are tools for model evaluation. There are many classes of models, and consensus on methods needs to be built up for the various classes of models. Such consensus methods can best gain widespread use if they are formulated as standards in the framework of standardisation organisations such as ASTM and ISO.
Funding for the creation of model evaluation tools has been scarce, but even without funding progress does occur - although slowly.

Also important in respect to harmonisation is harmonisation of model output.

Points of discussion

Comments, questions and suggestions from the group are listed below in keyword form - some with a bit of accompanying explanation.

1. What are "good" model characteristics? FB, NMSE?

   In view of all the work performed on model evaluation, are we now able to define good characteristics for model performance?

2. Publish datasets including model simulation results. Link to Model Documentation System web page.

   From the web pages of the EEA *Model Documentation System there should be links to model evaluation results, obtained through common, recognised methodologies, such as the Model Validation Kit and the ASTM standard. A modeller often has certain degrees of freedom when deciding which input data to use. Therefore, not only a summary of the results should be available, but also full input and output datasets for the model in question.

3. Darwin and model development

   There is a striking difference between the situation in the USA and in Europe: In the USA, relatively few models are used for regulatory purposes, whereas models abound in Europe. The USA system has evolved to a "stiff" system. It is difficult to imagine Europe accepting a similar system.
   A normal user is not able to judge about the quality of models.
   An "acceptance band" may help, but maybe Darwin - the survival of the fittest - will also.

4. We need harmonisation on decision making in the interpretation of "compliance"

   The result of decision-making very much depends on the decision-maker - more than on the models applied. This was mentioned as a particular problem in the UK, where regulators have much freedom when deciding how contributions from neighbouring sources should be handled.

5. Do we have consensus on how to rank model performance?

   How bad does a model have to be, before we will say that it does not agree with observations?

6. Why are there so many street canyon models?

   Could we summarise what is known and solve the question?

7. Need to have access to multiple data sets for robust testing?

8. Quality assured datasets should have funding.

9. We need a "champion"

10. There are differences in "openness" for emissions information (varies by country).

11. City building inventories.

12. What should be standardised?

    The value of standards was emphasised in the presentation by John Irwin. Methodologies, which are endorsed by standards organisations (such as the ASTM), are likely to get widespread acceptance. The question was posed: Within which areas is the need for standardisation greatest? The following list of priorities resulted from the meeting. (The numbers in brackets indicate votes resulting from a small ballot on priority within the group).

    (10) Data bases for evaluation

    (10) Methods for evaluation

    (5) How to describe a model (user guide content)

    (4) Standards on emissions data

    (2) Application of models (preparation of input data; preparation of output; definition of uncertainty)

    (1) Street canyon modelling

    (1) Data assimilation

    (0) Set out met instruments

    (0) Protocols for development of model evaluation data sets (how to plan an experiment)

This summary was written by Helge R. Olesen. Comments and corrections are welcome.