“ToxRTool”, a new tool to assess the reliability of toxicological data

Abstract

Evaluation of the reliability of toxicological data is of key importance for regulatory decision-making. In particular, the new EU Regulations concerning the registration, evaluation, authorisation and restriction of chemicals (REACH) and classification, labelling and packaging (CLP) according to the new globally harmonised system (GHS) rely on the integration of all available toxicological information. The so-called Klimisch categories, although well established and widely used, lack detailed criteria for assigning data quality to categories. A software-based tool (ToxRTool) was developed within the context of a project funded by the European Commission to provide comprehensive criteria and guidance for reliability evaluations of toxicological data. It is applicable to various types of experimental data, endpoints and studies (study reports, peer-reviewed publications) and leads to the assignment to Klimisch categories 1, 2 or 3. The tool aims to increase transparency and to harmonise approaches of reliability assessment. The tool consists of two parts, one to evaluate in vivo and one to evaluate in vitro data. The prototypes of the tool were tested in two independent inter-rater experiments. This approach allowed the analysis of the performance of the tool in practice and the identification and minimisation of sources of heterogeneity in evaluation results. The final version, ToxRTool, is publicly available for testing and use.

Introduction

The use of existing data on toxicological properties of chemicals is necessary in various regulatory contexts on grounds of animal welfare considerations (avoidance of unnecessary animal experiments) as well as for economic reasons. The new European chemicals policy, Regulation EC 1907/2006 (REACH), places strong emphasis on the use of existing data. According to the relevant REACH guidance documents (ECHA, 2008), the registration process should always start with a thorough evaluation of all available data with regard to whether the information is reliable and sufficient to fulfil the information requirements. Also, under the new globally harmonised system for classification and labelling of chemicals (GHS) existing data will be the most important source for classifying substances with respect to their hazardous properties.

However, existing data vary largely in quality and consequently the objective evaluation of data quality gains importance. The Occupational and Public Health Specialty Section of the US Society of Toxicology recently expressed the concern that differences in the evaluation of data quality may counteract the efforts to harmonise classification of substances under GHS (SOT, 2007).

In 1997, Klimisch et al. published a categorisation system with the aim of assigning toxicological data to one of four reliability categories (see Table 1). The authors introduced the following definitions (Klimisch et al., 1997):

• Reliability: evaluating the inherent quality of a test report or publication relating to preferably standardised methodology and the way the experimental procedure and results are described to give evidence of the clarity and plausibility of the findings;

• Relevance: covering the extent to which data and/or tests are appropriate for a particular hazard identification or risk characterisation; and

• Adequacy: defining the usefulness of data for hazard/risk assessment purposes. When there is more than one set of data for each effect, the greatest weight is attached to the most reliable and relevant.

These terms and the reliability categories are widely applied in various regulatory programmes such as the OECD High Production Volume Programme and REACH and are explicitly referred to in the REACH guidance documents (ECHA, 2008).

Whereas some aspects of data quality were mentioned by Klimisch et al. (1997), explicit criteria allowing conclusions to be reached on the reliability categories are lacking. Studies performed according to recent guidelines and under conditions of good laboratory practice (GLP) are generally considered to be of high reliability (category 1). However, existing data may originate from times when such quality-assured generation of data was not yet common or were published in peer-reviewed journals that differ with regard to the level of documentation required. In particular, guidance is lacking on how to distinguish reliability categories 2 and 3 (for definition see Table 1), which in the regulatory context is the most crucial differentiation. Data of reliability category 1 or 2 can be used as stand-alone information to cover a specific endpoint, whereas category 3 information at best may serve as additional information in weight-of-evidence approaches.

To provide such guidance, a research project was initiated by ECVAM, the European Centre for the Validation of Alternative Methods, of the European Commission's Joint Research Centre in Ispra, Italy. The project was aimed at developing criteria and a transparent methodology for evaluating the quality of existing experimental toxicological data. While a methodology has been proposed for the development of data quality criteria for ecotoxicological data (Hobbs et al., 2005), similar initiatives have not been taken for toxicological data.

As both relevance and adequacy of data depend on the specific regulatory context in which it is to be used, the project focused on the inherent quality, i.e. the reliability of toxicological data, which is independent of the use of these data for regulatory purposes. The objective was to establish a set of assessment criteria, with which to evaluate toxicological data, and implement these criteria in a readily available and user-friendly tool facilitating the documentation of the assessment. A reliability assessment tool called ToxRTool (Toxicological data Reliability Assessment Tool) was developed in this project intended to be used by scientists routinely dealing with reliability assessment of toxicological data. Ultimately the tool will increase transparency and provide guidance for more harmonised approaches to data quality evaluations. This effort is in agreement with recent activities focussing on the use of evidence-based approaches in toxicological practice (“evidence-based toxicology”, EBT, www.ebtox.org) (Hoffmann and Hartung, 2006).