MSC.1-Circ.1494 - Guidelines On Harmonization Of Testbed Reporting (Secretariat)

MSC.1/Circ.1494 21 November 2014

1 The Maritime Safety Committee, at its ninety-fourth session (17 to 21 November 2014), approved the Guidelines on Harmonization of testbed reporting, prepared by the Sub-Committee on Navigation, Communications and Search and Rescue at its first session (30 June to 4 July 2014), as set out in the annex.
2 Member Governments are invited to bring the present circular to the attention of those involved in the planning of testbeds related to e-navigation and the reporting of their results to the Organization.

MSC.1/Circ.1494 Annex, page 1
1 Introduction
1.1 This document offers guidance on the reporting of results of e-navigation testbeds.
2 Benefits and scope of the guidelines
2.1 Harmonization of the reporting of results from testbeds will allow the results of e-navigation solutions being tested to be shared and compared effectively. Harmonization also allows future meta-analyses1 of specific aspects. Different organizations can recreate trials both to verify results and refine various factors within the trials, in order to further develop the concepts being trialled.
2.2 This guideline includes the following:
.1 Considerations when planning a testbed (annex 1); and
.2 Template for reporting the results of testbeds (annex 2).
3 Background
3.1 During the development of e-navigation, a growing number of testbeds have been evaluated. Consequently, NAV 58 agreed to the development of Guidelines on Harmonization of testbed reporting.
4 Testbeds
4.1 A testbed (also commonly spelled as "test bed" in research publications) is a platform for trialling development projects. Testbeds generally involve rigorous, transparent and replicable testing of, for example, scientific theories, computational tools and new technologies.
4.2 E-navigation testbeds allow for early detection of new system functionality, operational usability, areas of enhancements and identification of weaknesses. Ideally, testbeds should be linked to human-centred design processes to ensure any operational usability issues are detected early. Testbeds should not, necessarily, be limited or restricted by current or planned architecture, data structures or existing procedures. Considerations when planning a testbed are given in annex 1.
4.3 Ideally, testbeds should be conducted in a controlled environment so that they do not adversely affect real-life situations, existing services and maritime safety. Conclusions can be drawn for many aspects of testbeds such as functionality, usability, feasibility and risk. As e-navigation evolves from concept to operational reality, the importance of testbeds will continue to grow.
1 Meta-analyses are when results from a great number of experiments/tests are gathered, compared and trends, if any, analysed. A single experiment or test usually only offers limited information on a specific question/hypothesis; meta-analyses, however, can represent a bigger picture.
MSC.1/Circ.1494 Annex, page 2
4.4 There are testbeds that, while being not directly identified as e-navigation testbeds, are nevertheless relevant to e-navigation. The reporting of results from such testbeds is encouraged.
5 Harmonization of reporting of testbed results
5.1 As a number of testbeds are established, it is important that the results of testbeds are shared, as there could be outcomes and lessons learnt that will be useful to the maritime community. In order to do this and to allow for ready comparison of the relevant elements of testbed results, reporting of the results of testing of e-navigation solutions, systems and services should be harmonized.
6 Testbed results
6.1 For testbed results to be useful to other parties, tests/simulations/trials should ideally have scientific rigour for set-up, collection of data, analysis, etc. Additionally:
.1 the results presented should be objective;
.2 trials should be reproducible;
.3 data gathered should be statistically sound and meet generally accepted "scientific standards"; and
.4 testbed results should be presented in acceptable scientific formats (e.g. they should be suitable for publication in a peer-reviewed publication).
6.2 A framework, by way of a template for reporting has been developed (see annex 2) that addresses the presentation of results. This should be taken into account when reporting results of testbeds related to e-navigation to the Organization.
MSC.1/Circ.1494 Annex, page 3
1 General
1.1 It is advisable that the following considerations are taken into account when planning testbeds as it will assist in the harmonized reporting of testbed results.
1.2 When planning testbeds, the e-navigation solutions selected should ideally be linked to user needs and the objectives of e-navigation. Where possible, solutions should address gaps identified in the e-navigation gap analysis.
1.3 It is recommended that testbeds take into account a structured, transparent, objective and repeatable methodology. Where the output is in the form of software tools, these should ideally be open-source, with arrangements in place for collaboration, incorporating user feedback and identified improvements.
2 Architecture
2.1 It is advisable that, without restricting innovation, testbeds align with the approved overarching e-navigation architecture and solutions including the technical/operational services in the Maritime Service Portfolios.
3 User and stakeholder involvement
3.1 Testbeds should ideally involve users and stakeholders at every stage – from planning to implementation and assessment of results.
4 Human-centred design and quality assurance principles
4.1 Human-centred design and quality assurance principles should be taken into account during the development of e-navigation solutions.
5 Data structures
5.1 The agreed Common Maritime Data Structure (CMDS) is the IHO S-100 Geospatial Information (GI) Registry. Testbeds should therefore preferably use the IHO S-100 framework for data modelling and exchange. Other data model frameworks may be used for testbeds. However, it is advisable that, for results to be of value to the development of e-navigation, steps should be taken to incorporate solutions into the IHO S-100 framework.
6 Reference to the e-navigation documentation
6.1 It is advisable that testbeds highlight links to user needs, gap analysis and solutions already identified.
7 Sharing of information
7.1 Information on testbeds should be provided to the Organization.
MSC.1/Circ.1494 Annex, page 4
1 General
1.1 The purpose of this reporting template is to serve as a harmonized framework for reporting results from e-navigation testbeds. In order to assist with the reporting of testbed results and to ensure these are valuable to the e-navigation development community, it is advisable that all headings are completed – even those for which there is no information.
1.2 Testbed information will assist other organizations to learn more about the solution being tested. It may also offer other ideas to expand and further develop the solution.
2 Contents of the reporting template
Note: Symbols have the following meanings:  Sub-section/Sub-heading o Tick box (choose one or more)  Free text field
1 General Information
 Name of testbed  Location of testbed  Time and duration of testbed  Status (planned, completed or ongoing)  Contact person(s)  Testbed website  Organization(s) involved  Funding programme and budget
2 Executive summary
3 Testbed Information
 The type of user group(s) involved in the test
o Shipboard users o Shore-based users o SAR users
 Details of e-navigation gap(s) considered for the testbed (some examples are given below. For a complete list, please refer to the MSC 91 report):
o Information/data management o Effective and robust voice communication and data transfer o Systems and equipment o Ship reporting o Traffic monitoring o Familiarization
MSC.1/Circ.1494 Annex, page 5
 The category of e-navigation gap(s) considered in the testbed
o Technical o Regulatory o Operational
 Details of e-navigation solution(s) considered in the testbed (the prioritized solutions are listed below):
o S1: Improved, harmonized and user-friendly bridge design o S2: Means for standardized and automated reporting o S3: Improved reliability, resilience and integrity of bridge equipment and navigation information o S4: Integration and presentation of available information in graphical displays received via communication equipment o S9: Improved Communication of VTS Service Portfolio
 The category of e-navigation solution(s) considered in the testbed
o Technical o Regulatory o Operational
 Links to similar / relevant testbeds (if any)
4 Testbed methodology
 Methodology used for data collection
 Method  Validity  Reliability
 Summary information on testbed respondents / participants
 Number  Background  Experience  Demographics (e.g. age, gender)
 Procedure used in the testbed
 Testbed setup  Technical solutions used  Standards  Guidance documents  Standard Operating Procedures  Analysis of data
5 Testbed results
 Summary of findings:
 Presentation of data (e.g. statistics)  Users assessment and experience  Other comments
MSC.1/Circ.1494 Annex, page 6
6 Conclusions and recommendations
 Conclusions
 Lessons learnt
 Recommendations
7 Publications
 Peer-reviewed publications  Technical papers  Reports  Communication material (e.g. videos, flyers, pamphlets, etc.)
8 Reference material
 List of reference material used in the testbed
No comments yet. Be the first to add a comment!


All logbook parts
LogBook Parts from Aleksandr Makarov