Introduction
The best way to understand the use and value of a solution is often through real world examples. This page presents two selected case studies that illustrate the use of the Test Bed:
Each case study has a similar structure, starting from the business context and needs, followed by details on how the Test Bed was used, and resulting in an account of the benefits in using it as a conformance testing solution.
Case study: Business Registers Interconnection System (BRIS)
The following case study explains how the Test Bed was used to successfully realise the conformance testing needs of the Business Registers Interconnection System (BRIS). It has been co-authored by the Test Bed team and Krzysztof Iwanski (DIGIT B.2), IT Project Manager – Project Manager for BRIS.
Business context
The Business Registers Interconnection System (BRIS) is a cross-border solution developed as a joint effort by EU governments and the European Commission. Its goal is to provide the infrastructure to facilitate access to information on EU companies for the public and to ensure that EU Business Registers can communicate to each other electronically in a safe and secure way in relation to cross-border mergers and foreign branches. The legal foundation for BRIS was laid out in Directive 2012/17/EU on the interconnection of Business Registers, now replaced by Directive (EU) 2017/1132, and the Implementing Regulation (EU) 2015/884.
Specifications and technical architecture
The public interface for BRIS is the European e-Justice Portal, that provides a single point of access to the information recorded in National Business Registers. Searches for information through this interface result in the querying of one or more Business Registers, with retrieved results being aggregated and presented in a uniform manner. To ensure the consistency of information, especially when dealing with cross-border events involving multiple Member States, BRIS foresees the exchange of messages between Business Registers, passing through a central EU node, termed the European Central Platform (ECP) that manages monitoring, routing and statistics.
To ensure the interoperability of National Business Registers, both between themselves and with the ECP, BRIS defines detailed specifications governing its underlying information exchange. These specifications refer to:
- The content of messages, representing information such as company details.
- The message exchange flows, defining per case the sequence of expected messages.
The message content specification is based on XML, defining for each message type the structure of its metadata and payload. As a messaging solution, BRIS uses the CEF eDelivery Building Block, providing reliable and secure message transfer using a four-corner model whereby Business Registers connect to the BRIS network via Access Points. Moreover, to facilitate message exchange, BRIS offers Business Registers a specific Access Point plug-in for simplified sending and receiving of messages.
Conformance testing needs and challenges
The key conformance testing goal for BRIS is to ensure that National Business Registers correctly implement its specifications before joining the live network. Specific needs are summarised as follows:
- Validate messaging flows ensuring conversational consistency (match responses to requests).
- Validate individual messages’ content based on the scenario being tested and the selected, per test session, test data.
- Test both communication directions (Business Register to the ECP and vice-versa).
- Cover error scenarios whereby Business Registers are expected to gracefully handle bad requests and unexpected failures.
A key testing challenge is in the fact that the BRIS message specifications evolve and that the connected Business Registers may be operating with different supported message type versions. This makes it critical to have a clear view over the exact message type support levels per Business Register. Considered along with the need to support error scenarios, this makes it difficult to use an actual BRIS endpoint as a test instance.
A further challenge is in the test data used to generate test messages. This test data needs to be synchronised with the National Business Registers, to ensure that e.g. when requesting the details for a company, the company data in question indeed exists, and expected responses can be predetermined. Message validation needs to be flexible, ensuring that it focuses on matching business information while skipping variable parts such timestamps and unique identifiers.
How was the Test Bed used?
The Test Bed was introduced for BRIS as a replacement of a custom “BRIS emulator” that was initially used for point-to-point tests with the Business Registers. Although this emulator implemented the BRIS messaging protocol and important validation logic it lacked a self-service approach to testing and lacked an interface to visualise testing progress, both for project staff and for Member State testers. The Test Bed was introduced to provide such missing features but also to reuse the work already invested in the emulator. This was achieved by extending it with two standard Test Bed SOAP APIs:
- The GITB messaging API, to trigger the sending and receiving of messages via the emulator.
- The GITB validation API, to request the emulator to validate a session’s message exchange.
Furthermore, the emulator was extended to manage Member State test data that could be selected through test cases and signalled to the emulator through its new messaging API (e.g. “use random Belgian company“). Given its integration with the emulator, the Test Bed, and in particular the shared Test Bed instance, was connected to the test BRIS network as a simulated central node (the ECP), allowing Business Registers to exchange messages with it as they would with the actual ECP.
In terms of test organisation, the combination of message type and version was mapped to the Test Bed’s “specification” concept, allowing Business Registers to be tested incrementally and in a fine-grained manner for their specific supported message types. The Test Bed’s “organisation” and “system” concepts were respectively mapped to Member States and Business Registers simplifying the testing of multiple Registers per Member State. Numerous test cases were defined, organised thematically in test suites, covering communication initiated by the Test Bed (simulating the ECP) as well as by Business Registers.
Testing progress and major milestones
BRIS developed its initial conformance test cases during 2018 in parallel to using its existing emulator software as a temporary testing solution. In February 2019 the Test Bed was used for the first time by Member States, specifically the Netherlands (Dutch Business Register), for “standalone” tests in the context of the specification compliance and integration testing for BRIS release 2.0. Once started, the first test suite, at the time containing 5 test cases, had been successfully passed within one week.
At the time of writing the testing implementation includes 15 test suites with a total of 96 test cases offering full functional coverage and making the first phase of integration testing completely autonomous for users. The test suites address not only the latest BRIS release (version 2.0) but also the previous one (version 1.4) offering thus full regression test support. Newly connecting countries, such as Portugal and Ireland, can undergo complete integration testing per BRIS message type based on their individual scope of implementation. In terms of current testing progress, 24 Member States in total have connected their Business Register implementations and have completed more than 8600 test sessions.
Benefits in using the Test Bed
The Test Bed has benefited BRIS by being able to fully realise its conformance testing needs in a way that would be difficult and costly to otherwise achieve. It allowed reuse of an online platform that offered a rich user interface to project staff and Member State testers, allowing tests to proceed in a fully self-service manner once the initial technical onboarding was complete. The readily available interface offered by the Test Bed represented a big benefit as this would otherwise need to be a custom implementation by project staff of significant effort. Moreover, the Test Bed proved flexible both at a technical level, by integrating with BRIS infrastructure with little effort, as well as functionally by being able to design and organise tests with the required level of granularity.
It is important to also mention that the Test Bed team has offered significant support to BRIS project staff as a testing partner, both through the initial design put in place, as well as in consulting for the overall testing setup. Requests for new features have been consistently delivered along with ad-hoc support when this was needed. It is also worth noting that by reusing the shared Test Bed instance, BRIS did not need to foresee hosting or operational effort as such services were provided by the Test Bed team.
It is estimated that the BRIS project would need to double the effort and cost in terms of person-days and budget to implement a complete conformance testing solution had the Test Bed not been available.
Conclusion
The conformance testing needs for BRIS presented significant challenges that were successfully addressed by use of the Interoperability Test Bed. Reuse of the Test Bed platform allowed it to avoid significant effort and focus on its core concerns while offering a user-centric and rich conformance testing experience to project staff and the Member States.
Case study: CEF eInvoicing Building Block
The following case study explains how the Test Bed was used to successfully realise the conformance testing needs of the CEF eInvoicing Building Block. It has been co-authored by the Test Bed team, the CEF support team and Caroline Corneau (DIGIT D.3), IT Project Manager – Project Manager for CEF eInvoicing.
Business context
The European Commission sees electronic invoicing and the wider procurement cycle as a central element of Europe’s digital transformation. eInvoicing touches on areas relating to secure cross-border data transition, the automation of business processes and payments, and as part of using digital technologies to green Europe’s economy. The coexistence of many different eInvoicing standards and syntaxes has also proven a barrier to the smooth functioning of Europe’s internal market.
The Connecting Europe Facility (CEF) eInvoicing Building Block therefore promotes the successful uptake of eInvoicing in Europe respecting the European standard on electronic invoicing (EN 16931), helping authorities in Europe transpose Directive 2014/55/EU on electronic invoicing in public procurement. This Building Block supports public entities and suppliers in setting up their invoice exchange as well as solution providers in delivering eInvoicing services and software. Apart from promoting the standard, the eInvoicing Building Block offers services on conformance testing, trainings, community management to facilitate stakeholder onboarding and access to its knowledge base, which includes a Registry of technical artefacts needed in eInvoicing implementations, as well as a dedicated service desk.
Specifications and technical architecture
On October 18th 2017, the European Committee for Standardization (CEN) formally published the European standard on eInvoicing . The binding syntaxes that are compliant to the standard are:
- UN/CEFACT XML Industry Invoice D16B (Cross Industry Invoice).
- ISO/IEC 19845 (UBL 2.1) invoice and credit notes.
Validation against these syntaxes is done using XML Schema and Schematron rules published on GitHub that were produced by an expert group in association with the CEF team and CEN. In terms of message transport to support eInvoicing interoperability, the guidelines foresee invoice exchanges based on three communication models:
- Direct connection (sellers directly exchange invoices with buyers).
- 3-corner model (sellers and buyers interconnect via a common communication hub).
- 4-corner model (sellers and buyers interconnect via connected communication hubs).
Conformance testing needs and challenges
Ensuring the conformance of eInvoicing with the European standard is fundamental to the standard’s adoption. The European Commission has therefore invested in insuring that a conformance testing service is in place.
CEF eInvoicing provides conformance testing to allow parties to test their eInvoicing implementation against the European standard (core invoice), as well as also to verify solutions developed with CEF grant funding. To do this, the conformance testing setup needs to foresee:
- A public validation service allowing validation of invoices via several communication channels.
- A formal conformance testing environment with scenario-based test cases to both send and receive invoices, that provides the necessary monitoring and reporting features to validate parties’ grant claims (as well as a scenario that should generate a negative response so the user can check their error message handling).
Validation channels include a web user interface to validate invoices via upload, as a web service API for machine-to-machine validation, as well as invoice exchange using the CEF eDelivery Building Block. The eDelivery Building Block is built using direct connections or a 4-corner model, based on the AS4 protocol, which is required when testing grant-funded eInvoicing solutions. Given the multiple interfaces involved, a key implementation point is the use of a common validation core that can be easily updated when new releases are made for the published validation artefacts.
An additional challenge coming from the public nature of the conformance testing service is to limit the support effort needed by the CEF team when onboarding new parties. The process to register new users, collect their information, and complete the technical configuration to start exchanging invoices (applicable in eDelivery scenarios) needs to be as streamlined as possible.
How was the Test Bed used?
The validation of invoices against the European standard’s core invoice is the central objective of the eInvoicing conformance testing service. Given that the supported syntaxes are XML-based, this was realised by means of the Test Bed’s core XML validation service that was used to create a new validator supporting the validation of invoices and credit notes for each syntax. The resulting validator is used in two ways:
- As a standalone service via its web user interface and web service API.
- As a conformance testing building block in test cases focusing on eDelivery.
Given that validation is provided by a common validator, it remains consistent regardless of the input channel and is easy to update when new validation artefacts become available. To cover the eDelivery scenarios, test cases are defined in the CEF Test Bed platform, a GITB instance operated by CEF, organised based on the AS4 protocol, with test cases further split in invoice sender and receiver roles. For AS4 message exchange a separate Domibus instance (the Commission’s AS4 eDelivery reference implementation) is used that the Test Bed integrates by means of its AS4 messaging adapter.
A noteworthy element of the conformance testing setup is the configuration put in place to streamline the onboarding of new parties. The CEF eInvoicing Test Bed is configured to enable party self-registration and uses custom properties to collect required party information. Furthermore, upon registration a process is automatically triggered to complete the technical steps needed to make eDelivery exchanges with the new party, sharing configuration information with the party via the Test Bed’s user interface.
Testing progress and major milestones
2017 - The CEF eInvoicing Building Block launched the first version of its conformance testing service in May, based on the then draft standard, including the discussed standalone invoice validator and conformance testing test cases with two test suites (one per the then supported eDelivery protocols - AS4 and the retired AS2 protocol).
2017 - The CEF team successfully implemented a procedure to update the conformance Test Bed with revised or updated validation artefacts. Following the official publication of the European standard, getting the Test Bed up and running against the core invoice, and having the validation artefacts up to date, constituted a major milestone for the project.
2018 - INEA started to base evaluations heavily on the CEF eInvoicing conformance testing service during the validation of the milestones pertinent to eInvoicing grants.
2019 - Major development was introduced following a special requirement of a user a (a service provider who applied for an eInvoicing grant), whose eInvoicing system was implemented only for receiving eInvoices. As INEA required eInvoicing validation also in this special case, the team opted to develop an eInvoice sender module. The service now provides both sender and receiver scenarios.
2020 - The CEF team implemented automation processes to streamline user onboarding. Special modules were developed and integrated into the Test Bed, making registration fully self-service, including the automatic generation of IDs, configurations, and welcome packages.
At the time of writing, a total of 10 built-in test cases are covering invoice sending and receiving scenarios. These have been used by 113 registered accounts, belonging to approximately 50 organisations that have completed more than 3200 test sessions.
With the help of CEF eInvoicing Conformance Testing, INEA has validated 33 eInvoicing grants.
Benefits in using the Test Bed
The Test Bed allowed the CEF eInvoicing Building Block to offer a customised conformance testing service without needing to invest in new software development. Savings are roughly estimated to be at least €50,000 and/or 300 person-days. Moreover, the Test Bed’s self-registration and customisation features made it possible to offer a streamlined experience to its users while at the same time significantly reducing support effort by CEF staff. It is important to also note that the effort to set up and operate the test environment was managed by the Test Bed team, allowing CEF staff to focus fully on the testing process itself.
The conformance testing service has continuously evolved since its initial launch in 2017. This evolution is based on the continued feedback from CEF support staff and its close collaboration with the Test Bed team that has provided several new releases with features to further improve the resulting service. The Test Bed team has also actively participated in test suite development when needed and the implementation of the processes that help automate user registration.
Conclusion
The use of the Test Bed has allowed the CEF eInvoicing Building Block to realise a customised conformance testing service with minimal cost. It has achieved significant reductions in support effort while providing a streamlined and continuously evolving service to its users.
The European Commission has recently published a list of conformant eInvoicing Service & Solution providers whose solution passed the conformance testing and who agreed to be listed. The Commission hopes more companies will join soon, including of course all CEF grant-funded projects who passed the CEF conformance tests.