Skip to content
Copyright © 2022- basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

basebox Testing Overview

Purpose

This document provides an overview of the test concept and all test activities which are performed to verify basebox. Since basebox is an Off-the-shelf Software, OTS (for details refer to the Intended Use description) the Medical Device validation requirement does not apply.

Basebox testing is focused on verification activities of different types.

Note

This document provides an overview of the basebox verification activities but does not contain detailed descriptions of test cases of any test level since this information is of sensitive nature. Test cases can be audited after consultation with the company.

Terms and Definitions

Clippy

A collection of lints (code analyzer) to catch common mistakes and improve your Rust code (Source: Introduction - Clippy Documentation (rust-lang.org)).

Git

Git is a distributed version control system that tracks changes in any set of computer files, usually used for coordinating work among programmers who are collaboratively developing source code during software development. Its goals include speed, data integrity, and support for distributed, non-linear workflows (thousands of parallel branches running on different computers), (Source: Git - Wikipedia).

Gitea

Gitea is a lightweight DevOps platform. It brings teams and developers high efficiency but easy operations from planning to production. Some of the core capabilities are:
Code Hosting - Gitea enables the creation and management of repositories, exploration of commit history and code files, review and merging of code commits, collaboration management, branch handling, and more…
Code Review - Code review accommodates both Pull Request and AGit workflows. Reviewers can easily examine code online and submit comments or inquiries. In response, submitters can view the feedback and address it directly online. Code reviews facilitate enhanced code quality for users and businesses.
CI/CD - Gitea features an integrated Continuous Integration (CI) / Continuous delivery (CD) system, Gitea Actions, that is compatible with GitHub Actions.
Projects - With Gitea you can efficiently manage a project's requirements, features, and bugs through issue tasks, labeling, and kanban project boards.
(excerpted from: Gitea - Lightweight DevOps Platform).

Rust

Rust is a programming language that is growing in popularity. While its user base remains small, it is widely regarded as a cool language. According to the Stack Overflow Developer Survey 2022, Rust has been the most-loved language for seven straight years. Rust boasts a unique security model, which promises memory safety and concurrency safety, while providing the performance of C/C++. (excerpted from the Carnegie Mellon SEI blog: Rust Software Security: A Current State Assessment (cmu.edu).

Test Planning

The following aspects were considered during the planning and development of basebox tests:

  • Use a programming language which is avoiding errors (like memory corruption) rather than finding errors during testing (if at all)! That`s why basebox was programmed using “Rust” as a robust and safer language. For details refer to Rust Software Security: A Current State Assessment (cmu.edu), published by the SW Engineering Institute of the Carnegie Mellon University.
  • Develop test cases while you write new or maintain code. Automate as much as possible and integrate testing in a continuous integration approach.
  • Leverage the test capabilities of the development framework and the programming language “Rust”.
  • Use dynamic (e.g., by running unit tests) and static testing techniques (e.g., code inspection). Consider the four-eye principle not only for the programming code but also for verifying inline test cases.
  • Perform testing (verification) across the following levels:
    • Unit
    • Integration (Interfaces)
    • System
  • Comply with good testing practices and regulatory requirements from:
    • IEC 62304, sections: 5.5 Software Unit implementation and verification, 5.6 Software integration and integration testing, 5.7 Software system testing, and 5.8.1 Ensure software verification is complete (also considering regression testing and software problem resolution process for all test level)

Test Strategy and Test Environment

The Integration of the basebox Testing in a Continuous Integration and Delivery Approach (CI/CD) was a primary goal of development planning. The CI/CD approach was established by implementing the following tools:

Gitea – is a frontend tool which supports the implementation of the basebox CI/CD approach. The following features of Gitea are used to support any testing:

  • Configuration management for distributed development environments and activities including testing.
  • Enable Branching, merging, and baselining of Configuration Items, including the identification of test versions or successfully verified (main) branches.
  • Align and integrate all necessary verification steps to ensure a successful generation of a main branch (a customer version).
  • Support code reviews.

Gitea Screenshot

Gitea User Interface showing a pull request

Rust – is the programming language which was used for the development of basebox. Rust is also being run by Gitea during CI/CD actions. The Rust programming framework consists of several verification steps during binary generation. The following sequence of verification perform static code analysis with an increasing level of rigor:

  • The Compiler checks for security weaknesses in the code.
  • Clippy provides a more thorough collection of lints (code analyzer) to catch more mistakes and improve your Rust code.

Rust Clippy Output

Sample Clippy output with hints (red = errors, green = notes)

Git – provides fundamental version control and configuration management functionality and supports distributed development. Gitea integrates Git.

Testing Level and Test Types

The following table provides an overview about which tests are performed in which sequence.

Verification Activities on Unit Test Level Test Type
Static Analysis
1. Code Reviews
  • Peer Code Reviews are performed to improve the code quality (including test code) before any merge.
  • These Reviews are supported and documented by Gitea features.
Peer Review
2. Code Analyis
  • The Rust compiler performs code analysis which comprises security checks.
  • Subsequent Clippy code analysis provides an additional level of rigor for the Rust code verification (for details refer to [Clippy's Lints - Clippy Documentation (rust-lang.org)](https://doc.rust-lang.org/nightly/clippy/lints.html=)).
The acceptance criteria are - no warnings appear, or all warnings are resolved. 		Static analysis to ensure (non-comprehensive list):
  • Security
  • Correctness
  • Complexity
  • Performance
  • Style
1. Vulnerability Analysis
  • Cargo Audit performs a dependency analysis (of OTS components).
  • Found dependencies are analyzed for existing vulnerabilities (based upon a Rust internal vulnerability DB).
The acceptance criteria are – all vulnerabilities are resolved. 		Security
4. Create SBOM Cargo SBOM creates a SBOM with formalized formats like SPDX. Security
Dynamic Tests
1. Functional Tests
  • Complex functions are supplemented by test code which is testing for operational correctness and function outputs.
  • Testing is performed against the detailed designs in the code with support from AI test generators.
  • Those tests are automatically executed before any merge and enable regression testing.
  • Rust build tools ensure conditional compilation for test routines.
The acceptance criteria are - no error messages or warnings appear, or all messages are resolved. 		Functionality
Performance
Regression Testing
  • Since test cases are part of the code all tests can be automatically executed and support regression testing.
The acceptance criteria are - no error messages or warnings appear, or all messages are resolved.
Verification Activities on Integration Level Test Type
Integration Tests
  • The software units are integrated into the basebox system.
  • Test code is developed to focus on testing of the internal and external interfaces of basebox, including edge cases.
  • Those tests are automatically executed before any merge and enable regression testing.
The acceptance criteria are - no error messages or warnings appear, or all messages are resolved. 		Internal / External Interfaces
Regression
Verification Activities on System Level Test Type
Functional Tests
  • The Software Requirements Specification build the foundation Functional Tests for functional tests on System Level.
  • The integration at customer sites is simulated to test basebox in a user environment.
  • Benchmarks are performed to analyze performance.
Functional
Stress/Load Test
  • GraphQL requests are created and simulated to execute stress and load tests.
 Stress
Load
Security
  • Different Open ID Connect provider are tested for security and integration reasons.
Security
Penetration Tests
  • Penetration testing is performed by an independent 3rd party test lab.
 Security
Reviews/td>
  • Independent Reviews were performed for the basebox Technical Documentation.
 Documentation

Test Results

All test results are persisted in gitea.

History

Released Who Changes
Dec-2023 basebox Initial Release