Improve Software Testing with Containers
Yuan Sun, Huawei, www.enjoytesting.cn, @yuansun82
Yan Wei, Huawei, echo3301 [at] gmail
With the advent of the era of the container technology, Docker and Rocket are the representatives of the excellent container engines available from the open source community. This article presents how container technology improves different software testing tasks (application test, middle layer test, kernel test, hardware driver test, compiling test) and its benefits. We will use test practices to demonstrate the opportunities that containers bring to software testing.
1. Sandbox of test engineering capacity containerization
The era of the container has brought great opportunities for the cloud industry. Cloud container platforms are popping up across the industry. Most of the cloud container platforms provides features like code management, image storage, compiling, testing, packaging, operation etc. In general, there are the same kernels and similar hardware computing resources in the containers of cloud container platform. Many software testing tasks can be improved or accelerated by cloud containers. However, some kinds of test tasks are not applicable to cloud containers. It should be noted that containers and cloud container platforms are different. Even if a test task can’t be improved by cloud containers, it is possible to improve the task by normal container.
The following sandbox covers server os, virtualization, real-time os, Web terminal, Linux kernel, Web service, security testing businesses and has some layers for each business. These layers include test tools, source code compiling, application layer, middle layer, kernel, hardware. Green means the business can be improved by cloud containers; Yellow means whether the business is suitable for cloud containers needs to be decided by test scenarios; Red indicates that the business does not apply to cloud containers. We will discuss the containerization improvement scheme of each business in the next section.
2. Containerization improvement schemes for different test types
The picture below shows containerization improvement schemes for some classic test tasks. Some schemes can be used in different test tasks. You can create your scheme according to your real test scenarios.
(click on figure to enlarge)
2.1. Test tool containerization
A cloud container service is a lightweight container virtualization platform. Some tools (continuous integration, big data tools etc) are included in the platform, which support users’ tool deployment. Improving test tools by container has two benefits. Test tools are packaged in a clean container and don’t contaminate file system. System administrators can manage Docker containers in Windows, Mac, Linux. When a Docker server is deployed in Linux, Docker clients can be deployed in Windows, Mac and Linux, which means that we can deploy and manage containers quickly. Please note that a Docker container in Linux can’t use Docker images of Windows unless a virtual machine (VM) is used.
Today, many popular testing tools have official Docker images, such as Kali in penetration testing domain and Jenkins, a continuous integration (CI) tool. Some test tools are still not containerized. You can try to containerize these tools by yourselves. With the popularity of software as a service (Saas) and container as a service (Caas), there are more test tools in cloud container platforms.
2.2. Code compiling test containerization
Code compiling test is a task running in user mode and is a CPU intensive. The test execution time required is inversely proportional to the system resources. The expense of cloud container service is based on the time and resource usage. We can use rapid deployment and rapid expansion features of Docker to get large system resources of cloud container service in a short time to accelerate test execution. After the test is completed, the resources can be released quickly to reduce costs. At the same time, Dockerfile can be used to manage compiling dependent packages (such as gcc) and compiling software installation, which clarifies the steps of environment preparation.
2.3. Test task containerization of application layer
Applications belong to the top layer of the operating system. Their tests are very suitable for containerization improvement in many different ways.
(1) Use Docker and Devops to create a consistent test environment for automated test tasks among developers, testers, and operators.
(2) To accelerate test execution by using scalable feature of Docker, or to simulate a number of end users for testing.
Selenium is an automated testing tool for Web applications, which supports a variety of browsers. It can simulate the operation of the user in the browser.
Selenium Grid is a distributed Web test tool. It can distribute test tasks to multiple host and execute test tasks in parallel. There are two major roles in Selenium Grid, master and slave. Hub is a master and nodes are slaves. Nodes are registered to Hub and execute automated test cases from Hub.
In traditional deployment, only one browser version can be installed and only one instance can be running in the operating system. In order to test different Chrome versions, you need to install different Chrome versions in different physical machines or virtual machines. This method of deployment costs more resources and takes a long time in deploying.
Through rapid expansion, environment isolation and other characteristics of cloud container service, we can quickly start a number of containers and execute test case for different browser versions in parallel. It effectively reduces material consumption and saves testing time.
(3) Make use of isolation and rapid environment cleanup features to execute software compatibility tests (such as installing different database versions) in parallel.
(4) To mock external environment needed by test execution. Containerize some classic test scenarios and store the images in a registry. These scenarios can be deployed in micro service mode by docker-compose, which is convenient for other teams to reuse these scenarios. Test strategy of micro service is beyond of the scope of this article. You can read Testing Strategies in a Microservice Architecture for details.
(5) If you want to run applications which are dependent on kernel in cloud container platform, you need to decouple the applications from kernel features and transplant the corresponding code of kernel to the applications.
2.4. Test task containerization of middle layer
Linux packages (rpm or deb packages) is in the middle layer of the operating system. Most of linux packages are not dependent on kernel and are suitable for the cloud container acceleration. Because kernel can’t be customized in cloud containers, linux packages which are strongly dependent on kernel are not suitable for the cloud container acceleration. For this test type, you can setup a private cloud with customized kernel to accelerate test execution.
In addition, there are two exceptions.
(1) Grub package: Grub is a boot loader package for multi operating systems. It allows users to select which OS boots in multiple operating systems. Because of the strong correlation with the host, it is not suitable for container acceleration.
(2) NTP package: NTP is the network time protocol. It is used to synchronize time in multiple hosts. Because the current kernel does not support time namespace, which means that multiple containers can’t have different time. Moreover, a container and a host can’t use different time. Note that it is the time, not the time zone. To change the time of the container, you have to modify the time of the host. So we can’t create test environment with different time. If you use containers in two hosts to work around the issue, it can’t benefit from rapid deployment of container.
2.5. Containerization of kernel test tasks
The linux test project (ltp) test suite contains the most comprehensive test cases for linux kernel. By default, it is executed in serial and can be improved by container in the following aspects.
(1) Run kernel function tests in parallel in multiple containers to accelerate test execution. However, there are some limitations in the kernel testing:
A. Some of test cases are failed in a container. For example, the pidns32 test case is used to test the maximum nested depth of namespace. Because there is within a default namespace in a container, the result is failed when the case still try checking the maximum nested depth. We need to adjust it for containerization improvement. The nested checking level of container should be one less than that of host.
B. Some of the kernel test cases are exclusive and need to run separately. For example, it is not reasonable to run kernel interrupt and register tests in a container.
C. It is not reasonable to execute kernel performance test because it would lead to data distortion.
(2) We can create kernel test environment in a container. Test cases of Ltp test suite are dynamically compiled and the test execution needs dynamic link libraries. However, some systems (such as embedded systems) often lack the dynamic link libraries, resulting in execution failure of some cases. The necessary dynamic link libraries can be added in a Docker image. This not only ensures the test cases can be executed smoothly, but also avoids environment contamination made by installing dynamic link libraries in a host.
(3) Use Docker to create kernel integration testing scenarios. Because kernel is an underlying technology, it is difficult to find a test scenario including multiple kernel features. The following Docker command can easily create integrated test scenarios of namespace, cgroup, capability, seccomp and other kernel features.
$ docker run --memory 20m --cap-add sys_admin --security-opt seccomp=unconfined --userns-remap default ubuntu:14.04 bash
2.6. Containerization of hardware driver test
Docker is designed to shield from differences among multiple platforms. It is difficult to improve hardware driver test. Fortunately, Docker can use the device cgroup feature of the kernel to access hardware devices. The devices of the host can be mapped to the container and the test can be executed in the container.
Benefit: Test environment doesn’t pollute the host, which is convenient to run device driver tests in parallel. Even if a host lacks library files needed by test, the test tools of the container still can be used to run tests.
Example: serial port test
$ docker run -device /dev/console:/dev/console ${your_image}
${your_test_script}
2.7. Containerization of stress and stability tests
Test execution without stress test is not enough. Some software works well in an environment without stress. However, some stability and fault tolerance problems occur when they are running under stress circumstances. Therefore for some businesses, endurance and stability tests are executed during integration and system test periods. The tests of server os, virtualization and kernel used to have the following issues.
(1) Test deployment is complicated.
The stress test suite is strongly dependent on platforms. When a test task involves complex networking in multi platforms, there will be a lot of compatibility problems. The installation of system information collecting tools is dependent on some packages and needs to adjust to different Linux distributions manually. The deployment of the set of endurance and stability test framework (including stress test, system checking, test execution) will cost a lot of manpower, affecting the test efficiency.
(2) A stress test model needs to run separately. It is not easy to overlap two stress test models.
(3) The fault tolerance ability is poor.
When the stress test program encounters abnormal exit caused by misoperation or resource competition, there is no any fault toleration ability.
We can use some containerization test practices to solve these issues. In the stress test, cpu, memory and io test suites are packaged into a docker image. We can start the stress test in the following command.
$ docker run -tid -m 3g --restart=always back_stress bash -c "bash cpu_stress/test_20.sh
First, it can solve the problem of such long time of compiling and installation of test tools. The options “-m 3g --restart=always” limit the memory usage to 3GB and avoid the crash of the stress test. If there is memory leak in the stress test and the test process try using memory which is more than 3GB, the system would trigger OOM error and the resource of the container would be cleaned up. Then the container will be restarted immediately and the stress test still be running in the container. The stress degree of the script test_20.sh is 20%. We can overlap multiple stress modules by adding containers. Some classic stress modules are shown above.
Secondly, we implement the containerization of system checking. The server end and the client of the zabbix tool are packaged into two images separately. For complex network environment for multiple platforms, it can shield from difference among platforms.
3. Summary
This article lists containerization improvement schemes for different test scenarios. You can create your own effective strategies according to your real test scenarios and don’t have to stick too precisely to the schemes of this article.
References
Docker userguide: https://docs.docker.com/engine/userguide/
Using Docker to manage and replicate test environments: https://testcollab.com/blog/using-docker-to-manage-and-replicate-test-environments/
The Linux Kernel's documentation: https://www.kernel.org/doc/html/latest/
Resource management in Docker: https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/