# File: hello.t TEST touch /var/www/html/index.html
Test Framework - Binnacle
Authors
-
Aravinda Vishwanathapura <aravinda@kadalu.io>
License
Binnacle is licensed under the Apache 2.0 license
Introduction
Proposal for a Test framework with the focus for Tester’s delight!
Automating tests in a distributed setup is always hard. Binnacle aims to simplify the ergonomics so that a tester without Programming language expertise can adapt to this quickly.
If someone knows which command to test, then write that command with
the prefix TEST
. For example, the command touch
/var/www/html/index.html
tries to create a file inside the directory
to see if it succeeds. To convert this into a test case, write as
below.
That’s it! Run the test file using the binnacle command, and it gives the beautiful Output as below.
$ binnacle hello.t
hello.t .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.03 usr 0.01 sys + 0.07 cusr 0.05 csys = 0.16 CPU)
Result: PASS
The verbose(-v
) Output gives other useful details about executed
command and error lines in case of errors.
Learning the new programming language is not required to write test cases.
Note: This framework is not for Unit testing
Features
Multi-node support
Just specify NODE=<nodename>
before any test to run in that
node. For example,
NODE=node1.example.com TEST command1 TEST command2 NODE=node2.example.com TEST command3
Note: Passwordless SSH access is required from the current node for all the nodes which are specified in Tests.
Bash style variable substitution
VOLNAME=gv1 TEST gluster volume start ${VOLNAME}
Validate Return code
TEST
helper validates the return code of the command, for example,
TEST stat /etc/glusterfs/glusterd.vol
Above command is equivalent to the following bash script.
stat /etc/glusterfs/glusterd.vol if [ $? -eq 0 ] then echo "SUCCESS" else echo "FAIL" fi
To test other return code use --ret=<return-code>
. For example,
TEST --ret=1 stat /non/existing/file
To test the return code is not some value. For example,
TEST --not=0 stat /non/existing/file
Validate the return value
To verify the Output of commands, EXPECT
can be used.
EXPECT -v 10 wc -l /test/output
To check if any expected value from the Output
EXISTS "127.0.0.1 node1.example.com" cat /etc/hosts
Timeout for tests
All the utilities discussed above have a variant with _WITHIN
suffix. For example,
TEST_WITHIN -t 60 stat /path/to/pid/file EXPECT_WITHIN -t 10 wc -l /test/output
Pipelines
Pass Output of one command to other function using |
. For example,
TEST cat /etc/hosts | grep "node1.example.com"
Extending functionalities by writing Python function or Bash function
The name of command after TEST
or any other utilities can be a
Python function. For example, below command calls Python/bash function
brick_kill
.
NODE=node2.example.com TEST brick_kill "/exports/bricks/${volname}/brick2/brick"
Note: Make sure to use the function name different than the actual
command. For example, if a function is created as gluster
then that
function will get called instead of actual Gluster command.
Recording Output of a command
Sometimes we can’t run the same command multiple times, but Output needs to be verified multiple times. For example,
TEST gluster volume info --record-output /tmp/status.dat EXPECT -v 2 grep "ID:" /tmp/info.dat | wc -l EXPECT -v 1 grep "Status:" /tmp/info.dat | grep "Running" | wc -l
Loop
Repeating tests are straightforward to implement, define LOOP_DATA
as JSON, and then use indentation to define loop tasks.
LOOP_DATA=""" [ { "node": "node1.example.com", "device": "/dev/vdc" }, { "node": "node2.example.com", "device": "/dev/vdc" }, { "node": "node3.example.com", "device": "/dev/vdc" } ] """ LOOP: NODE=$node TEST mkfs.xfs $device
In case of external JSON data,
LOOP_DATA_FILE=device_config.json LOOP: NODE=$node TEST mkfs.xfs $device
Ease of use
No special syntax. The example below is to test the Gluster Volume force start command.
N1=node1.example.com N2=node2.example.com N3=node3.example.com volname=gv1 NODE=$N1 TEST gluster volume create ${volname} \ $N1:/exports/bricks/${volname}/brick1/brick \ $N2:/exports/bricks/${volname}/brick2/brick TEST gluster volume start ${volname} TEST_WITHIN -t 60 gluster volume status ${volname} | match_num_bricks_online 2 NODE=$N2 TEST brick_kill "/exports/bricks/${volname}/brick2/brick" NODE=$N1 TEST_WITHIN -t 60 gluster volume status ${volname} | match_num_bricks_online 1 TEST gluster volume start ${volname} force TEST_WITHIN -t 60 gluster volume status ${volname} | match_num_bricks_online 2
Kadalu Test example
MASTER=master.example.com N1=kube-node1.example.com N2=kube-node2.example.com N3=kube-node3.example.com STORAGE_POOL_NAME=sp1 OPERATOR_YAML=operator.yaml PVC_NAME=pv1 PVC_FILE_NAME=pv1.yaml APP_POD_NAME=pod1 APP_POD_FILE=pod1.yaml NODE=$MASTER TEST kube_master_setup --record-output tmp/master-details.json LOOP_DATA=""" [ { "node": "$N1", "device": "/dev/vdc" }, { "node": "$N2", "device": "/dev/vdc" }, { "node": "$N3", "device": "/dev/vdc" } ] """ LOOP: NODE=$node TEST kube_node_setup TEST join_kube_master tmp/master-details.json TEST loop_device_setup $device NODE=$MASTER TEST kubectl create -f $OPERATOR_YAML EXPECT_WITHIN -t 180 -v 3 kubectl get pods -n kadalu | grep "Running" | wc -l TEST kubectl kadalu storage-add $STORAGE_POOL_NAME --type Replica3 \ --device $N1:/dev/vdc --device $N2:/dev/vdc --device $N3:/dev/vdc EXPECT_WITHIN -t 180 -v 3 kubectl get pods -n kadalu | grep $STORAGE_POOL_NAME | wc -l # Sample PVC Test TEST kubectl create -f $PVC_FILE EXPECT_WITHIN -t 60 -v 1 kubectl get pvc | grep $PVC_NAME | grep Bound | wc -l # Sample app Test TEST kubectl create -f $APP_POD_FILE EXPECT_WITHIN -t 180 -v 1 kubectl get pods | grep $APP_POD_NAME | grep Running | wc -l
How it works?
Similar to Gluster upstream tests, Binnacle uses Test Anything Protocol(TAP). This test framework provides syntax sugar on top of the shell script.
Binnacle generates shell script from the given test file and then executes the generated script using prove command.
Core functions
Core functions like TEST
, EXPECT
, EXISTS
etc. will be written
using Python and bash. These functions print the Output in TAP format
so that prove tool can understand it.
Binnacle test to Shell script conversion
A few special syntaxes used are not valid bash script. Those need to
be converted. For example, Pipe character needs to be escaped
otherwise, TEST
only runs the first part, and all other commands
will be run based on the TEST output instead of the first command’s
Output.
Similarly, the utility file needs to be included in every test file to
understand TEST
, EXPECT
, and other functions.
Loop syntax used in the test file needs to be expanded into a valid bash script.
How is this different from Gluster upstream Tests
We are inspired by the Gluster upstream regression tests, which use shell scripts for writing test cases. Binnacle provides an additional layer on top of shell scripts for ease of use. In addition to ease of use, Binnacle also offers the following features compared to the Gluster tests framework.
-
Multi-node support
-
Plugin support - Extend/add the functionalities easily using Python/Bash
-
Detailed easy to parse Output in verbose mode
-
Ease of use
Code repository
https://github.com/kadalu/binnacle will be
the project repository for the Binnacle project. While running, it
looks for plugins in ~/.local/lib/binnacle/plugins
directory.
Binnacle repository will not contain Test cases and plugins specific to different projects.
Known issues
-
The test framework will not work on the Windows operating system. Windows support can be adopted in the future but not a priority for now.
Implementation
Not yet started.