Urchin: Unix-style tests

Urchin is a portable shell program that runs directories of Unix programs and produces pretty output. It was originally developed as test-runner, based on the idea that a test case should be an ordinary Unix-style program. It's called "Urchin" because sea urchin shells are called "tests".

I'll discuss how one uses Urchin, and I'll show examples of tests written in Urchin. I'll also compare it to other approaches to testing shell programs.

Bio

I originally prepared this article for a talk at NYC*BUG, so here's a picture of me standing in front of a BSD sign.

BSD

Thomas Levine is a neodada artist with an interest in sleep. He enjoys writing intuitive and minimal user interfaces, like Urchin, that are thus easy to learn and easy to reverse-engineer once the source code has been lost and only the data are left.

Tom's principles for software development

Here are some principles that I follow in most of my software.

  1. Follow standard conventions.
  2. User data should be legible on their own;
  3. I should be able reverse-engineer the software from user data.
  4. Installation should be easy.

Follow standard conventions (POSIX, in the case of Urchin). User data (test suites in this case) should be legible on their own; if I lose the software (Urchin) and am left only with the user data, I should be able to reverse-engineer the software. Also, Installation should be easy.

It turns out that I take these more seriously than most people do. My attention to these principles can explain why Urchin is so different from the other test harnesses that will mention.

Intuition and standards

I think that intuition and standardization are the same thing; our intuitions are shaped by our environments, and standards get adopted when they match our intuitions. To me, designing intuitive products is often about figuring out the implicit standards that we follow.

I originally developed Urchin because I wanted to test system configuration programs written in shell. I styled my tests to match ordinary shell/Unix conventions; because of this, Urchin interacts well with other Unix-style programs, and people who are used to Unix-style programs will quickly grasp how Urchin works.

Overview

  1. Demonstration of Urchin
  2. Discussion of alternatives and when to use them
  3. Further commentary on the merits of following standards
  4. Infrastructure for testing Urchin portability
  5. Request for projects that could use Urchin tests
  6. Appendix: Fun things I learned about sh and other shells

Demonstration

Here are the things I'll demonstrate. I should save the terminal output somehow and put it here afterwards.

First we'll do a simple test suite.

  1. Write a small shell program.
  2. Write some tests for that program.
  3. Run Urchin on the tests.
  4. Demonstrate the -v flag.
  5. Demonstrate the -F flag.
  6. Write a test with temporary files, and explain that they get cleaned up.
  7. Run on just two shells to demonstrate the -s flag.
  8. Run with the -r flag. There will be a problem with the PATH.
  9. Set the -p flag.

Then I'll show Urchin's test suite run on a couple remotes. Maybe nsa, macosx, and urchin@localhost?

Example output

Tests are files

In Urchin, tests are represented as files, one file per test. A test file is an ordinary Unix-style program; in particular,

  • It is marked as executable.
  • It may print to STDOUT and STDERR.
  • Exit code 0 indicates success.

Files that begin with dots are hidden, as with other programs, so urchin ignores these; they are thus good places to put fixtures.

Files that are not hidden and not executable are considered skipped tests.

Running Urchin

Urchin tests are ordinary shell programs that set ordinary exit codes to indicate their outcomes, so you can understand them without understanding Urchin. The only strange thing is that Urchin defines one environment variable, TEST_SHELL, (It is part of the cross-shell testing feature.) but we explain in the documentation how you can write your test cases such that they work when this variable is not defined.

Tests are organized in directories, going as many levels deep as you want.

tests/
  setup
  setup_dir
  bar/
    setup
    test_that_something_works
    teardown
  baz/
    jack-in-the-box/
      setup
      test_that_something_works
      teardown
    cat-in-the-box/
      .fixtures/
        thingy.pdf
      test_thingy
  teardown

When you run Urchin on a directory, Urchin runs all executable non-hidden files in the directory and prints output based on the code with which each file exited. If it sees any of the files setup, setup_dir, teardown, teardown_dir, it executes that file just before or after processing the directory or each particular file, as appropriate. If it sees any directories, it runs recursively on that directory. Once you have create a directory like this, you can run your tests with

urchin tests

The idea behind Urchin is that tests should be ordinary shell programs that are run in a simple manner that is thus easy to understand.

Cross-shell testing

Urchin can run your tests in several shells, allowing you to test your software across several shells.

$ ./urchin -vv tests/Molly\ guard/The\ -f\ flag\ should\ disable\ the\ Molly-guard. 
Running tests at 2016-04-04T00:42:42

Molly guard/
> The -f flag should disable the Molly-guard.
. bash (0 seconds)
. dash (0 seconds)
. mksh (0 seconds)
. sh (0 seconds)
. zsh (0 seconds)

Done, took 1 second.
5 tests passed.
0 tests skipped.
0 tests failed.

Urchin uses a hard-coded list of shells by default, and you can override that list with the -s flag. Urchin passes each of these shells to each test file. The exact behavior depends on the shebang line of the test file.

With each particular combination of a test file and a shell, the environment variable TEST_SHELL is set to the specified shell. We use this in the test files for cross-shell testing of code that we invoke in subprocesses.

If the test file lacks a shebang line or has a shebang line of #!/bin/sh, the test file is also executed with each particular TEST_SHELL. This is how we do cross-shell testing of code that the tests need to . ("source").

Alternatives

It turns out that Urchin is just one of many approaches for testing shell programs. I categorize approaches thusly.

  • No automated tests
  • Simple comparison of a single file's output
  • Tests are sh functions
  • Separate files for different parts of input and output
  • Ordinary shell (JSON.sh, bocker, Urchin)
  • Ordinary shell tests with special sectioning functions

On the adherence to standards

I have categorized shell testing tools based on how they expect tests to be organized. Aside from this, they may seem to do very similar things, and that is because they otherwise are quite similar.

Let's categorized the testing programs differently: Which tools are the most standard?

The following expect their tests to be written in ordinary shell.

  • Roundup
  • shunit
  • ts
  • rnt
  • shpec
  • Urchin
  • sharness
  • testlib.sh
  • JSON.sh

While Roundup, shunit, and ts expect valid shell files, they also use their own parsers to list the test functions within a file; that leaves us with the following that rely on the shell interpreter to parse the test files.

  • shpec
  • rnt
  • Urchin
  • sharness
  • testlib.sh
  • JSON.sh

While they each use a standard shell interpreter, rnt and shpec both add several new conventions. rnt expects the shell components of its test cases to be accompanied with other files to indicate the expected exit code, stdout, and stderr. shspec defines the functions describe, it, assert, and end, among others.

rnt and shpec each create several new conventions that turn them into their own frameworks/languages; I think of their test cases as being written in their own respective languages, not in ordinary shell.

Urchin, JSON.sh and sharness are both very small wrappers around shell. The main difference I see between them is that Urchin and JSON.sh expect tests to be specified as files and sharness expects tests to be specified as function calls.

Things I purposely did not implement

Let's look at features of other systems that I didn't put in Urchin.

Assertion functions

Many other testing softwares implement their own assertion functions. Urchin does not; you can use test. If you are comparing complicated data values, you can save them to files and run diff to get nice output. So assertion functions are one thing that I don't need to maintain.

Interestingly enough, I think that JSON.sh is the only other testing software that I have listed that does not define its own assertion functions. Does the convenience of the test function have something to do with the specification of tests as files?

Test descriptions

Urchin prints the test file name as description of the test. Other systems have their own ways of encoding a test's description, usually by passing it as an argument to a shell function.

Implementing your own way of specifying a description is easy, but it makes your documentation longer, so it still helps that I use filenames for descriptions.

Looking for tests

Systems that put multiple tests in one file need a special way of listing the tests in one file. Urchin does not; it uses "*", like in echo *.

Software is easier to write when it follows standards.

As I discussed earlier, I follow these principles when I write software.

  1. Follow standard conventions (POSIX in this case).
  2. User data (test suites in this case) should be legible on their own;
  3. If I lose the software (Urchin) and am left only with the user data, I should be able to reverse-engineer the software.
  4. Installation should be easy.

Aside from making software easy to use, following these principles also makes software easier to write.

Urchin has some features that the other tools I have referenced do not, and I think that Urchin's adherence to standard POSIX features made it easier to implement the features.

  • Cross-shell testing
  • Cross-OS testing
  • Running tests in parallel
  • Timeouts

Because Urchin follows standards, I the author (and the other authors) can more easily understand what is going on when I work on it. Debugging these relatively fancy features is a lot easier for me when I know that I am running ordinary shell programs like I am used to.

Furthermore, because Urchin avoids providing its own version of features that are already present in shell, I simply have much less code to maintain, and I can focus on adding more advanced features.

Review

Urchin is shell program that runs lots of other shell programs and prints their output nicely. I originally designed it for running tests, and this is how it is usually used. Here are some of its noteworthy features.

  • Urchin is written in sh.
  • Urchin runs your tests in multiple shells.
  • Urchin is a single file, so it is easy to install.
  • Urchin tests are ordinary shell programs.
  • Urchin runs tests in parallel.
  • Output is available either both Test Anything Protocol and in a custom format designed for the display of cross-shell test results.
  • You can specify time limits for your tests.

These features follow naturally from my concern for following standards.

Standards are intuitions, and vice-versa; follow standards when you can.

Infrastructure for testing Urchin portability

I presently have several slow systems on which I test Urchin.

freebsd, solaris, openbsd, netbsd, debian, hpux, redhat, qnx, irix, tru64, openindiana, solaris-x86, ubuntu, scosysv, unixware, dragonfly, centos, miros, hpux-ia64, raspbian, pidora, debian-ppc, vax, ultrix, suse, alpha, mandriva, macosx, aix

These are configured however Zoltan happened to configure them, so I have few shells on each.

  • Mostly OS-specific utilities or GNU coreutils
  • Few different shells, no busybox

I could always use more.

I request more shell accounts for testing Urchin.

Request for projects that could use Urchin tests

Do you have anything that might benefit from Urchin? I can write the tests for you or help you write them.

End

Questions and trivia

Here we can take questions. If people want, they can ask for fun things I learned when writing Urchin.

CDPATH

According to a vague bug report, in some systems, the presence of a CDPATH variable causes cd to produce output. I don't think I have ever managed to replicate this problem, but other shell testing tools handle this specific issue too, so I think it's a real thing.

anti_read

As far as I could tell, there is no standard opposite to read, so I mostly implemented one. Look in the source code for an example.

sort and * order

sort and * put things in an order based on the enviroment's locale.

$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
!c
@a
~b
$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
@a
~b
!c
$ printf '!c\n@a\n~b\n' | sort -d
@a
~b
!c

File (<) versus pipe (|)

Pipe starts a sub-shell and thus makes the variables local.

# This does what I want
while read the_test_shell; do
  echo ...
done<<EOF
"${shell_list}"
EOF

# This doesn't.
echo "${shell_list}" |
while read the_test_shell; do
  echo ...
done

Nagios

I have heard of a software called Nagios that expects a test suite of a format very similar to the one that Urchin expects. Has anyone ever used Nagios? Could Urchin substitute for Nagios?

exec

# start a subprocess
./program.sh # execve() ?

# replace the present shell with program.sh;
# don't start a subprocess
exec ./program.sh

Is there an explicit command for the version that starts a subprocess? I think it's execve in C.

Parametrized tests

I thought a neat way to write a bunch of very similar tests would be to hard-link a file with a bunch of different names

echo "$(basename "${0}")" | my-program | grep blah

Unfortunately, I could find no version control software other than tar that recognizes hard links.