Skip to content

⚙️📒 CI for documentation. Testing if documentation is runable.

License

Notifications You must be signed in to change notification settings

ottomatica/docable

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

docable

CI for documentation: Testing if documentation is runable.

Suppose you wanted to test the following tutorial (inspired from benmoral.)

Example Tutorial

Let’s create a small server using PHP. I can see your eyebrows rising, but it works surprisingly well. We’ll listen to UDP port 10000, and return any message received with a ROT13 transformation:

<?php
$sock = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_bind($sock, '0.0.0.0', 10000);
for (;;) {
    socket_recvfrom($sock, $message, 1024, 0, $ip, $port);
    $reply = str_rot13($message);
    socket_sendto($sock, $reply, strlen($reply), 0, $ip, $port);
}

Let’s start it:

$ php server.php

And test it in another terminal:

client$ echo 'Hello, world!' | nc -q 1 -u 127.0.0.1 10000
Uryyb, jbeyq!

Cool, it works. Now we want this script to run at all times, be restarted in case of a failure (unexpected exit), and even survive server restarts. That’s where systemd comes into play.

...


How would we test these instructions actually work? After you updated them? On different platforms and versions of php (5 vs 7) or tools (netcat openbsd vs traditional)? After a year has passed? That's what docable helps do for you.

Testing with docable

To test this example tutorial, we can run the following:

docable report examples/unix-service/steps.yml

docable uses a stepfile, such as the following.

setup:
#  local: {}  
  slim:
    image: ubuntu16.04-php

unix-service.md: 
  steps:
    - selectAsFile: ROT13 transformation => server.php
    - selectAndServe: "start it:"
    - selectAndExpect: "another terminal:"

The stepfile using a "select-assert" pattern for testing documentation. That is, we first find an instruction step in the tutorial using a lightweight selector, then, we transform the step into an action we can perform within a "headless infrastructure" (such as a vm), and finally we assert it's behavior is correct.

The result looks something like this:

example report

Turns out that our tutorial breaks if using a different version of netcat, which does not support the -q 1 option. Oops!

After fixing the issue by changing to using "-w 1" (terminated after timeout instead of time since no STDIN input). Running docable test, we verify have fixed the tutorial:

example execution

Installing docable

$ git clone https://github.com/ottomatica/docable
$ npm install
$ npm link

Using slim provider requires installing slim. You can also switch the stepfile to use a local provider under setup:.