7. Autotesting Transcrypt code
******************************


7.1. Why it's needed
====================

A powerful autotest facility was added to Transcrypt right from the
start for the following reasons:

1. Any programming language compiler has to be reliable, since a large
   investment in code may come to depend upon it. Languages and
   libraries should be able to evolve without regression bugs being
   introduced. In a rich language many constructs are possible which
   all should be tested with each new release. This can only be done
   if testing is automated.

2. Since Transcrypt compiles not all of Python but a fairly extensive
   subset, it has to be rigorously clear what can be compiled and what
   not. The sourcecode of a set of automated tests can be an effective
   means of laying down what is possible in the language. Whereas code
   examples and documents may lag back or deviate from reality, test
   code has to cover the essential features of the language and is, by
   nature, constantly exercised to match the latest status of the
   language.


7.2. How it works
=================

When code is being tested, a reference is needed of what is considered
to be correct. With Transcrypt that reference is CPython. Autotesting
Transcrypt code is simple and boils down to the following.

1. Along with developing production code, a growing set of *testlets*
   is developed. A testlet is a small module testing a certain feature
   or group of features. It repeatedly calls method
   *org.transcrypt.autotester.AutoTester.check (self, *args)* to build
   a well defined sequence of output data.

2. A series of testlets are imported into an application called an
   *autotest*.

3. The autotest is first run from the command line: *python transcrypt
   -r autotest*. This will generate file *autotest.html* in the
   working directory, containing the *reference data sequence*
   produced by CPython, in an HTML DIV.

4. After that, the autotest is compiled to JavaScript: *python
   transcrypt -b autotest*. This will generate file *autotest.js* in
   the corresponding target directory. Note that you may need extra
   command line switches to activate options needed for your testcode,
   e.g. -c if you use complex numbers or -da if you use assertions.

5. Click on *autotest.html* to load the autotest into the browser and
   run *autotest.js*. This will generate the *test data sequence*, now
   using the Transcrypt runtime.

6. After this, the test data sequence is automatically compared to the
   reference data sequence that was part of the html, and an error
   report is shown in the browser.

An example of two testlets combined into the 'hello' autotest, that is
part of the distribution:

+------------------------------------------------------------------------------+------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| a u t o t e s t . p y import org.transcrypt.autotester  import testlet0      | t e s t l e t 0 . p y def run (autoTester):     autoTester.check ('hello')   | t e s t l e t 1 . p y def run (autoTester):     autoTester.check ('goodbye') |
| import testlet1  autoTester = org.transcrypt.autotester.AutoTester ()        | autoTester.check ('world')                                                   | autoTester.check ('moon')                                                    |
| autoTester.run (testlet0, 'testlet0') autoTester.run (testlet1, 'testlet1')  |                                                                              |                                                                              |
| autoTester.done ()                                                           |                                                                              |                                                                              |
+------------------------------------------------------------------------------+------------------------------------------------------------------------------+------------------------------------------------------------------------------+

Steps to run the tests:

   transcrypt -r autotest
   transcrypt -b autotest

At this point, if you start a webserver and load *autotest.html* in
your browser from localhost, you will see that all tests passed. In
order to trigger an error, open *testlet1.py*, search for *goodbye*
and replace it with *badbye*. After that, recompile with:

   transcrypt -b autotest

Once again open *autotest.html* in your browser, it will show:

   [image: Output of 'hello autotest'][image]

Some testlets may require additional switches. The main *transcrypt*
autotest e.g. requires and extra *-c -da*, since it uses both complex
numbers and debugging assertions.