Testing the compiler and code coverage

[Mildred Ki'Lya]

On 03/06/2010 09:45 AM, Nicolas Boulay wrote:
> The test have a problem not often understood : it must be simpler than
> the code it self. If the test is as big as the code size, you will
> take time to debug also the test part.
>
> Specification have the drawback to be unprecise. Then you define a way
> to define precise specification (it's a harder work) and you find that
> you can compile this description, now you have problem with the upper
> level spec. It will always be like that.
>   

Sure, tests are not that perfect, they don't test really everything,
bugs can still be there when we have tests. I understand that. But I
think that no method is perfect, a combinaison must be used.
> Lisaac have the most advance feature : contract. We must use it. This
> is the check part of a test (low level test). Then you need "only"
> higher level test.
>   

Contracts are at the level of the code, not really the specification level.
We should use them, but the problem is that to write contracts, we must
know the code we write contracts for.

And for the moment, the compiler source is except for Benoît still very
obscure. Well, it is for me anyway. I can't write contracts on the
compiler. But I can write higher level tests.

Moreover, tests in English can become a specification, and the scenarios
in tests can becomme acceptance tests. It sets in stone the behaviour of
the compiler, not having to rely on the even more unprecuse reference
manual.

Don't be mistaken, the reference manual is great to learn the language,
but it doesn't makes a formal specification. (Firtunately, or beginners
would run away)

> Coverage is nice, but there is 3 kinds of it. What you describe is
> statement coverage. In DO178b software design, they use decision
> coverage (DC) and modified condition/decision coverage (MC/DC). DC
> need to have every if-clause, to be true and false. In statement
> coverage, an if-clause without "else" part, is covered with "true"
> only.
>   
I became awayre of the DO178B, especially since I work in a tool similar
to cucumber but for the Ada language, and AdaCore (the creators of the
GNAT compiler, Ada for GCC). And they work with companies that are bound
to the DO178B.

By the way, if you are in Paris the 11th of March, there is the Open-DO
Conference. Unfortunately I won't be there (I go to my IUT instead)
http://www.open-do.org/conference-2010/
Open-Do is a project started by AdaCore to gather open-source tools for
the DO178B.
> MC/DC is even worse. It asks that in the condition of the IF, we must
> find 2 vectors for each boolean variable to make the condition true
> and false. This reflect the effect of each boolean.
>   

Worse or better ?

Unfortunately, I gave some thought, and I don't think this is applicable
to Lisaac for there is absolutely no test constructions in Lisaac. If
are only a special case of polymorphic call. And the constraint that all
polymorphic calls must call all the types of their hierarchy is far too
much to be useful.
> Coverage is used to find dead code or unexpected result. But it deal
> only with input vector of the test. So, if there is no checker (or
> contract), it did not check anything.
>   

Of course, tests must check the result is correct
But coverage is very good to notice special cases that were not included
in the test suite. I do that every day and I find it very useful.


Mildred

-- 
Mildred Ki'Lya
╭───────── ｍｉｌｄｒｅｄ５９３＠ｏｎｌｉｎｅ．ｆｒ ──────────
│ Jabber, GoogleTalk: <mildred at jabber.fr>
│ Website: <http://ki.lya.online.fr>           GPG ID: 9A7D 2E2B
│ Fingerprint: 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 9A7D 2E2B


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/lisaac-devel/attachments/20100306/ec76d388/attachment-0001.pgp>