Mistakes we make: Test that never fails

We all make mistakes - this is the way we learn. When it comes to automated testing, one of the most error-prone areas, where the learning curve turns to a sharp rise, is identifying the scope of the tests. First steps in automated testing often require some rewiring of the brain.

In the very first post of the testing series (see the related posts section for the reference), I touched the topic of the testing objectives, which is very much related to the problem of defining the scope. In that post I attempted to outline the primary goal of regression testing and reflect on the things a test should do to achieve this goal. Now I would like to look at the same problem from a different perspective and talk about things that a test should not do.

To illustrate some common beginners' mistakes, I will use a made-up example of a Business Central application with a single table and a table extension. The sample application arranges customers in billing groups, assigning a billing date formula to each group. A table "Customer Billing Group" stores the billing groups, and a table extension "Customer Billing" on the Customer table allows to link customers with groups.

table 69100 "TD Customer Billing Group"
{
    DataClassification = CustomerContent;

    fields
    {
        field(1; Code; Code[20])
        {
            Caption = 'Code';
        }
        field(2; Piority; Integer)
        {
            Caption = 'Priority';
        }
        field(3; "Billing Period"; DateFormula)
        {
            Caption = 'Billing Period';
        }
    }

    keys
    {
        key(PK; Code)
        {
            Clustered = true;
        }
    }
}

Once a billing group code is selected on a customer card, a validation trigger on the table extension copies the date formula and the invoicing priority from the group settings.

tableextension 69100 "TD Customer Billing" extends Customer
{
    fields
    {
        field(50000; "TD Billing Period Date Calc."; DateFormula)
        {
            Caption = 'Billing Period Date Calc.';
            DataClassification = CustomerContent;
        }

        field(50001; "TD Billing Group Code"; Code[20])
        {
            Caption = 'Customer Billing Group';
            DataClassification = CustomerContent;
            TableRelation = "TD Customer Billing Group";

            trigger OnValidate()
            begin
                if Rec."TD Billing Group Code" = '' then
                    ResetBillingParams()
                else
                    AssignBillingParams("TD Billing Group Code");
            end;
        }
    }

    local procedure ResetBillingParams()
    var
        EmptyDateFormula: DateFormula;
    begin
        Evaluate(EmptyDateFormula, '');
        Rec.Validate(Priority, 0);
        Rec.Validate("TD Billing Period Date Calc.", EmptyDateFormula);
    end;

    local procedure AssignBillingParams(BillingGroupCode: Code[20])
    var
        CustomerBillingGroup: Record "TD Customer Billing Group";
    begin
        CustomerBillingGroup.Get(BillingGroupCode);
        Rec.Validate(Priority, CustomerBillingGroup.Priority);
        Rec.Validate(
            "TD Billing Period Date Calc.",
            CustomerBillingGroup."Billing Period");
    end;
}

This is a very simplistic example to demonstrate basic testing techniques and biases. The combination of field Priority, which is declared in the base application, and the custom "Billing Period Date Calc." could be used to prioritize customers for billing purposes and automatically calculate the invoicing date. But in this simple demo, we won't go beyond the field validation trigger.

And now we are approaching the key question of this post: how do we test this code and ensure its quality? Since the aim I am pursuing now is to illustrate erroneous testing techniques, the following example is the opposite to proper test, it shows the test that should not be.

[Test]
procedure AssignBillingPeriodCalcFormula()
var
    Customer: Record Customer;
    PeriodDateFormula: DateFormula;
    DateFormulaTxt: 
        Label '<%1D>',
        Comment = '%1 = No. of days', Locked = true;
begin
    Evaluate(
        PeriodDateFormula,
        StrSubstNo(DateFormulaTxt, LibraryRandom.RandInt(10)));
    LibrarySales.CreateCustomer(Customer);

    Customer.Validate("TD Billing Period Date Calc.", PeriodDateFormula);

    Assert.AreEqual(
        PeriodDateFormula, Customer."TD Billing Period Date Calc.",
        StrSubstNo(
            UnexpectedFieldValueErr,
            Customer.FieldCaption(
                "TD Billing Period Date Calc."), 
                Customer.TableCaption()));
end;

As I stated before, this test is incorrect. The sample test case assigns a random value to the field "TD Billing Period Date Calc." on the Customer table and verifies that the value was indeed assigned, asserting the value on the same field. It does follow the general test structure with Setup/Execute/Verify steps, but what is being verified here? In one of my previous posts, I defined basic principles of testing, one of which is: a test has a value as long as it can fail. And looking at the code above, we should assess not only the formal correctness of the test case, but also its value for the product being developed. And the key question we should ask to make this assessment is "can this test fail"? Or to rephrase it, "Can we make changes in the application which will cause the test failure?"

The only way to make the test fail is to change the type of the field "TD Billing Period Date Calc." to make it incompatible with the date formula, thus breaking the assignment of the value. Although, seemingly the test is adding value protecting the field type from unwanted changes, in fact this value is more than doubtful because the assertion statement can never be evaluated to false. If the data type changes, execution will be interrupted before reaching the verification step, and there is no way to trigger a false assertion within the application code. This means that the test is attempting to verify functionality outside of the client application, delving into the BC platform. Theoretically, the assertion can fail in case of disastrous changes in the platform which scramble the field value, and the assignment statement sets the field to 'B' when you try to assign 'A'. But we trust that the server code has good test coverage, and things like the assignment of a field value is well tested, don't we?

As for possible protection against the data type change, it can make sense in some cases, for instance to verify that a text field has sufficient length to store the necessary data, but static code analysis is much better for this kind of verification - we just need to keep an eye on compiler warnings and fix them appropriately. And don't warry about someone changing your DateFormula field to Integer or Text. A breaking change like this will trigger a lot of other controls, so that we can rest assured that the test case above is completely redundant.