Mistakes we make, Part II

Continuing the previous post, I will linger a bit longer on the topic of testing mistakes, and demonstrate some more examples of testing antipatterns, or tests that should not be.

In my previous post, I was writing about tests that can't fail, thus adding up little to no value for the application under test. Example that I gave was verifying the assignment of a value to a table field, shifting the focus from the client application into the server code. Another variation of the previous example that can be seen sometimes, is the assignment of the field value on a page with the subsequent verification of the same value.

[Test]
procedure AssignBillingPeriodCalcFormulaOnPage()
var
    Customer: Record Customer;
    PeriodDateFormula: DateFormula;
    CustomerCard: TestPage "Customer Card";
    DateFormulaTxt: Label '<%1D>', Comment = '%1 = No. of days',
        Locked = true;
begin
    Evaluate(
        PeriodDateFormula,
        StrSubstNo(DateFormulaTxt, LibraryRandom.RandInt(10)));

    LibrarySales.CreateCustomer(Customer);
    CustomerCard.OpenEdit();
    CustomerCard.GoToRecord(Customer);

    CustomerCard.TDBillingPeriodDateCalc.SetValue(PeriodDateFormula);
    CustomerCard.Close();

    Customer.Find();
    Assert.AreEqual(
        PeriodDateFormula, Customer."TD Billing Period Date Calc.",
        StrSubstNo(
            UnexpectedFieldValueErr,
            Customer.FieldCaption("TD Billing Group Code"), 
            Customer.TableCaption()));
end;

This sample test is obviously just another version of the one reviewed in the previous post, yet even more platform-oriented in a sense that all test actions are carried out by the Business Central server with no client code being executed. The number of actions the test above has to perform is even higher compared to the one clearly oriented on the database objects:

1. Create a customer record.

2. Open the customer page.

3. Navigate to the newly created customer record.

4. Set the date formula value on the card.

5. Close the customer page to save the record.

6. Re-read the customer record from the database and verify the date formula value.

This looks like a cumbersome process. Following the same consistent approach to the test assessment, let's ask ourselves the key question in judgement of the test quality: how can we make this test fail? Compared to the data object-oriented test, this example introduces another possibility for a test failure - it can fail if the field TDBillingPeriodDateCalc is not found on the Customer Card page. But even this little gain does not make the assertion statement any more useful, it still can't evaluate to false in any circumstances other than a bug on the server side. Since we are testing the client application, possible server-side issues should be out of scope. Let's trust that the server team can do their job.

So we can agree that the assertion in the sample test above has no value. But what about the verification of the user interface? This test still fulfills its purpose binding the UI and ensuring that the custom field is displayed on the page and is editable. This can be considered a valuable contribution to the project in some cases - if the field being verified is crucial for the application functionality or usability. Still, considering the flexibility of the UI that Business Central designer offers, this advantage remains doubtful in most cases. If a field can be easily added to a page or removed from it to satisfy personal needs of each user, there is not much value in a dedicated test to verify its presence.

So far, the discussion of the test coverage was concentrated on the date formula field TD Billing Period Date Calc. But the extension introduced two fields, and the second one, TD Billing Group Code, is worth special attention in this discussion. If we follow the same testing pattern, we can assign a value to the table field and assert the value after the validation. Although the test steps are nearly the same as those discussed previously, there is an important difference: firstly, the group code has a table relation linking the field value with the Customer Billing Group table, and secondly, there is AL code in the field's OnValidate trigger.

Below is a test that follows the template of assigning a value to a table field and verifying the same value after the assignment. Let's take a look at the code, whilst keeping in mind the question of its value for the application test coverage.

[Test]
procedure AssignBillingGroupCode()
var
    Customer: Record Customer;
    CustomerBillingGroup: Record "TD Customer Billing Group";
begin
    LibrarySales.CreateCustomer(Customer);
    CustomerBillingGroup.Validate(Code, LibraryUtility.GenerateGUID());
    CustomerBillingGroup.Insert(true);

    Customer.Validate("TD Billing Group Code", CustomerBillingGroup.Code);

    Assert.AreEqual(
        CustomerBillingGroup.Code, Customer."TD Billing Group Code",
        StrSubstNo(
            UnexpectedFieldValueErr,
            Customer.FieldCaption(
                "TD Billing Group Code"), Customer.TableCaption()));
end;

I already pointed out the drawbacks of this kind of tests while writing about the DateFormula field, emphasizing the lack of value for the quality assurance. But the billing group code is something different because of the table relation and the trigger code. Can this be a game changer that allows the test to fulfill its purpose? To answer this question, we still need to ask another one first, our key testing question: what is actually being verified and what changes in the application code can make the test fail?

One change we have to make in the test to make it work is the record in the related table TD Customer Billing Group which must be inserted before validating the billing group code in the customer table extension. If the group is not created prior to the assignment, the test will fail, and often this additional inevitable setup step is erroneously taken for the verification as such: "a billing group created in a test case to satisfy the table relation" is assumed to be equivalent to "the billing group table relation is tested". This assumption is wrong for one simple reason: the test does not protect the table relation from changes. Remove the property value, and the test will still continue running successfully.

It is important to note, though, that there is a potential disruptive change which can be captured by this test - change of the table relation which would link the TD Billing Group Code field with any table other than TD Customer Billing Group. Switching the relation to the table G/L Entry, for example, is going to break the test, and this brings one us step closer to reliable regression test coverage.

Still drawbacks of the test outweigh its value.

1. Assertion statement can never be evaluated to false. In any erroneous scenarios (such as the change of the field data type or modification of the table relation property, as discussed above), execution will be interrupted before reaching the verification step.

2. Possible erroneous scenarios do not execute the validation trigger because the exception will be thrown by the BC server before entering the trigger code. This shifts the focus of the test away from the custom trigger code into the platform.

3. Successful test execution increases code coverage (validation trigger is executed in the normal flow), but due to the fact that the trigger is not verified and no code changes in the trigger can cause the test to fail, this coverage is deceptive and can lead to wrong conclusions in the code coverage analysis.

To conclude the overview, I want to emphasize once again that an AL test that assigns a value to a table field with the only intention of verifying the same value immediately after assignment, does not make much sense. Still even this apparent waste of effort can be turned into an asset. Given the right context, a simple assignment of a value can become a valuable test. But this will be a topic for a separate post.

This was a short introduction into typical basic mistakes which can lead your testing effort in a direction of never failing tests and false expectations. Of course a few examples I gave in the two posts are very far from representing an exhaustive list, and I will return to this topic probably more than once. But the next post will be dedicated to the proper test coverage of the same made-up billing extension.

All the code samples used in this demo can be found in my GitHub repository:

https://github.com/adrogin/CustomerBillingExtension