The case of a missing newline character. One almost detective story.

This almost detective story started when I was building a client app in Business Central which sent data to a JavaScript control add-in to render an image with some text captions. In this simple schema, Business Central extension collects all necessary data, packs everything into a JSON object and sends the bundle to the control add-in, which does its job of drawing the picture. All worked perfectly well until the moment when I needed to enable multiline captions on images. Quick googling and reading suggested that this was perfectly doable - all I needed was to include the newline symbol \n in my caption string. Sure, what can be easier?! And - it did not work.

Instead of the expected multiline caption (like this, for example)

Hello
World

...all I saw was my string with the newline character:

Hello\nWorld

I was blaming the rendering package first, thinking that it just ignored the newline. But the documentation was very clearly specifying required parameters for the line breaks, and all related StackOverflow discussions too evidently supported the correctness of the package. So I started looking for the problem in the data, and very soon found that indeed the JSON object was not formed as I expected, because the newline character was escaped. So the actual property in JSON looked like this:

{ "greetings": "Hello\\nWorld" }

Instead of the newline, the control add-in was receiving an escaped backslash, followed by the "n" letter. And absolutely reasonably rendered this input as a series of printable characters \n instead of the line break.

So, after all, the problem appeared to be on the BC side which was preparing the data. Apparently, this escaping was added to my string at some point when it was being packed into the JSON object. I just added an unescape method to the JavaScript code to remove the unwanted extra backslash symbol and received my multiline captions. Problem solved.

And this could have been the end of the story, but curiosity defeated the common sense once again, and I started digging into the depth of the code to find out what exactly was going on. And this is where the detective story begins.

Short outcome if you want to skip the rest of the story

Long story short, I was using the method JsonObject.Add(Text, Text) to append values:

MyJsonObject.Add("greetings", "Hello\nWorld");

The original code is a bit more complicated, but the gist is in calling the Add method, which (I will add this statement to tickle the reader's own curiosity) does not actually add the escaping sequence, but preserves escaping coming from elsewhere. After running this statement, the JSON object will contain the escaped backslash, as shown above:

{ "greetings": "Hello\\nWorld" }

On the other hand, what I found after some experimenting, is that if I initialize the tokens with the ReadFrom method before adding them to the object, the escaping does not show up:

procedure FormatJsonObject()
var
    GreetingsJson: JsonObject;
    JTok: JsonToken;
begin
    JTok.ReadFrom('"Hello\nWorld"');
    GreetingsJson.Add('MoreGreetings', JTok);
end;

With this initialization, the object demonstrates the neat and nice string with the line break in the middle:

{ "MoreGreetings": "Hello\nWorld" }

So ReadFrom on JsonToken does not preserve the escaping. But where does it come from in the first place? That's what the rest of the story is all about.

The detective part

Now this is the part which you can safely skip if you don't want to wade through code snippets from all the dead ends that I explored on my way to finding the missing line break.

By this time, after my initial experimenting, I had a slightly modified version of the so familiar "Hello World" page extension where I put my code.

trigger OnOpenPage();
var
    GreetingsText: Text;
    GreetingsJson: JsonObject;
    JTok: JsonToken;
begin
    GreetingsText := 'Hello\nworld';
    GreetingsJson.Add('GreetingsText', GreetingsText);
    JTok.ReadFrom('"Hello\nWorld"');
    GreetingsJson.Add('MoreGreetings', JTok);
end;

And this code results in the fancy JSON object with uneven line breaks:

The same JsonObject for better visibility:

{"GreetingsText":"Hello\\nworld","MoreGreetings":"Hello\nWorld"}

As you can see, at this point I already tried to fiddle with the assignments to see if the trick may reside in the type cast from NavText to NavJsonToken and tried to assign it to a text variable first instead of initializing the token from a text constant. But no, this did not work.

So I switched from AL to .Net to delve a bit deeper. The following examples are in C# and require a few libraries to be included in the project. If you want to try it too, here is the using section for all samples.

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Microsoft.Dynamics.Nav.Runtime;
using Microsoft.Dynamics.Nav.Types;

And the complete code can be found in my Blog Samples repository.

Test #1 - Simply write a NavText string to console to see how the output is handled.

The fist idea I tried was making sure that the type cast between NavText and .Net String does not introduce any unexpected side effects. Here, I don't do anything more than initializing a NavText variable and writing it to the console.

void writeMultilineNavTextToConsole() {
    NavText navText = NavText.Create("Multiline\nvalue");
    Console.WriteLine(navText);
    Console.WriteLine();
}

And the console output does not pose any surprises. It's the expected multiline string.

Multiline
value

Test #2 - Instantiate a Newtonsoft JObject by parsing a string.

Internally Business Central relies on the Newtonsoft.Json library to handle JSON data, and deep inside, Business Central's Json* types (JsonObject, JsonValue, JsonToken, and JsonArray), are wrappers around respective Newtonsoft classes. In my second test, I tried to peek into Newtonsoft's initialisation to see if it might add escape sequences to the input.

Here, I declare a string constant, feed it into the JObject.Parse method, and observe the output.

void jsonObjectInitializedFromString() {
    string jsonString = @"{
        'key': 'Multiline\nvalue'
    }";

    JObject jObj = JObject.Parse(jsonString);

    Console.WriteLine(jObj);
    Console.WriteLine();
}

So far, no additional escaping in the console output.

{
  "key": "Multiline\nvalue"
}

Test #3 - Add properties to NavJsonObject from a .Net string and a NavText

In this test, I take another baby step forward, into the world of Business Central entities. Here, I declare two variables: one of the .Net string type, and the other one NavText, and pass both to the NavJsonObject.Add method to see if the source type can make any difference.

void navJonObjectInitializedFromNavText() {
    string dotnetString = "Multiline\nvalue";
    NavText navText = NavText.Create("One\nmore\nmultiline");

    NavJsonObject navJObj = NavJsonObject.Default;

    navJObj.ALAdd(DataError.ThrowError, "key", dotnetString);
    navJObj.ALAdd(DataError.ThrowError, "anotherKey", navText);

    Console.WriteLine(navJObj);
    Console.WriteLine();
}

And still no difference: both values are successfully added to the object, and no extra escaping backslashes are visible in the output.

{"key":"Multiline\nvalue","anotherKey":"One\nmore\nmultiline"}

The only noticeable difference is that JSON is not prettified when printed from a NavJsonObject.

Test #4 - Initialize a NavJsonToken by calling ReadFrom and add the token to a NavJsonObject

Here, I am loading the text content into a JsonToken via ReadFrom and adding the token to the object. Since this test is just a variation of my "sunshine scenario", only transferred from AL to C#, I don't really expect any surprises.

void navJsonObjectInitializedByReadFrom() {
    NavJsonObject navJObj = NavJsonObject.Default;
    NavJsonToken navJTok = NavJsonToken.Default;

    navJTok.ALReadFrom(DataError.ThrowError, "\"Hello\nWorld\"");
    navJObj.ALAdd(DataError.ThrowError, "key", navJTok);

    Console.WriteLine(navJObj);
    Console.WriteLine();
}

And rightly so - the JSON content in the console shows one newline.

{"key":"Hello\nWorld"}

AL to C# translation

Finally, after all the previous exercises I did what I should have done from the very beginning - I went to look at the transpiled code of my Hello World page in C#. And here I found the key to the solution. Quite obvious solution, I must say. In the C# source, all quotation marks and control characters are already escaped.

protected override void OnRun()
{
    StmtHit(0);
    this.greetingsText = new NavText("Hello\\nworld");
    StmtHit(1);
    this.greetingsJson.ALAdd(
        DataError.ThrowError, "GreetingsText", this.greetingsText);
    StmtHit(2);
    this.jTok.ALReadFrom(DataError.ThrowError, "\"Hello\\nWorld\"");
    StmtHit(3);
    this.greetingsJson.ALAdd(
        DataError.ThrowError, "MoreGreetings", this.jTok);
}

This is where the mistake in my logic was hiding. I was assuming that my text constants contain unescaped newline characters, but of course this is wrong. It's the AL transpiler itself that injects escape sequences into the text. And this makes perfect sense. In AL, we use single quotes to denote text constants and don't worry about escaping anything but an occasional apostrophe in the text. .Net strings are marked by double quotes and must escape all quotes and backslashes inside, so the translation from AL to C# has to deal with string transformation.

One answer is found, now I know the source of my trouble. But another question still remains - why does JsonToken.ReadFrom behaves so differently from the direct assignment of a text constant to the token?

Let's take another deep dive to answer this question.

Test #5 - Code behind JsonToken.ReadFrom(Text)

The first step towards our goal (which is understanding the inner differences in two methods of the token initialisation) is to see what hides behind AL functions.

When we look behind the AL façade of the ReadFrom method, we will find the following sequence.

void jTokenInitializedByReadFrom() {
    NavJsonObject navJObj = NavJsonObject.Default;

    using StringReader stringReader =
        new StringReader("\"Hello\\nWorld\"");
    using JsonTextReader jsonTextReader =
        new JsonTextReader(stringReader);
    JToken jTok = JToken.ReadFrom(jsonTextReader);
    NavJsonToken navJTok = NavJsonToken.Create(jTok);

    navJObj.ALAdd(DataError.ThrowError, "key", navJTok);

    Console.WriteLine(navJObj);
    Console.WriteLine();
}

ReadFrom creates an instance of System.IO.StringReader, which is passed to the constructor of Newtonsoft.Json.JsonTextReader. It is JsonTextReader that will execute the StringReader to consume the raw text data and parse the input into a JSON object.

When I execute the code above, I see that the resulting token is unescaped.

{"key":"Hello\nWorld"}

Now the last step to pinpoint this unescaping brings me into the Newtonsoft library, the code of the JsonTextReader.

And here we can see that although the input string in the StringReader is still escaped, each escape sequence is considered a single character. The char buffer is unescaped after executing the StringReader.Read method.

This was the source of the mystery. It wasn't any JSON manipulation adding the extra backslash. In fact, it's the StringReader that removes escape characters inserted by the AL transpiler.

Test #6 - Code behind JsonObject.Add(Text, Text)

And, as the final chord, one more code snippet attempting to reveal the internal process of the JsonObject.Add(Text, Text) method.

void implicitConversionNavTextToNavJsonToken() {
    Func<string, NavJsonToken> initializeToken = (value) => {
    Func<NavJsonToken> initializeToken = () => {
        NavJsonValue navJValue = NavJsonValue.Default;
        navJValue.ALSetValue(value);
        return navJValue;
    };

    NavJsonObject navJObj = NavJsonObject.Default;
    navJObj.ALAdd(
        DataError.ThrowError, "key",
        initializeToken("\"Hello\\nWorld\""));

    Console.WriteLine(navJObj);
    Console.WriteLine();
}

Internally it calls an implicit conversion operator which I imitated with the delegate. The internal conversion creates an instance of NavJsonToken and assigns the text value unchanged, so the output of this code (unlike the previous example) has all the escape sequences in place.

{"key":"\"Hello\\nWorld\""}

Here my detective story ends. Thank you for reading if you managed to follow all the twists of the plot.

I already outlined the conclusion above, but can add the main take-away for myself: never assume that your AL text string will look the same after the translation to .Net.