Mastering Arduinojson 6: Efficient Json Serialization For Embedded C++
Mastering Arduinojson 6: Efficient Json Serialization For Embedded C++
CREATOR OF ARDUINOJSON
Mastering ArduinoJson 6
Efficient JSON serialization for embedded C++
SECOND EDITION
Contents
Contents iv
1 Introduction 1
1.1 About this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Code samples . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 What’s new in the second edition . . . . . . . . . . . . . . . . . 3
1.2 Introduction to JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 What is JSON? . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 What is serialization? . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.3 What can you do with JSON? . . . . . . . . . . . . . . . . . . 5
1.2.4 History of JSON . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.5 Why is JSON so popular? . . . . . . . . . . . . . . . . . . . . . 8
1.2.6 The JSON syntax . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.7 Binary data in JSON . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.8 Comments in JSON . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3 Introduction to ArduinoJson . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.1 What ArduinoJson is . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.2 What ArduinoJson is not . . . . . . . . . . . . . . . . . . . . . 14
1.3.3 What makes ArduinoJson different? . . . . . . . . . . . . . . . 15
1.3.4 Does size really matter? . . . . . . . . . . . . . . . . . . . . . . 17
1.3.5 What are the alternatives to ArduinoJson? . . . . . . . . . . . . 18
1.3.6 How to install ArduinoJson . . . . . . . . . . . . . . . . . . . . 20
1.3.7 The examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2.3 Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3.1 What is a pointer? . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3.2 Dereferencing a pointer . . . . . . . . . . . . . . . . . . . . . . 36
2.3.3 Pointers and arrays . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.4 Taking the address of a variable . . . . . . . . . . . . . . . . . 38
2.3.5 Pointer to class and struct . . . . . . . . . . . . . . . . . . . 38
2.3.6 Pointer to constant . . . . . . . . . . . . . . . . . . . . . . . . 39
2.3.7 The null pointer . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3.8 Why use pointers? . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.4 Memory management . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.4.1 malloc() and free() . . . . . . . . . . . . . . . . . . . . . . . . 43
2.4.2 new and delete . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.4.3 Smart pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.4.4 RAII . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.5.1 What is a reference? . . . . . . . . . . . . . . . . . . . . . . . 47
2.5.2 Differences with pointers . . . . . . . . . . . . . . . . . . . . . 47
2.5.3 Reference to constant . . . . . . . . . . . . . . . . . . . . . . . 48
2.5.4 Rules of references . . . . . . . . . . . . . . . . . . . . . . . . 49
2.5.5 Common problems . . . . . . . . . . . . . . . . . . . . . . . . 49
2.5.6 Usage for references . . . . . . . . . . . . . . . . . . . . . . . . 50
2.6 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.6.1 How are the strings stored? . . . . . . . . . . . . . . . . . . . . 51
2.6.2 String literals in RAM . . . . . . . . . . . . . . . . . . . . . . . 51
2.6.3 String literals in Flash . . . . . . . . . . . . . . . . . . . . . . . 52
2.6.4 Pointer to the “globals” section . . . . . . . . . . . . . . . . . . 53
2.6.5 Mutable string in “globals” . . . . . . . . . . . . . . . . . . . . 54
2.6.6 A copy in the stack . . . . . . . . . . . . . . . . . . . . . . . . 55
2.6.7 A copy in the heap . . . . . . . . . . . . . . . . . . . . . . . . 56
2.6.8 A word about the String class . . . . . . . . . . . . . . . . . . 57
2.6.9 Pass strings to functions . . . . . . . . . . . . . . . . . . . . . 58
2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7 Troubleshooting 231
7.1 Program crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
7.1.1 Undefined Behaviors . . . . . . . . . . . . . . . . . . . . . . . . 232
7.1.2 A bug in ArduinoJson? . . . . . . . . . . . . . . . . . . . . . . 232
7.1.3 Null string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7.1.4 Use after free . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7.1.5 Return of stack variable address . . . . . . . . . . . . . . . . . 235
7.1.6 Buffer overflow . . . . . . . . . . . . . . . . . . . . . . . . . . 236
7.1.7 Stack overflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
7.1.8 How to diagnose these bugs? . . . . . . . . . . . . . . . . . . . 238
7.1.9 How to prevent these bugs? . . . . . . . . . . . . . . . . . . . . 241
7.2 Deserialization issues . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
7.2.1 IncompleteInput . . . . . . . . . . . . . . . . . . . . . . . . . . 243
7.2.2 InvalidInput . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
7.2.3 NoMemory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.2.4 NotSupported . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
7.2.5 TooDeep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
7.3 Serialization issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
7.3.1 The JSON document is incomplete . . . . . . . . . . . . . . . . 252
7.3.2 The JSON document contains garbage . . . . . . . . . . . . . . 252
7.3.3 Too much duplication . . . . . . . . . . . . . . . . . . . . . . . 254
Contents xi
9 Conclusion 310
Index 311
Chapter 3
Deserialize with ArduinoJson
It is not the language that makes programs appear simple. It is the pro-
grammer that make the language appear simple!
– Robert C. Martin, Clean Code: A Handbook of Agile Software
Craftsmanship
Chapter 3 Deserialize with ArduinoJson 63
Now that you’re familiar with JSON and C++, we’re going to learn how to use Arduino-
Json. This chapter explains everything there is to know about deserialization. As we’ve
seen, deserialization is the process of converting a sequence of bytes into a memory
representation. In our case, it means converting a JSON document to a hierarchy of
C++ structures and arrays.
In this chapter, we’ll use a JSON response from GitHub’s
API as an example. As you already know, GitHub is a
hosting service for source code; what you may not know,
however, is that GitHub provides a very powerful API that
allows you to interact with the platform.
There are many things we could do with GitHub’s API, but
in this chapter, we’ll only focus on a small part. We’ll get
your ten most popular repositories and display their names,
numbers of stars, and numbers of opened issues.
There are several versions of GitHub’s API; we’ll use the
latest one: the GraphQL API (or v4). We’ll use this one
because it allows us to make only one request and because
it returns a considerably smaller response.
If you want to run the example, you’ll need a user account on GitHub and a personal
access token. Don’t worry; we’ll see that later.
Because GitHub only allows secure connections, we need a microcontroller that supports
HTTPS. We’ll use the ESP8266 with the ESP8266HTTPClient as an example. If you
want to use ArduinoJson with EthernetClient, WiFiClient, or WiFiClientSecure, check
out the case studies in the last chapter.
Now that you know where we are going, we’ll back up a few steps and start with a basic
example. Then, we’ll progressively learn new things so that, by the end of the chapter,
we’ll finally be able to interact with GitHub.
Chapter 3 Deserialize with ArduinoJson 64
We’ll begin this tutorial with the simplest situation: a JSON document in memory.
More precisely, our JSON document resides in the stack in a writable location. This
fact is going to matter, as we will see later.
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 4241
},
"issues": {
"totalCount": 12
}
}
As you see, it’s a JSON object that contains two nested objects. It contains the name
of the repository, the number of stars, and the number of open issues.
In the previous chapter, we saw that this code creates a duplication of the string in
the stack. We know it’s a code smell in production code, but it’s a good example for
learning. This unusual construction create a writable (i.e., not read-only) input string,
which is important for our first contact with ArduinoJson.
Chapter 3 Deserialize with ArduinoJson 65
As we saw in the introduction, one of the unique features of ArduinoJson is its fixed
memory allocation strategy.
Here is how it works:
1. First, you create a JsonDocument to reserve a specified amount of memory.
2. Then, you deserialize the JSON document.
3. Finally, you destroy the JsonDocument, which releases the reserved memory.
The memory of the JsonDocument can be either in the stack or in the heap. The location
depends on the derived class you choose. If you use a StaticJsonDocument, it will be in
the stack; if you use a DynamicJsonDocument, it will be in the heap.
A JsonDocument is responsible for reserving and releasing the memory used by Arduino-
Json. It is an instance of the RAII idiom that we saw in the previous chapter.
When you create a JsonDocument, you must specify its capacity in bytes.
In the case of DynamicJsonDocument, you set the capacity via a constructor argument:
DynamicJsonDocument doc(capacity);
As it’s a parameter of the constructor, you can use a regular variable, whose value can
change at run-time.
Chapter 3 Deserialize with ArduinoJson 66
In the case of a StaticJsonDocument, you set the capacity via a template parameter:
StaticJsonDocument<capacity> doc;
As it’s a template parameter, you cannot use a variable. Instead, you must use a
constant expression, which means that the value must be computed at compile-time.
As we said in the previous chapter, the compiler manages the stack, so it needs to know
the size of each variable when it compiles the program.
Now comes a tricky question for every new user of ArduinoJson: what should be the
capacity of my JsonDocument?
To answer this question, you need to know what ArduinoJson stores in the JsonDocument.
ArduinoJson needs to store a data structure that mirrors the hierarchy of objects in the
JSON document. In other words, the JsonDocument contains objects which relate to one
another the same way they do in the JSON document.
Therefore, the capacity of the JsonDocument highly depends on the complexity of the
JSON document. If it’s just one object with few members, like our example, a few
dozens of bytes are enough. If it’s a massive JSON document, like OpenWeatherMap’s
response, up to a hundred kilobytes are needed.
ArduinoJson provides macros for computing precisely the capacity of the JsonDocument.
The macro to compute the size of an object is JSON_OBJECT_SIZE(). It takes one argu-
ment: the number of members in the object.
Here is how to compute the capacity for our sample document:
Since our JsonDocument is small, we can keep it in the stack. By using the stack, we
reduce the size of the executable and improve the performance, because we avoid the
overhead due to the management of the heap.
Here is our program so far:
Of course, if the JsonDocument were bigger, it would make sense to move it to the heap.
We’ll do that later.
Now that the JsonDocument is ready, we can parse the input with deserializeJson():
if (err) {
Serial.print(F("deserializeJson() failed with code "));
Serial.println(err.c_str());
}
In the “Troubleshooting” chapter, we’ll look at each error code and see what can cause
the error.
Chapter 3 Deserialize with ArduinoJson 69
There are multiple ways to extract the values from a JsonDocument; let’s start with the
simplest:
Not everyone likes implicit casts, mainly because it messes with overload resolution,
with template parameter type deduction, and with the auto keyword. That’s why
ArduinoJson offers an alternative syntax with explicit type conversion.
Implicit or explicit?
We saw two different syntaxes to do the same thing. They are all equivalent
and lead to the same executable.
I prefer the implicit version because it allows using the “or” operator, as
we’ll see. I use the explicit version only to solve an ambiguity.
We saw how to extract values from an object, but we didn’t do error checking. Let’s
see what happens when a value is missing.
When you try to extract a value that is not present in the document, ArduinoJson
returns a default value. This value depends on the requested type:
The two last lines (JsonArray and JsonObject) happen when you extract a nested array
or object, we’ll see that in a later section.
Chapter 3 Deserialize with ArduinoJson 71
No exceptions
ArduinoJson never throws exceptions. Exceptions are an excellent C++ fea-
ture, but they produce large executables, which is unacceptable for micro-
controllers.
Sometimes, the default value from the table above is not what you want. In this
situation, you can use the operator | to change the default value. I call it the “or”
operator because it provides a replacement when the value is missing or incompatible.
Here is an example:
This feature is handy to specify default configuration values, like in the snippet above,
but it is even more useful to prevent a null string from propagating.
Here is an example:
strlcpy(), a function that copies a source string to a destination string, crashes if the
source is null. Without the operator |, we would have to use the following code:
char hostname[32];
const char* configHostname = config["hostname"];
if (configHostname != nullptr)
strlcpy(hostname, configHostname, 32);
else
strcpy(hostname, "arduinojson.org");
We’ll see a complete example that uses this syntax in the case studies.
Chapter 3 Deserialize with ArduinoJson 72
In the previous section, we extracted the values from an object that we knew in advance.
Indeed, we knew that the JSON object had three members: a string named “name,” a
nested object named “stargazers,” and another nested object named “issues.” In this
section, we’ll see how to inspect an unknown object.
Now that we have a JsonObject, we can look at all the keys and their associated values.
In ArduinoJson, a key-to-value association, or a key-value pair, is represented by the
type JsonPair.
We can enumerate all pairs with a simple for loop:
To know the actual type of the value in a JsonVariant, you need to call
JsonVariant::is<T>(), where T is the type you want to test.
// Is it a string?
if (p.value().is<char*>()) {
// Yes!
// We can get the value via implicit cast:
const char* s = p.value();
// Or, via explicit method call:
auto s = p.value().as<char*>();
}
If you use this with our sample document, you’ll see that only the member “name”
contains a string. The two others are objects, as is<JsonObject>() would confirm.
There are a limited number of types that a variant can use: boolean, integer, float,
string, array, object. However, different C++ types can store the same JSON type; for
example, a JSON integer could be a short, an int or a long in the C++ code.
The following table shows all the C++ types you can use as a parameter for
JsonVariant::is<T>() and JsonVariant::as<T>().
More on arduinojson.org
The complete list of types that you can use as a parameter for
JsonVariant::is<T>() can be found in the API Reference.
Chapter 3 Deserialize with ArduinoJson 75
If you have an object and want to know whether a key is present or not, you can call
JsonObject::containsKey().
Here is an example:
However, I don’t recommend using this function because you can avoid it most of the
time.
Here is an example where we can avoid containsKey():
The code above is not horrible, but it can be simplified and optimized if we just remove
the call to containsKey():
This code is faster and smaller because it only looks for the key “error” once (whereas
the previous code did it twice).
Chapter 3 Deserialize with ArduinoJson 76
We’ve seen how to parse a JSON object from GitHub’s response; it’s time to move up
a notch by parsing an array of objects. Indeed, our goal is to display the top 10 of
your repositories, so there will be up to 10 objects in the response. In this section, we’ll
suppose that there are only two repositories, but you and I know that it will be more in
the end.
Here is the new sample JSON document:
[
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 4241
},
"issues": {
"totalCount": 12
}
},
{
"name": "pdfium-binaries",
"stargazers": {
"totalCount": 116
},
"issues": {
"totalCount": 9
}
}
]
Let’s deserialize this array. You should now be familiar with the process:
Chapter 3 Deserialize with ArduinoJson 77
// Parse succeeded?
if (err) {
Serial.print(F("deserializeJson() returned "));
Serial.println(err.c_str());
return;
}
As said earlier, an hard-coded input like that would never happen in production code,
but it’s a good step for your learning process.
You can see that the expression for computing the capacity of the JsonDocument is quite
complicated:
• There is one array of two elements: JSON_ARRAY_SIZE(2)
• In this array, there are two objects with three members: 2*JSON_OBJECT_SIZE(3)
• In each object, there are two objects with one member: 4*JSON_OBJECT_SIZE(1)
Chapter 3 Deserialize with ArduinoJson 78
For complex JSON documents, the expression to compute the capacity of the
JsonDocument becomes impossible to write by hand. I did it above so that you un-
derstand the process, but in practice, we use a tool to do that.
This tool is the “ArduinoJson Assistant.” You can use it online at arduinojson.org/
assistant.
You just need to paste your JSON document in the box on the left, and the Assistant
displays the expression in the box on the right. Don’t worry; the Assistant respects your
privacy: it computes the expression locally in the browser; it doesn’t send your JSON
document to a web service.
Chapter 3 Deserialize with ArduinoJson 79
The process of extracting the values from an array is very similar to the one for objects.
The only difference is that arrays are indexed by an integer, whereas objects are indexed
by a string.
To get access to the repository information, we need to get the JsonObject from the
JsonDocument, except that, this time, we’ll pass an integer to the subscript operator ([]).
Of course, we could have inlined the repo0 variable (i.e., write doc[0]["name"] each
time), but it would cost an extra lookup for each access to the object.
It may not be obvious, but the program above uses implicit casts. Indeed, the subscript
operator ([]) returns a JsonVariant that is implicitly converted to a JsonObject.
Again, some programmers don’t like implicit casts, that is why ArduinoJson offers an
alternative syntax with as<T>(). For example:
All of this should sound very familiar because it’s similar to what we’ve seen for ob-
jects.
Chapter 3 Deserialize with ArduinoJson 80
When we learned how to extract values from an object, we saw that, if a member is
missing, a default value is returned (for example, 0 for an int). Similarly, ArduinoJson
returns a default value when you use an index that is out of the range of an array.
Let’s see what happens in our case:
The index 666 doesn’t exist in the array, so a special value is returned: a null JsonObject.
Remember that JsonObject is a reference to an object stored in the JsonDocument. In
this case, there is no object in the JsonDocument, so the JsonObject points to nothing:
it’s a null reference.
You can test if a reference is null by calling isNull():
A null JsonObject evaluates to false, so you can easily verify that it’s not null:
A null JsonObject looks like an empty object, except that you cannot modify it. You
can safely call any function of a null JsonObject, it simply ignores the call and returns
a default value. Here is an example:
In the previous section, our example was very straightforward because we knew that the
JSON array had precisely two elements, and we knew the content of these elements. In
this section, we’ll see what tools are available when you don’t know the content of the
array.
Do you remember what we did when we wanted to enumerate the key-value pairs of an
object? Right! We began by calling JsonDocument::as<JsonObject>() to get a reference
to the root object.
Similarly, if we want to enumerate all the elements of an array, the first thing we have
to do it to get a reference to it:
Again, JsonArray is a reference to an array stored in the JsonDocument; it’s not a copy
of the array. When you apply changes to the JsonArray, the changes are reflected on
the JsonDocument.
If you know absolutely nothing about the input (which is strange), you need to determine
a memory budget allowed for parsing the input. For example, you could decide that
10KB of heap memory is the maximum you accept to spend on JSON parsing.
This constraint looks terrible at first, especially if you’re a desktop or server application
developer; but, once you think about it, it makes complete sense. Indeed, your program
will run in a loop on dedicated hardware. Since the hardware doesn’t change, the
amount of memory is always the same. Having an elastic capacity would just produce
a larger and slower program with no additional value; it would also increase the heap
fragmentation, which we must avoid at all costs.
Chapter 3 Deserialize with ArduinoJson 83
However, most of the time, you know a lot about your JSON document. Indeed, there
are usually a few possible variations in the input. For example, an array could have
between zero and four elements, or an object could have an optional member. In that
case, use the ArduinoJson Assistant to compute the size of each variant, and pick the
biggest.
The first thing you want to know about an array is the number of elements it contains.
This is the role of JsonArray::size():
As the name may be confusing, let me clarify: JsonArray::size() returns the number
of elements, not the memory consumption. If you want to know how many bytes of
memory are used, call JsonDocument::memoryUsage():
Note that there is also a JsonObject::size() that returns the number of key-value pairs
in an object, but it’s rarely useful.
3.7.4 Iteration
Now that you have the size of the array, you probably want to write the following code:
The code above works but is terribly slow. Indeed, ArduinoJson stores arrays as linked
lists, so accessing an element at a random location costs O(n); in other words, it takes
n iterations to get to the nth element. Moreover, the value of JsonArray::size() is not
cached, so it needs to walk the linked list too.
That’s why it is essential to avoid arr[i] and arr.size() in a loop. Instead, you should
use the iteration feature of JsonArray, like so:
With this syntax, the internal linked list is walked only once, and it is as fast as it gets.
I used a JsonObject in the loop because I knew that the array contains objects. If it’s
not your case, you can use a JsonVariant instead.
We test the type of array elements the same way we did for object members: by using
JsonVariant::is<T>().
Here is an example:
Chapter 3 Deserialize with ArduinoJson 85
// Same in a loop
for (JsonVariant elem : arr) {
// Is the current element an object?
if (elem.is<JsonObject>()) {
JsonObject obj = elem;
// ...
}
}
There is nothing new here, as it’s exactly what we saw for object members.
Chapter 3 Deserialize with ArduinoJson 86
3.8.1 Definition
At the beginning of this chapter, we saw how to parse a JSON document from a
writable source. Indeed, the input variable was a char[] in the stack, and therefore, it
was writable. I told you that this fact would matter, and it’s time to explain.
ArduinoJson behaves differently with writable inputs and read-only inputs.
When the argument passed to deserializeJson() is of type char* or char[], ArduinoJson
uses a mode called “zero-copy.” It has this name because the parser never makes any
copy of the input; instead, it stores pointers pointing inside the input buffer.
In the zero-copy mode, when a program requests the content of a string member,
ArduinoJson returns a pointer to the beginning of the string in the input buffer. To
make it possible, ArduinoJson inserts null-terminators at the end of each string; it is
the reason why this mode requires the input to be writable.
3.8.2 An example
To illustrate how the zero-copy mode works, let’s have a look at a concrete example.
Suppose we have a JSON document that is just an array containing two strings:
["hip","hop"]
Chapter 3 Deserialize with ArduinoJson 87
And let’s says that the variable is a char[] at address 0x200 in memory:
After parsing the input, when the program requests the value of the first element,
ArduinoJson returns a pointer whose address is 0x202 which is the location of the string
in the input buffer:
deserializeJson(doc, input);
We naturally expect hip to be "hip" and not "hip\",\"hop\"]"; that’s why ArduinoJson
adds a null-terminator after the first p. Similarly, we expect hop to be "hop" and not
"hop\"]", so a second null-terminator is added.
'[' '"' 'h' 'i' 'p' '"' ',' '"' 'h' 'o' 'p' '"' ']' 0
0x200
deserializeJson()
'[' '"' 'h' 'i' 'p' 0 ',' '"' 'h' 'o' 'p' 0 ']' 0
0x202 0x208
Adding null-terminators is not the only thing the parser modifies in the input buffer. It
also replaces escaped character sequences, like \n by their corresponding ASCII charac-
ters.
Chapter 3 Deserialize with ArduinoJson 88
I hope this explanation gives you a clear understanding of what the zero-copy mode is
and why the input is modified. It is a bit of a simplified view, but the actual code is
very similar.
As we saw, in the zero-copy mode, ArduinoJson returns pointers to the input buffer.
So, for a pointer to be valid, the input buffer must be in memory at the moment the
pointer is dereferenced.
If a program dereferences the pointer after the destruction of the input buffer, it is very
likely to crash instantly, but it could also work for a while and crash later, or it could
have nasty side effects. In the C++ jargon, this is what we call an “Undefined Behavior”;
we’ll talk about that in “Troubleshooting”.
Here is an example:
// Declare a pointer
const char *hip;
// New scope
{
// Declare the input in the scope
char input[] = "[\"hip\",\"hop\"]";
// Parse input
deserializeJson(doc, input);
JsonArray arr = doc.as<JsonArray>();
// Save a pointer
hip = arr[0];
}
// input is destructed now
We saw how ArduinoJson behaves with a writable input, and how the zero-copy mode
works. It’s time to see what happens when the input is read-only.
Let’s go back to our previous example except that, this time, we change its type from
char[] to const char*:
As we saw in the C++ course, this statement creates a sequence of bytes in the “globals”
area of the RAM. This memory is supposed to be read-only; that’s why we need to add
the const keyword.
Previously, we had the whole string duplicated in the stack, but it’s not the case anymore.
Instead, the stack only contains the pointer input pointing to the beginning of the string
in the “globals” area.
As we saw in the previous section, in the zero-copy mode, ArduinoJson stores pointers
pointing inside the input buffer. We saw that it has to replace some characters of the
input with null-terminators.
With a read-only input, ArduinoJson cannot do that anymore, so it needs to make copies
of "hip" and "hop". Where do you think the copies would go? In the JsonDocument, of
course!
In this mode, the JsonDocument holds a copy of each string, so we need to increase its
capacity. Let’s do the computation for our example:
1. We still need to store an object with two elements, that’s JSON_ARRAY_SIZE(2).
2. We have to make a copy of the string "hip", that’s 4 bytes including the null-
terminator.
3. We also need to copy the string "hop", that’s 4 bytes too.
Chapter 3 Deserialize with ArduinoJson 91
In practice, you should not use the exact length of the strings. It’s safer to add a bit
of slack, in case the input changes. My advice is to add 10% to the longest possible
string, which gives a reasonable margin.
3.9.3 Practice
Apart from the capacity of the JsonDocument, we don’t need to change anything to the
program.
Here is the complete hip-hop example with a read-only input:
// A read-only input
const char* input = "[\"hip\",\"hop\"]";
Chapter 3 Deserialize with ArduinoJson 92
const char* is not the sole read-only input that ArduinoJson supports. For example,
you can also use a String:
It’s also possible to use a Flash string, but there is one caveat. As we said in the C++
course, ArduinoJson needs a way to figure out if the input string is in RAM or in Flash.
To do that, it expects a Flash string to have the type const __FlashStringHelper*. If
you declare a char[] PROGMEM, ArduinoJson will not consider it as Flash string, unless
you cast it to const __FlashStringHelper*.
Alternatively, you can use the F() macro, which casts the pointer to the right type:
Chapter 3 Deserialize with ArduinoJson 93
In the next section, we’ll see another kind of read-only input: streams.
Chapter 3 Deserialize with ArduinoJson 94
In the Arduino jargon, a stream is a volatile source of data, like the serial port or a
TCP connection. As opposed to a memory buffer, which allows reading any bytes at
any location (after all, that’s what the acronym “RAM” means), a stream only allows
reading one byte at a time and cannot rewind.
This concept is materialized by the Stream abstract class. Here are examples of classes
derived from Stream:
std::istream
b In the C++ Standard Library, an input stream is represented by the class
std::istream.
ArduinoJson can use both Stream and std::istream.
As an example, we’ll create a program that reads a JSON file stored on an SD card.
We suppose that this file contains the array we used as an example earlier.
The program will just read the file and print the information for each repository.
Chapter 3 Deserialize with ArduinoJson 95
// Open file
File file = SD.open("repos.txt");
Now is the time to parse the real data coming from GitHub’s API!
As I said, we need a microcontroller that supports HTTPS, so we’ll use an ESP8266
with the library “ESP8266HTTPClient.” Don’t worry if you don’t have a compatible
board; we’ll see other configurations in the case studies.
Chapter 3 Deserialize with ArduinoJson 96
Access token
Before using this API, you need a GitHub account and a “personal access token.” This
token grants access to the GitHub API from your program; we might also call it an
“API key.” To create it, open GitHub in your browser and follow these steps:
1. Go to your personal settings.
2. Go in “Developer settings.”
3. Go in “Personal access token.”
4. Click on “Generate a new token.”
5. Enter a name, like “ArduinoJson tutorial.”
6. Check the scopes (i.e., the permissions); we only need “public_repo.”
7. Click on “Generate token.”
8. GitHub shows the token, copy it now because it disappears after a few seconds.
You can see each step in the picture below:
Chapter 3 Deserialize with ArduinoJson 97
GitHub won’t show the token again, so don’t waste any second and write it in the source
code:
With this token, our program can authenticate with GitHub’s API. All we need to do is
to add the following HTTP header to each request:
The request
To interact with the new GraphQL API, we need to send a POST request (as opposed to
the more common GET request) to the URL https://api.github.com/graphql.
The body of the POST request is a JSON object that contains one string named “query.”
This string contains a GraphQL query. For example, if we want to get the name of the
authenticated user, we need to send the following JSON document in the body of the
request:
{
"query": "{viewer{name}}"
}
The GraphQL syntax and the details of GitHub’s API are obviously out of the scope of
this book, so I’ll simply say that the GraphQL query allows you to select the information
you want within the universe of information that the API exposes.
In our case, we want to retrieve the names, numbers of stars, and numbers of opened
issues of your ten most popular repositories. Here is the corresponding GraphQL query:
{
viewer {
name
repositories(ownerAffiliations: OWNER,
orderBy: {
direction: DESC,
Chapter 3 Deserialize with ArduinoJson 98
field: STARGAZERS
},
first: 10) {
nodes {
name
stargazers {
totalCount
}
issues(states: OPEN) {
totalCount
}
}
}
}
}
To find the correct query, I used the GraphQL API Explorer. With this tool, you can
test GraphQL queries in your browser. You’ll find it in GitHub’s API documentation.
We’ll reduce this query to a single line to save some space and bandwidth; then, we’ll
put it in the “query” string in the JSON object. Since we haven’t talked about JSON
serialization yet, we’ll hard-code the string in the program.
To summarize, here is how we will send the request:
HTTPClient http;
http.begin("https://api.github.com/graphql", GITHUB_FINGERPRINT);
http.addHeader("Authorization", "bearer " GITHUB_TOKEN));
http.POST("{\"query\":\"{viewer{name,repositories(ownerAffiliations:...");
To get the current fingerprint, open api.github.com in your browser, click on the pad-
lock, then click on “Certificate.” From there, you should find the fingerprint. The
picture below shows where to find it with Chrome on Windows:
Maintenance nightmare
Fingerprint validation is mandatory with ESP8266HTTPClient.
It’s good from a security perspective because you are sure to send your
credentials to the right entity. However, it’s terrible from a maintenance
perspective because you need to update the code every time the certificate
changes (every year in this case).
In the case studies, we’ll see how to perform an HTTPS request without
validating the certificate.
The response
As you see, we call getStream() to get the internal stream. Unfortunately, when we
do that, we bypass the part of ESP8266HTTPClient that handles chunked transfer
Chapter 3 Deserialize with ArduinoJson 100
encoding. To make sure GitHub doesn’t return a chunked response, we must set the
protocol to HTTP 1.0:
Because the version of the protocol is part of the request, we must call useHTTP10()
before calling POST().
Now that we have the stream, we can pass it to deserializeJson():
Here, we used a DynamicJsonBuffer because it is too big for the stack. As usual, I used
the ArduinoJson Assistant to compute the capacity.
The body contains the JSON document that we want to deserialize. It’s a little more
complicated than what we saw earlier. Indeed, the JSON array is not at the root but
under data.viewer.repositories.nodes, as you can see below:
{
"data": {
"viewer": {
"name": "Benoît Blanchon",
"repositories": {
"nodes": [
{
"name": "ArduinoJson",
"stargazers": {
"totalCount": 3420
},
"issues": {
"totalCount": 21
}
},
Chapter 3 Deserialize with ArduinoJson 101
{
"name": "WpfBindingErrors",
"stargazers": {
"totalCount": 48
},
"issues": {
"totalCount": 3
}
}
]
}
}
}
}
So, compared to what we saw earlier, the only difference is that we’ll have to walk
several objects before getting the reference to the array. The following line will do:
The code
// Disconnect
http.end();
If all works well, this program should print something like so:
You can find the complete source code of this example in the GitHub folder in the zip file
provided with the book. Compared to what is shown above, the source code handles the
connection to the WiFi network, check errors, and uses Flash strings when possible.
Chapter 3 Deserialize with ArduinoJson 103
3.11 Summary
In this chapter, we learned how to deserialize a JSON input with ArduinoJson. Here
are the key points to remember:
• JsonDocument:
– JsonDocument stores the memory representation of the document.
– StaticJsonDocument is a JsonDocument that resides in the stack.
– DynamicJsonDocument is a JsonDocument that resides in the heap.
– JsonDocument has a fixed capacity that you set at construction.
– You can use the ArduinoJson Assistant to compute the capacity.
• JsonArray and JsonObject:
– You can extract the value directly from the JsonDocument as long as there is
no ambiguity.
– To solve an ambiguity, you must call as<JsonArray>() or as<JsonObject>().
– JsonArray and JsonObject are references, not copies.
– The JsonDocument must remain in memory; otherwise, the JsonArray or the
JsonObject contains a dangling pointer.
• JsonVariant:
– JsonVariant is also a reference and supports several types: object, array,
integer, float, and boolean.
– JsonVariant differs from JsonDocument because it doesn’t own the memory;
it just points to it.
– JsonVariant supports implicit conversion, but you can also call as<T>().
• The two modes:
– The parser has two modes: zero-copy and classic.
– It uses the zero-copy mode when the input is a char*.
– It uses the classic mode with all other types.
Chapter 3 Deserialize with ArduinoJson 104
That was a free chapter from “Mastering ArduinoJson”; the book contains seven chap-
ters like this one. Here is what readers say:
I think the missing C++course and the troubleshooting chapter are worth
the money by itself. Very useful for C programming dinosaurs like myself.
— Doug Petican
The short C++section was a great refresher. The practical use of Arduino-
Json in small embedded processors was just what I needed for my home
automation work. Certainly worth having! Thank you for both the book
and the library. — Douglas S. Basberg
For a really reasonable price, not only you’ll learn new skills, but you’ll also be one of
the few people that contribute to sustainable open-source software. Yes, giving
money for free software is a political act!
The e-book comes in three formats: PDF, epub and mobi. If you purchase the e-book,
you get access to newer versions for free. A carefully edited paperback edition is
also available.
Ready to jump in?
Go to arduinojson.org/book and use the coupon code THIRTY to get a 30% discount.