Recently, I had to set up the Azure Health check service for an application protected with the Microsoft OAuth2 authentication provider, and the configuration process made me wish for some more basic level information on the setup. After repeating this setup a couple of times for different apps, I realize how astonishingly simple this configuration is, indeed. Still I feel that a more detailed walkthrough could help rookie Azure developers to avoid a few possible pitfalls. So this is a post for beginners in Azure development who want to use the Azure Health check service.
Health check service is well explained in an article on Microsoft Learn portal, but in a nutshell, this service calls a dedicated endpoint of your Azure app to make sure that it's up and running. Collected app availability information is stored in application insights for analysis and can be used to generate notifications, but what is even more exciting, based on the health status, the app service can remove an unhealthy app instance from a multi-instance deployment or attempt to replace the instance.
Sounds cool. But on the other hand, this means that the health status function must be the most robust part of the app and reliably reflect its state. We don't want the app service to start disabling healthy instances just because the health check endpoint throws an occasional exception, whilst the app is perfectly capable of servicing client requests.
This function is not provided by Azure functions extensions - this is something we need to write ourselves, so we need to find the right level of the complexity of the medical checks we perform on the app.
Health check function can verify every connection and availability of all related resources, but it can also be as simple as one-liner sending the 200 status response to every request, so the app state is deemed unhealthy only if it is completely unreachable for clients. It is recommended for the health check function to verify availability of resources vital for the app functioning, like the database connection, for instance.
In my example i will use a simple Azure function app, which reads a blob storage and returns a list of blobs in a container. Full application code is my GitHub repository, although it is not much larger that this function.
In this example I do not require function-level authorisation allowing anonymous access because I am going to enable the application-level OAuth2 authentication.
ListBlobs is the only function that exposes app functionality to its consumers. Now let's add another function that will serve as the health check endpoint.
To check the app health status, I will be sending a request to get account properties and return the 200 response if the account reachable and the properties are successfully retrieved. The response from the storage account API is not important, as long as it does not fail. Similarly, I don't write intricate exception handling - any response outside of 2xx range will simply mark the app instance as "degraded".
The account name that this function verifies is stored in an environment variable DefaultStorageAccount. This may be not quite consistent with the ListBlobs function which receives the account name as its parameter, but will work as an illustration of the concept.
Once the health check function is published, we need to enable the service and set the endpoint which will be queried for the health status. Remember that the default HTTP route of a function includes the "api" prefix, unless it is customized, so the endpoint path for my health check function is /api/healthcheck
As an alternative to setting the account name in environment variables, the health check endpoint can include parameters, similar to the main ListBlobs function, just keep in mind that the length of the endpoint path is limited to 64 characters. Besides, if I want to configure the resources whose health I want to verify during the health check, I would rather prefer to keep this configuration in the app settings anyway.
As you can see in the screenshot, the health service notifies me that the fact that my app is single instance does not allow me to use all its benefits, and the load balancer setup does not apply here. Load balancer will not be able to disable a degraded instance and rebalance incoming requests when this instance is the only one available.
This is all it takes to enable the service in its basic configuration. But let's now enable the OAuth2 authentication for the app and see how the health check works with this configuration. Health check function is a public endpoint and must follow the same authentication/authorisation process as any other client request.
So I've enabled an authentication provider for the app and restricted access to authenticated requests only.
What about the health check, how will it behave now?
The only thing that Microsoft Learn article tells about the health check with authentication is this:
Health check integrates with App Service's authentication and authorization features. No other settings are required if these security features are enabled.
This is amazing! But how does the health check service obtain an OAuth2 token to access the function?
In fact, this service follows the client credentials OAuth flow and uses the client secret which is automatically created when the authentication provider is enabled. As you can see in the previous screenshot, my authentication provider is linked to an app registration AzHealthCheckDemo which I created for this demo. But I did not create any app secrets. If I now review the client secrets for this app, I will find one generated by the application service.
This secret has an expiration date 10 years from now. And the secret value is now saved in the application environment variables.
This is something to be aware of when you configure authentication for an Azure app. It may be a good idea to limit the app secret's life span and save it in a key vault.
This is what I do next - create a new secret in a key vault to store the authentication app secret value and refer to it from the function app configuration.
In the environment variables of my function app, I keep the variable name MICROSOFT_PROVIDER_AUTHENTICATION_SECRET, but change the value. Instead of keeping the secret directly in the variable, I add a key vault reference:
@Microsoft.KeyVault(VaultName=bcblogdemo;SecretName=AzHealthCheckDemo)
One remaining piece of configuration is allowing the app to access the key vault and read the secret. With managed identity enabled, the application needs the Key Vault Secret User role to retrieve the secret value. The Storage Blob Data Contributor role allows the app to read and write blobs in the storage account.
My implementation of the health check function that reads account properties requires higher privileges and needs the Storage Account Contributor role. And this is another consideration to keep in mind in practical implementations: extending app permissions on the storage account just for the status check may not be the best idea of the health check.
And that's it. Now the Overview blade of my function app shows its status.
To view retrospective health data, I can click on the Metrics link in the Health check blade. This link simply opens the predefined metric, which I can as well set up directly in the application metrics under the Monitoring section.
Once a minute the application service sends a request on the health check endpoint and saves the response as a success or failure. Average success rate aggregated over time will be displayed in a graph that can be pinned to a dashboard. The graph below shows another one of my apps that was disabled with some occasional uptime intervals until I fixed all issues and reenabled it.
This graph is based on the app health data collected in a custom metric which I can find in the application logs. KQL query on the customMetrics table returns all underlying entries, including the success and failure rates, and the duration of the health check procedure.
And this is all it takes to set up this very helpful Azure service for my apps, even if the app does not allow unauthenticated requests. Although, despite the fact that the service does indeed work with OAuth2 authentication without any additional configuration, I still prefer to move the app secret from environment variables to a key vault to keep it safe.