Azure Functions: Merge Algorithm Runtime Comparison (Part 2)

Introduction

This is the second part of a three part series on algorithmic runtime comparison. For the full intro to the series check out the first post.

Today we will convert an existing HTTP Trigger into a ServiceBusTopic Trigger and use it to pull a message from ServiceBus, write the output of our merge algorithm to Azure Blob Storage and write the performance data to Azure Table Storage.

Prerequisites

The source for today's demo is located on and deployed from GitHub. If you want to save some copy/pasting grab it here: AzureFunctionsBlogDemos. Feel free to clone the repository and use it for a reference as we go through the demo.
Today's demo assumes that you have completed the demo for part one of the series. If you haven't already, please complete the first demo now.

Convert the HttpTrigger to ServiceBusTopic Trigger

We have already seen WQUWPCTopicTrigger.cs but there are some key changes. Let's touch on those really quickly.

First, rather than expecting an HTTP Request the Trigger is now processing a message from the ServiceBusTopic. This is the first parameter of the function and is bound via function.json which we will touch on momentarily.
Second, since we have the MergePerformance object we created with the last post we will use that to write to Table Storage.
Third, because of the size of the arrays we can't write them to Table Storage. They're just too big and will blow out the maximum column size of 64kb. So, we will create a file and write the file to Blob Storage. This parameter is the IBinder param.

From there, the code is relatively straightforward to read. If you have any questions I highly recommend attaching the debugger in Visual Studio and walking through the Trigger.

WQUWPCTopicTrigger.cs

  
using Microsoft.Azure.WebJobs;  
using Microsoft.Azure.WebJobs.Host;  
using System;  
using System.Diagnostics;  
using System.IO;  
using System.Security.Cryptography;

namespace AzureFunctionsBlogDemos.Merging  
{
    public class WQUWPCTopicTrigger
    {

        public static void Run(MergingArray myQueueItem, TraceWriter log, IAsyncCollector outputTable,
            IBinder binder)
        {
            log.Info("WQUWPCTopicTrigger processed a request.");

            Stopwatch stopwatch = new Stopwatch();
            stopwatch.Start();

            MergingArray.Merge(myQueueItem, Shared.Enums.MergeAlgorithms.WeightedQuickUnionWithPathCompression);

            stopwatch.Stop();

            var performance = new MergePerformance();
            performance.Runtime = stopwatch.Elapsed;
            performance.AlgorithmName = "WeightedQuickUnionWithPathCompression";
            performance.PartitionKey = "WeightedQuickUnionWithPathCompression";
            performance.RowKey = Guid.NewGuid().ToString();
            performance.NumberOfElements = myQueueItem.Output.Length;

            outputTable.AddAsync(performance);

            var blobPath = "merging" + "/" + "wquwpctopictrigger" + DateTime.UtcNow.ToString("yyyyMMddHHmmss") + ".txt";

            using (var outputBlob = binder.Bind(
                new BlobAttribute(blobPath)))
            {
                outputBlob.WriteLine($"Number to Union From: {string.Join(",", myQueueItem.NumberToUnionFrom)}");
                outputBlob.WriteLine($"Number to Union To: {string.Join(",", myQueueItem.NumberToUnionTo)}");
                outputBlob.WriteLine($"Output of Merge: {string.Join(",", myQueueItem.Output)}");
                outputBlob.WriteLine();
                outputBlob.WriteLine($"Runtime: {performance.Runtime.ToString()}");

                // create Sha1 Hash
                var sha = new SHA512CryptoServiceProvider();
                // This is one implementation of the abstract class SHA512.
                var result = sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes(blobPath + myQueueItem.NumberToUnionFrom + myQueueItem.NumberToUnionTo + myQueueItem.Output));
                outputBlob.WriteLine($"Hash of Inputs, Output and Runtime: {BitConverter.ToString(result).Replace("-", "")}");
            }
        }
    }
}

Let's go over the binding changes and then take a look at how they work:

As I mentioned earlier, the function now takes in a message from the ServiceBus Topic rather than an HTTP Request. Unfortunately, you can't have two input parameters (which is actually a good architectural decision to be honest) so if you want to keep the HTTP Trigger you will need to create a separate trigger.
Let's take a look at the ServiceBusTrigger bindings first. The "name" binds our message to the parameter used in code. "topicName" binds to which topic we will use. "subscriptionName" binds to which subscription we are are going to read our message from. The "connection" is the same connection that we set up in the last post. Lastly, "type" and "direction" are pretty self-explanatory.
The elements for Table storage are actually quite simple so I won't touch on them.
The elements for Blob storage are also all simple other than "path." For "path" we are actually setting this value on like 34 of the code. So, essentially this value is just a placeholder and we never use it.

function.json

  
{
  "bindings": [
    {
      "name": "myQueueItem",
      "topicName": "algorithmsmerge",
      "subscriptionName": "WeightedQuickUnionWithPathCompression",
      "connection": "AzureWebJobsServiceBus",
      "accessRights": "manage",
      "type": "serviceBusTrigger",
      "direction": "in"
    },
    {
      "connection": "AzureWebJobsStorage",
      "direction": "out",
      "name": "outputTable",
      "tableName": "Merging",
      "type": "table"
    },
    {
      "connection": "AzureWebJobsStorage",
      "direction": "out",
      "name": "outputBlob",
      "path": "Merging/{QuickUnionTopicTrigger}.txt",
      "type": "blob"
    }
  ],
  "disabled": false,
  "entryPoint": "AzureFunctionsBlogDemos.Merging.WQUWPCTopicTrigger.Run",
  "scriptFile": "..\\bin\\AzureFunctionsBlogDemos.dll"
}

Run and Test WQUWPCTopicTrigger

Now we should be able to hit F5 and run our new WQUWPCTopicTrigger which will read our messages from our ServiceBus Topic, run the merge algorithm and write the output to Blob and Table storage.

In Visual Studio, set a breakpoint so you we can walk through the Function.

If you've looked at the GitHub repo - AzureFunctionsBlogDemos you may have noticed that I wrote a little Console App and included it with the Solution. This is used to send in HTTP Requests to our Merge Trigger.

Let's take a look at that really quick. Currently I have it set to send 100 requests to an HTTP endpoint.

Program.cs

  
using System;  
using System.Configuration;  
using System.Net.Http;  
using System.Net.Http.Headers;  
using System.Threading.Tasks;

// Client code adapted from the following example: https://docs.microsoft.com/en-us/aspnet/web-api/overview/advanced/calling-a-web-api-from-a-net-client
namespace HttpClientSample  
{
    class Program
    {
        static HttpClient client = new HttpClient();


        static void Main()
        {
            RunAsync().Wait();
        }

        static async Task RunAsync()
        {
            var debug = bool.Parse(ConfigurationManager.AppSettings["Debug"]);
            string baseUrl = debug == true ? ConfigurationManager.AppSettings["DebugBaseUrl"] : ConfigurationManager.AppSettings["BaseUrl"];

            client.BaseAddress = new Uri(baseUrl);
            client.DefaultRequestHeaders.Accept.Clear();
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

            var numberOfRequests = 100;

            for (int i = 0; i < numberOfRequests; i++)
            {

                try
                {
                    // Create a new Array
                    int arraySize = 30000;
                    var inputArray = new AzureFunctionsBlogDemos.Merging.MergingArray();
                    inputArray.NumberToUnionFrom = CreateIntegers(arraySize);
                    inputArray.NumberToUnionTo = CreateIntegers(arraySize);

                    var url = await CreateProductAsync(inputArray);
                    Console.WriteLine($"Created at {url}");

                }
                catch (Exception e)
                {
                    Console.WriteLine(e.Message);
                }
            }

            Console.ReadLine();
        }

        static async Task CreateProductAsync(AzureFunctionsBlogDemos.Merging.MergingArray inputArray)
        {
            var routeAndKey = new Uri(client.BaseAddress + "api/MergeTrigger?code=" + ConfigurationManager.AppSettings["ApiKey"]);

            HttpResponseMessage response = await client.PostAsJsonAsync(routeAndKey, inputArray);
            response.EnsureSuccessStatusCode();

            // return URI of the created resource.
            return response.RequestMessage.RequestUri;
        }

        private static int[] CreateIntegers(int input)
        {
            int[] returnArray = new int[input];

            Random r = new Random();

            for (int i = 0; i < input; i++)
            {
                int number = r.Next(0, input * 4);

                returnArray[i] = number;
            }

            return returnArray;
        }
    }
}

Let's take a look at the app.config file:

<?xml version="1.0" encoding="utf-8"?>  
<configuration>  
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2"/>
    </startup>
  <appSettings file="secretappsettings.config">
  </appSettings>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral"/>
        <bindingRedirect oldVersion="0.0.0.0-10.0.0.0" newVersion="10.0.0.0"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

Nothing really interesting here except for a reference to secretappsettings.config. Note: with the secretappsettings.config add a reference to the file in your .gitignore file so that you don't accidentally check it in to your repository. Otherwise, you may expose some secrets to the world.

secretappsettings.config

<appSettings>  
  <add key="BaseUrl" value="https://keithblogfunctiondemo.azurewebsites.net/" />
  <add key="ApiKey" value="==" />
  <add key="Debug" value="true"/>
  <add key="DebugBaseUrl" value="http://localhost:7071/"/>
</appSettings>

There, all you need to do in order to test your functions is set the Debug value, add your ApiKey and update the BaseUrl values.

Start up a debug instance of your Functions project. Next, start up a debug instance of the Console App.

Debug Session

This will send in 100 requests. You don't have to debug the whole thing if you don't want to.

More Debugging

Commit to Git

Now, we're ready to make a commit from Powershell which will trigger another deployment.

  
    git add -A
    git commit -m "Added WeightedQuickUnionWithPathCompression as a ServiceBus Trigger."
    git push origin master

It should take a minute or so and then your function will be visible. Grab the URL just like we did yesterday. Can now use the console app to send in 100 requests to MergeTrigger which will create 100 messages for the WQUWPC Trigger.

Note: switch the value for Debug in secretappsettings.config back to true. Otherwise we will be sending to the wrong endpoint.

Postman Sending in Requests

Using Azure Storage Explorer

If you haven't already, take a moment to download Azure Storage Explorer. Authenticate to your Azure subscription and then you can drill into Subscription --> Storage Accounts and then select the Storage Account you're using for your Functions.

When I select the "Merging Table", this is set in the "function.json" file under the "tablename" key, I see the following:

Merging Table in Azure Storage Explorer

I can go to the "merging" blob using Azure Storage Explorer and take a look at the merge output we wrote to the blob:

Looking at the blob output

I can even download one of the txt files from the blob and check it out using Notepad++.

Inspecting WQUWPC Output

Conclusion

Today we altered our HTTP Trigger for WeightedQuickUnionWithPathCompression (WQUWPC) to use Azure ServiceBus Topics as its input rather than an HTTP Trigger. We debugged the function locally and were able to see that we can run it smoothly locally. We deployed the function using CI/CD from GitHub. Lastly, we ran the same program against our deployed function to ensure it behaved the same in our Azure Function app and inspected the output.

What's next:

After we alter our HTTP Trigger to be a ServiceBus Trigger we will add QuickFind, QuickUnion and WeightedQuickUnion which will enable us to discuss the performance and complexity of the algorithms.