PGP in Azure Storage and Batch Service

This article will demonstrate how to PGP encrypt an Azure Blob Storage file and how to run that operation as Batch Service on the Azure Cloud.

Contents

  1. Encrypting Blob from Azure Storage Container
  2. Azure Batch Service that performs PGP cryptography
  3. Running Batch Service from Function App

Encrypt Blob from Azure Storage Container

This example is a pure Console Application that PGP encrypts a Blob located in Azure Storage Container.

In order to access the Storage, we need its Connection String. Then we just simply read the input Blob as Stream and write to the output Blob:

string connectionString = "...";
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);
 
CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient();
 
CloudBlobContainer containerSrc = blobClient.GetContainerReference("data");
CloudBlob blobSrc = containerSrc.GetBlobReference(sourceBlob); // source file name
 
CloudBlobContainer containerDest = blobClient.GetContainerReference("dataoutput");
CloudBlockBlob blobDest = containerDest.GetBlockBlobReference(destinationBlob);
 
String inlinePublicKey = "...";
 
using (Stream input = blobSrc.OpenRead())
using (Stream output = blobDest.OpenWrite())
{
  DidiSoft.Pgp.PGPLib pgp = new DidiSoft.Pgp.PGPLib();
 
  bool asciiArmorOutput = false;
  pgp.EncryptStream(input, sourceBlob, new MemoryStream(System.Text.Encoding.UTF8.GetBytes(inlinePublicKey)), output, asciiArmorOutput);
}

The source code for this Console Application is available on GitHub

Azure Batch Service that performs PGP cryptography

Azure Batch Service is suitable for long-running tasks and is an affordable alternative to Logical Apps.

First, we will create a batch service ‘didisoftbatch‘ from the Azure Portal.

Running a Batch Service programmatically requires creating :

  1. Pool with Virtual Machines
  2. CloudJob that will run inside the pool
  3. CloudTask that will be executed as part of batch job

Cloud tasks can execute applications registered in the Batch Service (menu Applications from the sidebar). Here it is good to know which version of .NET Framework is available.  For Windows 2019 we will use .NET Framework 4.7. Then we should ZIP the Release folder of the application (In our case Conso0le Application EncryptBlobPgp with the executable file EncryptBlobPgp.exe)

The possible types of Virtual Machines can be seen if we try to create a Pool manually from the sidebar (just to see what are the possible values and the we can cancel the operation)

In our example, we selected 2019-datacenter

ImageReference imageReference = new ImageReference(
                                     publisher: "MicrosoftWindowsServer",
                                     offer: "WindowsServer",
                                     sku: "2019-Datacenter",
                                     version: "latest");

// Create a VM configuration
VirtualMachineConfiguration vmConfiguration =
new VirtualMachineConfiguration(imageReference: imageReference,
                                 nodeAgentSkuId: "batch.node.windows amd64");

Then we have to create the CloudJob

CloudJob unboundJob = batchClient.JobOperations.CreateJob();
unboundJob.Id = jobId;
string appId = "EncryptBlobPgp";
string appVersion = "1.0";
 
// For this job, ask the Batch service to automatically create a pool of VMs when the job is submitted.
unboundJob.PoolInformation = new PoolInformation()
{
 AutoPoolSpecification = new AutoPoolSpecification()
 {
  AutoPoolIdPrefix = "EncryptBlobPgp",
  PoolSpecification = new PoolSpecification()
 {
  TargetDedicatedComputeNodes = 1,
  VirtualMachineSize = "Standard_A2_v2",
  VirtualMachineConfiguration = vmConfiguration
 },
 KeepAlive = false,
 PoolLifetimeOption = PoolLifetimeOption.Job
 }
};
// Commit Job to create it in the service
unboundJob.Commit();

And finally, we have to execute the actual application that has already been registered for the Batch Service.

Here it is interesting that we specify the application that we need in the CloudTask’s ApplicationPackageReferences. Then the actual path to the console application will be available as an environment variable that starts with AZ_BATCH_APP_PACKAGE_  but differs on Windows and Linux:

On Windows, it is uppercase and version delimited by “#”  %AZ_BATCH_APP_PACKAGE_ENCRYPTBLOBPGP#2.7
On Linux, it is a normal case and version delimited by “_” AZ_BATCH_APP_PACKAGE_EncryptBlobPgp_2_7

string appPath = String.Format("%AZ_BATCH_APP_PACKAGE_{0}#{1}%", "EncryptBlobPgp".ToUpper(), "1.0");
string cmdLine = String.Format("cmd /c {0}\\EncryptBlobPgp.exe {1} {2} {3} {4}",
      appPath, sourceContainer, sourceBlob, destinationContainer, destinationBlob);
 
CloudTask cloudTask = new CloudTask("task-encrypt", cmdLine)
{
 ApplicationPackageReferences = new List<ApplicationPackageReference>
 {
  new ApplicationPackageReference
  {
   ApplicationId = appId,
   Version = appVersion
  }
 }
};
await batchClient.JobOperations.AddTaskAsync(jobId, cloudTask);

Source code for the complete application that starts a Batch Service Job can be found here https://github.com/didisoft/Azure-Pgp-Function-App/tree/main/BatchService.

Running Batch Service from Function App

When processing large files with Function Apps it is a good idea to leverage this task to a Batch Service.

In our example, the Batch Service Job is started by the Function App that listens for new Blobs. Then when the Batch Service ends and the PGP encrypted output Blob is written a new Blob Trigger will run and stop the Batch Job (the Bath Job name is the output file name and this is the way that the Blob Trigger that listens for the output Blob knows which Batch Service Job to stop!).   Here are links to both Blob Trigger apps:

Starting Batch Job – https://github.com/didisoft/Azure-Pgp-Function-App/tree/main/DidiSoftBlobFunction
Ending Batch Job – https://github.com/didisoft/Azure-Pgp-Function-App/tree/main/DidiSoftBlobDeleteFunction

Summary

This tutorial illustrates how to encrypt in OpenPGP format Blobs located at Azure Storage and also how to start a Bath Service Job that executes that operation.

This is all suitable for processing large files, which exceed the running time of a Function App (e.g. Blob or Http Trigger).

The examples can be changed to perform other PGP operations, not just encrypting,  in order to suit your own application use cases.