1. What is Azure Event Hub?
Azure Event Hubs is a highly scalable event ingestion service, capable of processing millions of events per second with low latency and high reliability. Conceptually, Event Hubs can be thought of as a liaison between “event producers” and “event consumers” as depicted in the diagram below.
2. Key Concepts
Event Producer
An entity that sends data to an Event Hub (ingress). Event publishers can publish events using HTTPS or AMQP 1.0 or Apache Kafka (1.0 and above)
Event Consumer
An entity that reads data from an Event Hub (egress).
Capture
Capture is a feature that allows streaming data to be automatically stored in either Azure Blob Storage or Azure Data Lake.
Partitions
Partitions is a mechanism in which data can be organised, enabling consumers to only read a specific subset (or partition) of a stream. Paritions also enable downstream parallelism in scenarios where there are multiple consuming applications.
Partition Key
Used by publishers to map event data to a specific partition. If no key is specified, a round-robin assignment is used.
Consumer Groups
A consumer group is a view (state, position or offset) of an entire event hub. Consumer groups allow multiple consumers to have distinct views of an event stream (i.e. read at their own pace). Consumer groups can be thought of as a subscription to an event stream, if another application (i.e. consumer) would like to subscribe to the same stream but process the data differently, this would be an example of where an additional consumer group could be beneficial.
Note: Partitions can only be accessed via a consumer group (there is always a default in an event hub).
Throughput Units (TU)
Capacity is defined by selecting a number of Throughput Units (TU) during the initial creation of an Event Hub namespace. The number of throughput units applies to all event hubs within a namespace.
Each unit is entitled to:
Ingress (events sent into Event Hub): Up to 1 MB per second or 1,000 events per second (whichever comes first)
Egress (events sent out of Event Hub): Up to 2 MB per second
Shared Access Signatures (SAS)
Event publishers require a SAS token to identify and authenticate themselves to an event hub.
The following diagram illustrates the relationships between the core elements.
While it may be easier to conceptualise the high-level flow with single entities (i.e. Producer, Hub, Consumer), in reality, there can be multiple entities sending events to an event hub and by the same token, multiple consumers reading from an event stream.
Event Hub in Action:
1) Create an event hub using Azure portal:
Prerequisites
To complete this quickstart, make sure that you have:
- Azure subscription. If you don't have one, create a free account before you begin.
- Visual Studio 2017 Update 3 (version 15.3, 26730.01) or later.
- .NET Standard SDK, version 2.0 or later.
a) Create a resource group
A resource group is a logical collection of Azure resources. All resources are deployed and managed in a resource group. Do the following to create a resource group:
1. Sign in to the Azure portal.
2. In the left navigation, click Resource groups. Then click Add.
3. Type a unique name for the resource group. The system immediately checks to see if the name is available in the currently selected Azure subscription.
4. In Subscription, click the name of the Azure subscription in which you want to create the resource group.
5. Select a geographic location for the resource group.
6. Click Create.
b) Create an Event Hubs namespace
An Event Hubs namespace provides a unique scoping container, referenced by its fully qualified domain name, in which you create one or more event hubs. To create a namespace in your resource group using the portal, do the following actions:
1. In the Azure portal, and click Create a resource at the top left of the screen.
2. Click Internet of Things, and then click Event Hubs.
3. In Create namespace, enter a namespace name. The system immediately checks to see if the name is available.
4. After making sure the namespace name is available, choose the pricing tier (Basic or Standard). Also, choose an Azure subscription, resource group, and location in which to create the resource.
5. Click Create to create the namespace. You may have to wait a few minutes for the system to fully provision the resources.
6. Select Alerts, and then select the deployment with the same name as the name of event hub namespace.
7. Select your event hub namespace from the list of resources created in the deployment.
8. On the Event Hubs Namespace page, select Shared access policies, and then click RootManageSharedAccessKey.
9. Click the copy button to copy the RootManageSharedAccessKey connection string to the clipboard. Save this connection string in a temporary location, such as Notepad, to use later.
c) Create an event hub
To create an event hub within the namespace, do the following actions:
1. On the Event Hubs Namespaces page, select click Event Hubs.
2. At the top of the window, click + Event Hub.
3. Type a name for your event hub, then click Create.
Congratulations! You have used the portal to create an Event Hubs namespace, and an event hub within that namespace.
2) Sample code to send event Azure Event Hub using .NET Framework
GitHub Reference: https://github.com/shyamnarayan2001/DotNet_Projects/tree/master/AzureEventHub/Sender
GitHub Reference: https://github.com/shyamnarayan2001/DotNet_Projects/tree/master/AzureEventHub/Sender
a) Create a console application
In Visual Studio, create a new Visual C# Desktop App project using the Console Application project template. Name the project Sender.
b) Add the Event Hubs NuGet package
In Solution Explorer, right-click the Sender project, and then click Manage NuGet Packages for Solution.
Click the Browse tab, then search for WindowsAzure.ServiceBus. Click Install, and accept the terms of use.
Visual Studio downloads, installs, and adds a reference to the Azure Service Bus library NuGet package.
c) Write code to send messages to the event hub
1) Add the following using statements at the top of the Program.cs file:
using System.Threading;
using Microsoft.ServiceBus.Messaging;
using System.Threading;
using Microsoft.ServiceBus.Messaging;
2) Add the following fields to the Program class, substituting the placeholder values with the name of the event hub you created in the previous section, and the namespace-level connection string you saved previously. You can copy connection string for your event hub from Connection string-primary key under RootManageSharedAccessKey on the Event Hub page in the Azure portal. For detailed steps, see Get connection string.
static string eventHubName = "Your Event Hub name";
static string connectionString = "namespace connection string";
3) Add the following method to the Program class:static string eventHubName = "Your Event Hub name";
static string connectionString = "namespace connection string";
static void SendingRandomMessages()
{
var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
while (true)
{
try
{
var message = Guid.NewGuid().ToString();
Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message);
eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(message)));
}
catch (Exception exception)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);
Console.ResetColor();
}
Thread.Sleep(200);
}
}
This method continuously sends events to your event hub with a 200-ms delay.
4) Finally, add the following lines to the Main method:
Console.WriteLine("Press Ctrl-C to stop the sender process");
Console.WriteLine("Press Enter to start now");
Console.ReadLine(); SendingRandomMessages();
5) Run the program, and ensure that there are no errors.
Congratulations! You have now sent messages to an event hub. You can verify the messages in Azure Portal by going to your EventHub --> Overview tab, as shown below.
2) Sample code to receive events from Azure Event Hub using .NET Framework
GitHub Reference: https://github.com/shyamnarayan2001/DotNet_Projects/tree/master/AzureEventHub/Receiver
a) Create an Event Hubs namespace and an event hub:
The first step is to use the Azure portal to create a namespace of type Event Hubs, and obtain the management credentials your application needs to communicate with the event hub. To create a namespace and an event hub, follow the procedure in this article, then proceed with the following steps explained in the section 1 "Create an event hub using Azure portal" of this tutorial.
b) Create a storage account for Event Processor Host
The Event Processor Host is an intelligent agent that simplifies receiving events from Event Hubs by managing persistent checkpoints and parallel receives. For checkpointing, the Event Processor Host requires a storage account. The following example shows how to create a storage account and how to get its keys for access:
1. In the Azure portal, and select Create a resource at the top left of the screen.
2. Select Storage, then select Storage account - blob, file, table, queue.
3. On the Create storage account page, take the following steps:
Enter a name for the storage account.
Choose an Azure subscription that contains the event hub.
Select the resource group that has the event hub.
Select a location in which to create the resource.
Then click Review + create.
4. On the Review + create page, review the values, and select Create.
5. After you see the Deployments Succeeded message, select Go to resource at the top of the page. You can also launch the Storage Account page by selecting your storage account from the resource list.
6. In the Essentials window, select Blobs.
7. Select + Container at the top, enter a name for the container, and select OK.
8. Select Access keys in the left-side menu, and copy the value of key1.
Save the following values to Notepad or some other temporary location.
c) Create a console application
In Visual Studio, create a new Visual C# Desktop App project using the Console Application project template. Name the project Receiver.
Add the Event Hubs NuGet package
1. In Solution Explorer, right-click the Receiver project, and then click Manage NuGet Packages for Solution.
2. Click the Browse tab, then search for Microsoft Azure Service Bus Event Hub - EventProcessorHost. Click Install, and accept the terms of use.
Visual Studio downloads, installs, and adds a reference to the Azure Service Bus Event Hub - EventProcessorHost NuGet package, with all its dependencies.
Implement the IEventProcessor interface
1. Right-click the Receiver project, click Add, and then click Class. Name the new class SimpleEventProcessor, and then click Add to create the class.
2. Add the following statements at the top of the SimpleEventProcessor.cs file:
using Microsoft.ServiceBus.Messaging;
using System.Diagnostics;
3. Substitute the following code for the body of the class:
class SimpleEventProcessor : IEventProcessor
{
Stopwatch checkpointStopWatch;
async Task IEventProcessor.CloseAsync(PartitionContext context, CloseReason reason)
{
Console.WriteLine("Processor Shutting Down. Partition '{0}', Reason: '{1}'.", context.Lease.PartitionId, reason);
if (reason == CloseReason.Shutdown)
{
await context.CheckpointAsync();
}
}
Task IEventProcessor.OpenAsync(PartitionContext context)
{
Console.WriteLine("SimpleEventProcessor initialized. Partition: '{0}', Offset: '{1}'", context.Lease.PartitionId, context.Lease.Offset);
this.checkpointStopWatch = new Stopwatch();
this.checkpointStopWatch.Start();
return Task.FromResult<object>(null);
}
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (EventData eventData in messages)
{
string data = Encoding.UTF8.GetString(eventData.GetBytes());
Console.WriteLine(string.Format("Message received. Partition: '{0}', Data: '{1}'", context.Lease.PartitionId, data));
}
//Call checkpoint every 5 minutes, so that worker can resume processing from 5 minutes back if it restarts.
if (this.checkpointStopWatch.Elapsed > TimeSpan.FromMinutes(5))
{
await context.CheckpointAsync(); this.checkpointStopWatch.Restart();
}
}
}
This class is called by the EventProcessorHost to process events received from the event hub. The SimpleEventProcessor class uses a stopwatch to periodically call the checkpoint method on the EventProcessorHost context. This processing ensures that, if the receiver is restarted, it loses no more than five minutes of processing work.
4. Update the Main method to use SimpleEventProcessor
1. In the Program class, add the following using statement at the top of the file:
using Microsoft.ServiceBus.Messaging;
2. Replace the Main method in the Program class with the following code, substituting the event hub name and the namespace-level connection string that you saved previously, and the storage account and key that you copied in the previous sections.
static void Main(string[] args) {
string eventHubConnectionString = "{Event Hubs namespace connection string}";
string eventHubName = "{Event Hub name}";
string storageAccountName = "{storage account name}";
string storageAccountKey = "{storage account key}";
string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", storageAccountName, storageAccountKey);
string eventProcessorHostName = Guid.NewGuid().ToString();
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, EventHubConsumerGroup.DefaultGroupName, eventHubConnectionString, storageConnectionString);
Console.WriteLine("Registering EventProcessor...");
var options = new EventProcessorOptions();
options.ExceptionReceived += (sender, e) => { Console.WriteLine(e.Exception); };
eventProcessorHost.RegisterEventProcessorAsync<SimpleEventProcessor>(options).Wait();
Console.WriteLine("Receiving. Press enter key to stop worker.");
Console.ReadLine();
eventProcessorHost.UnregisterEventProcessorAsync().Wait();
}
3. Run the program, and ensure that there are no errors.
Congratulations! You have now received messages from an event hub using the Event Processor Host. You can verify the messages in Azure Portal by going to your EventHub --> Overview tab, as shown below, where you could see "Outgoing Messages" and also there is a downward curve in the "Requests" and Messages" which indicate that the messages are being consumed.
Also if you go to your storage account and click "Storage Explorer (preview)" on the left hand menu option, you will see the Event messages are stored in Blob Containers as Blob messages. The 0 and 1 indicate the Event Hub partitions.
Service Bus Explorer
In your journey to work with Azure Event Hubs or Service Bus or Relay or any other Azure messaging platform, you will intensively need Service Bus Explorer. It will act as a debugger tool for you. It is one of the best tools to verify and visualize the messages available. This a tool developed by Azure client experience team, you can find more details and download @Service Bus Explorer -
How to set up your service bus explorer,
1. Download the explorer from the link above.
2. Unzip the folder to a local drive.
3. Run ServiceBusExplorer.exe'
4. Click the "File" and select "Connect". Select Enter connection string from the drop-down. Leave the rest as default. Go to Azure SAS Policy as in step 8 above, copy the connection string and past in the box. Click OK. Also you can click "Save" button to save the input.
5. Now, you will see that all the entities create with your event hub namespace as in the screenshot below. Also, to view messages in each of the Partition, you will have to create a listener with each of the partitions. The listener will just read the message but it will not be removed from the Event Hub partition.
6. Partition Listener looks as below. Also, you have options to select the duration from when you want to view the message, refresh rate etc. Once configured, click the Start button. Select Event Tab as below, as soon as, you send a new message to the hub, it will be displayed in the tab. Below is the screen shot of Event Hub receiving messages depicted in a graph.
That's all to cover on this topic. Hope this was useful.
2) Sample code to receive events from Azure Event Hub using .NET Framework
GitHub Reference: https://github.com/shyamnarayan2001/DotNet_Projects/tree/master/AzureEventHub/Receiver
a) Create an Event Hubs namespace and an event hub:
The first step is to use the Azure portal to create a namespace of type Event Hubs, and obtain the management credentials your application needs to communicate with the event hub. To create a namespace and an event hub, follow the procedure in this article, then proceed with the following steps explained in the section 1 "Create an event hub using Azure portal" of this tutorial.
b) Create a storage account for Event Processor Host
The Event Processor Host is an intelligent agent that simplifies receiving events from Event Hubs by managing persistent checkpoints and parallel receives. For checkpointing, the Event Processor Host requires a storage account. The following example shows how to create a storage account and how to get its keys for access:
1. In the Azure portal, and select Create a resource at the top left of the screen.
2. Select Storage, then select Storage account - blob, file, table, queue.
3. On the Create storage account page, take the following steps:
Enter a name for the storage account.
Choose an Azure subscription that contains the event hub.
Select the resource group that has the event hub.
Select a location in which to create the resource.
Then click Review + create.
4. On the Review + create page, review the values, and select Create.
5. After you see the Deployments Succeeded message, select Go to resource at the top of the page. You can also launch the Storage Account page by selecting your storage account from the resource list.
6. In the Essentials window, select Blobs.
7. Select + Container at the top, enter a name for the container, and select OK.
8. Select Access keys in the left-side menu, and copy the value of key1.
Save the following values to Notepad or some other temporary location.
- Name of the storage account
- Access key for the storage account
- Name of the container
c) Create a console application
In Visual Studio, create a new Visual C# Desktop App project using the Console Application project template. Name the project Receiver.
Add the Event Hubs NuGet package
1. In Solution Explorer, right-click the Receiver project, and then click Manage NuGet Packages for Solution.
2. Click the Browse tab, then search for Microsoft Azure Service Bus Event Hub - EventProcessorHost. Click Install, and accept the terms of use.
Visual Studio downloads, installs, and adds a reference to the Azure Service Bus Event Hub - EventProcessorHost NuGet package, with all its dependencies.
Implement the IEventProcessor interface
1. Right-click the Receiver project, click Add, and then click Class. Name the new class SimpleEventProcessor, and then click Add to create the class.
2. Add the following statements at the top of the SimpleEventProcessor.cs file:
using Microsoft.ServiceBus.Messaging;
using System.Diagnostics;
3. Substitute the following code for the body of the class:
class SimpleEventProcessor : IEventProcessor
{
Stopwatch checkpointStopWatch;
async Task IEventProcessor.CloseAsync(PartitionContext context, CloseReason reason)
{
Console.WriteLine("Processor Shutting Down. Partition '{0}', Reason: '{1}'.", context.Lease.PartitionId, reason);
if (reason == CloseReason.Shutdown)
{
await context.CheckpointAsync();
}
}
Task IEventProcessor.OpenAsync(PartitionContext context)
{
Console.WriteLine("SimpleEventProcessor initialized. Partition: '{0}', Offset: '{1}'", context.Lease.PartitionId, context.Lease.Offset);
this.checkpointStopWatch = new Stopwatch();
this.checkpointStopWatch.Start();
return Task.FromResult<object>(null);
}
async Task IEventProcessor.ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach (EventData eventData in messages)
{
string data = Encoding.UTF8.GetString(eventData.GetBytes());
Console.WriteLine(string.Format("Message received. Partition: '{0}', Data: '{1}'", context.Lease.PartitionId, data));
}
//Call checkpoint every 5 minutes, so that worker can resume processing from 5 minutes back if it restarts.
if (this.checkpointStopWatch.Elapsed > TimeSpan.FromMinutes(5))
{
await context.CheckpointAsync(); this.checkpointStopWatch.Restart();
}
}
}
This class is called by the EventProcessorHost to process events received from the event hub. The SimpleEventProcessor class uses a stopwatch to periodically call the checkpoint method on the EventProcessorHost context. This processing ensures that, if the receiver is restarted, it loses no more than five minutes of processing work.
4. Update the Main method to use SimpleEventProcessor
1. In the Program class, add the following using statement at the top of the file:
using Microsoft.ServiceBus.Messaging;
2. Replace the Main method in the Program class with the following code, substituting the event hub name and the namespace-level connection string that you saved previously, and the storage account and key that you copied in the previous sections.
static void Main(string[] args) {
string eventHubConnectionString = "{Event Hubs namespace connection string}";
string eventHubName = "{Event Hub name}";
string storageAccountName = "{storage account name}";
string storageAccountKey = "{storage account key}";
string storageConnectionString = string.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", storageAccountName, storageAccountKey);
string eventProcessorHostName = Guid.NewGuid().ToString();
EventProcessorHost eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, EventHubConsumerGroup.DefaultGroupName, eventHubConnectionString, storageConnectionString);
Console.WriteLine("Registering EventProcessor...");
var options = new EventProcessorOptions();
options.ExceptionReceived += (sender, e) => { Console.WriteLine(e.Exception); };
eventProcessorHost.RegisterEventProcessorAsync<SimpleEventProcessor>(options).Wait();
Console.WriteLine("Receiving. Press enter key to stop worker.");
Console.ReadLine();
eventProcessorHost.UnregisterEventProcessorAsync().Wait();
}
3. Run the program, and ensure that there are no errors.
Congratulations! You have now received messages from an event hub using the Event Processor Host. You can verify the messages in Azure Portal by going to your EventHub --> Overview tab, as shown below, where you could see "Outgoing Messages" and also there is a downward curve in the "Requests" and Messages" which indicate that the messages are being consumed.
Also if you go to your storage account and click "Storage Explorer (preview)" on the left hand menu option, you will see the Event messages are stored in Blob Containers as Blob messages. The 0 and 1 indicate the Event Hub partitions.
Service Bus Explorer
In your journey to work with Azure Event Hubs or Service Bus or Relay or any other Azure messaging platform, you will intensively need Service Bus Explorer. It will act as a debugger tool for you. It is one of the best tools to verify and visualize the messages available. This a tool developed by Azure client experience team, you can find more details and download @Service Bus Explorer -
How to set up your service bus explorer,
1. Download the explorer from the link above.
2. Unzip the folder to a local drive.
3. Run ServiceBusExplorer.exe'
4. Click the "File" and select "Connect". Select Enter connection string from the drop-down. Leave the rest as default. Go to Azure SAS Policy as in step 8 above, copy the connection string and past in the box. Click OK. Also you can click "Save" button to save the input.
5. Now, you will see that all the entities create with your event hub namespace as in the screenshot below. Also, to view messages in each of the Partition, you will have to create a listener with each of the partitions. The listener will just read the message but it will not be removed from the Event Hub partition.
6. Partition Listener looks as below. Also, you have options to select the duration from when you want to view the message, refresh rate etc. Once configured, click the Start button. Select Event Tab as below, as soon as, you send a new message to the hub, it will be displayed in the tab. Below is the screen shot of Event Hub receiving messages depicted in a graph.
That's all to cover on this topic. Hope this was useful.
