Exchange Server Throttled by Back Pressure Due to Internal Message

You may experience issues resulting in mail failing to be delivered to internal users. This may be difficult to detect using the common tactics for Exchange management.

In the situation experienced, an internal user had sent an email over 1.5GB in size. Normally this wouldn’t be a problem but due to a mis-configuration the internal receive connectors were set to accept messages up to 2047MB, which is also the maximum message size limit. This resulted in the exchange service attempting to receive the message rather than generating an NDR response, which would have stopped the message delivery from being retried.

This resulted in the Transport service being put under pressure and the incoming queue being throttled by back pressure, with no messages being accepted or delivered. If left alone the problem would not auto-resolve unlike a back pressure issue caused by legitimate email load.

Exchange Mail Flow

To explain how this happened we need to understand how the transport service works for delivering all email. Microsoft provide the following mail flow diagram to show how this works

Exchange Server 2016 Mail Flow Diagram

No matter where an email originates it needs to traverse the transport service to be delivered. External messages are typically what an admin deals with. These come into the front-end service as an SMTP message. They are checked to see whether they are authorised before being transferred to the transport service and then forwarded into the mailbox delivery service for delivery.

Messages sent by internal users will also transit the transport service but these take a slightly different route. The client does not send the message via SMTP but rather puts the message into the outbox folder for the mailbox. From here the message is sent by the store driver submit process which submits the message to the transport service. This then processes the message and sends it back to the store driver to deliver the message

Back Pressure

An added complication to this process is back pressure detection.

Exchange Server detects pressure on the transport service and starts to reduce the speed that messages are accepted to ensure that the server remains operational. Even though emails may be delivered at a slower rate the server remains operational in this situation.

This back pressure is detected based on several metrics. The easiest way to see the current state of these metrics is to run the powershell command:

[xml]$bp=Get-ExchangeDiagnosticInfo [-Server <enter-Exchange-server-name-here> ] -Process EdgeTransport -Component ResourceThrottling; $bp.Diagnostics.Components.ResourceThrottling.ResourceTracker.ResourceMeter

Most of these counters are fairly self explanatory and relate to free disk and memory on the server but one that may be new to you is UsedVersionBuckets. This is the number of uncommitted message queue database transactions in memory. So what does this mean?

When a message is being received the exchange server will be receiving it into memory. Once the complete message is received then it can save it into the mail queue database. While the message is still in memory though the UsedVersionBuckets will increase. This can happen either when the server is receiving many small messages or a small number of very big messages.

How large messages impact the transport service

In this case a single very large message was causing this pressure. Every time the message was submitted by the Store Driver it would result in the UsedVersionBuckets soaring to over 3000. At this point the transport service stopped processing any messages into the mailbox submission queue and stalled. While the service could be restarted once the message was resubmitted the same behaviour was repeated.

Typically advise includes looking at the message queues to see how many messages are being received and how big they are, but in this case the queues could even be clear. The message hasn’t made it to the queue yet as it’s still being transferred by the store driver.

In order to find this message you will need to perform a system wide search of the outbox folder of all mailboxes using the powershell command

Get-mailbox -Resultsize Unlimited | get-mailboxfolderstatistics -folderscope outbox | fl |ft identity,ItemsinFolder,FolderSize

Once the offending message is found it needs to be removed from the client side to make sure that it doesn’t get resubmitted to the store driver. There is no way to do this from the server side.

Configure Hybrid Public Folder with Exchange 2013/2016 (aka Modern Public Folders)

Public Folders don’t seem to have the usage that they used to so it’s been a while since we worked with Public Folders in Exchange. So long in fact that what we last configured is now called Legacy Public Folders with the new version, introduced in Exchange 2013 called Modern Public Folders.

A Refresher on Exchange Public Folders

In order to understand the new process of setting up Hybrid mode with Exchange Online you first need to understand some changes to how Public Folders work.

In Exchange 2010 public folders were stored in dedicated Public Folder Databases. These also had their own log files and had to be managed independently of any User Mailbox Databases.

With Modern Public Folders they have been moved into Mailboxes which are stored in a standard user database. The environment can contain multiple public folder mailboxes, each of which can contain different parts of the public folder hierarchy.

When a user accesses a public folder they are actually opening the mailbox that contains that part of the hierarchy. Unlike previous versions the data is only accessible from the server hosting the active database rather than any server hosting a public folder replica.

Configuring Hybrid Public Folders

What does this mean for configuring Hybrid mode Public Folders?

First of all if you searched for something like “Configure Exchange Public Folder Hybrid” and found this Exchange 2019 article referring to Exchange 2010 SP3 or later then you’ve got the wrong article. You need to look for this article which is only on the Exchange Online documents site.

This newer article ignores all of the steps setting up new Public Folder Mailboxes resulting in just three steps:

1) Download the following files from Mail-enabled Public Folders – directory sync script

  • Sync-MailPublicFolders.ps1
  • SyncMailPublicFolders.strings.psd1

2) On Exchange Server, run the following command to synchronize mail-enabled public folders from your local on-premises Active Directory to O365.

Sync-MailPublicFolders.ps1 -Credential (Get-Credential) -CsvSummaryFile:sync_summary.csv

3) Enable the exchange online organization to access the on-premises public folders. You will point to all of you on-premises public folder mailboxes.

Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes PFMailbox1,PFMailbox2,PFMailbox3

Issues When Configuring Hybrid Mode

There are a few things to be aware of with this process though, particularly the final step.

1) Remember that the new Public Folders are stored in User Mailboxes which are associated with AD user accounts. If you aren’t syncing your entire Active Directory forest then the Public Folder Mailbox objects may not be synced to Exchange Online. So where are these stored by default? Well the Users container in your exchange enabled domain of course.

It’s likely that you haven’t synced this but you CAN move these objects to an OU that is being synced without any impact. Unfortunately this requirement isn’t included in the documentation. If these objects aren’t synced to Exchange Online then you’ll get the following message

Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes PFMailbox1
Couldn't find object "PFMailbox1". Please make sure that it was spelled correctly or specify a different object.
    + CategoryInfo          : NotSpecified: (:) [Set-OrganizationConfig], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=SYAPR01MB2717,RequestId=d79eaa00-ff32-4076-8791-54ba22e3cb76,TimeStamp=26/11/201
   8 7:13:26 AM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] C4302B7C,Microsoft.Exchange.Management.Sy
    + PSComputerName        :

2) Once you’ve moved the public folder mailbox objects remember that the -RemotePublicFolderMailboxes PFMailbox1,PFMailbox2,PFMailbox3 syntax is referring to the Public Folder Mailboxes and NOT the public folder names. You can find these in the ECP under Public Folder Mailboxes.

3) You also need to list all public folder mailboxes in the one command. If you add an additional public folder mailbox in the future then include all the mailboxes and not just the new one.

4) Finally remember that your on-premises address book is different from your online address book. This means that any new mail enabled public folders will only appear in your online address book if you sync them using the Sync-MailPublicFolders.ps1 script. If users can create these objects then you may want to think about scheduling this task.

Only users who have been created on-premises and migrated to Exchange Online can access the on-premises Public Folder store. Only these users exist in the on-premises address book used to authenticate access.

It may not seem that way but ultimately this is a simple service to configure with just a few little gotchas to be aware of.