I’ve often had questions regarding how the self throttling in BizTalk works, this is actually one of my favorite parts about BizTalk, and I do find it strange that I never blogged about it before. Microsoft BizTalk is an amazing tool to interact between applications, databases or via web. BizTalk have many uses, and are extremely adaptable.

You can download the entire article as a PDF document.
BizTalk Self Throttling.

 Compared to the other similar products on the marked Microsoft BizTalk actually provides a better standard, at least in my opinion (I’m not payed by Microsoft to say this), with many adapters and different pipelines BizTalk can be used for everyone to simplify the communication between systems.

But over to the main reason why I’m writing this blog post. BizTalk is some sort of artificial intelligence (AI) to be exact, by default BizTalk reads the activity and changes settings accordingly. When everything is running smooth BizTalk acts like everything is normal. But when an abnormality happen, lets say you start to get four times the amount of messages then your used to, BizTalk act on this and perform actions to stabalize the environment. BizTalk has two kind of ways to work it is publishing (writing data to the database) and delivery (delivers the message either from a send port or orchestration). With these two steps BizTalk looks for abnormality and does the desired tuning accordingly. If you have 20 hosts instances on one server all instances have dedicated resources by default you can change these settings by performing manual throttling to the hosts or you can depend on BizTalk to do it for you, I remember we hade a course with John Callaway from Quicklearn some years ago and John showed us the settings for host throttling and said “Never touch these settings”. My recommendation is the same unless you really know what you’re doing. this guide may show you the way.

Microsoft describes the self throttling like this “The BizTalk Server host throttling mechanism continually monitors for a throttling condition, calculates the severity of the throttling condition, and applies host throttling progressively depending on the calculated severity. The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server processing scenarios. BizTalk Server host throttling exposes several configurable options that can be used to tune throttling for specific scenarios.” in my opinion this is pretty amazing.

According to the severity of the throttling conditions a few statements are published in the performance monitor on the server. Depending if it is publishing or delivery throttling. Both states are found in Performance Monitor under BizTalk:Message Agent

All the states have different meanings and possible fixes.

Here are the following states described including how to monitor it.

 Message Publishing throttling state:

2 – Incoming rate exceeds outgoing rate
This will happen if the incoming message rate exceeds the message outgoing rate, and the database experience problems with coping with the increasing amount of messages being published to the BizTalk databases. In order for BizTalk to deal with this problems it simply blocks the publishing of messages until incoming rate is at par with the message outgoing rate. You can view this scenario in real-time by using the performance monitor and look at the counters in described in the table below;

Counter Sub counter
BizTalk:Message Agent Message publishing incoming rate
BizTalk:Message Agent message publishing outgoing rate

4 – memory exceeds specified threshold
This may happen if you have a steep increment of memory use in for example large batch files. BizTalk will automatically reduce the size of the EPM thread pool, and stops any creation of new threads, if the batch requires a lot of resources it will actually slow down the process in order to maintain the availability for the rest of the environment. Sometimes the message if it is to big and including too many messages may be blocked out completely by BizTalk. In order to monitor this you need to open the following views in Performance Monitor;

Counter Sub counter
BizTalk:Message Agent High process memory
BizTalk:Message Agent process memory usage (MB)
BizTalk:Message Agent process memory usage threshold (MB)

5-  System memory exceeds threshold
This state couldnt be described any better, the system memory exceeds the specified threshold and simply stops the EPM to start processing new messages and reduce the current EPM thread count, if the reason for this state is because a steep memory usage of a batch file it will take longer to finish it, and if it is too big it will be dropped by BizTalk. use the following counters too see this issue in Performance Monitor

Counter Sub counter
BizTalk:Message Agent Physical memory usage threshold (MB)
BizTalk:Message Agent Physical memory usage (MB)
You can download the entire article as a PDF document.
BizTalk Self Throttling.

6 – Host message size
This is ony amazing state, where im currently stationed I look for this all the time, it basically says that the spool table or the tracking tables exceeds the threshold. The reason I look at this all the time is that we often have the spool table up to 60.000 rows. Still the environment coops good without any issue yet. The reason for this maybe that some of your SQL jobs have stopped, tracking host not running or the number of suspended messages is high. This state will as the one above create the same scenario as above, it will reduce the threads for EPM, it also blocks the EPM to process any new messages and creates a publishing delay for all messages.  In this scenario there are sevral counters to monitor

Counter Sub counter
BizTalk:Message Agent database size
BizTalk:MessageBox : General Counters Spool size
Tracking data size
BizTalk: MessageBox : Host Counters Host queue length
Host queue suspend message length

8 – Database session exceeds threshold
This state simply states that the sessions used by the host towards the database exceeds the specified threshold. The auto throttling here will do the same as in state 4 and 6, reduce the threads for EPM and stop the execution of any the threads, it will also delay all ongoing messages until the database is back to normal. If this happens i recommend you to reduce the database connection threshold for the host. You can monitor this by looking at the following counters in Performance Monitor

Counter Sub counter
BizTalk:Message Agent Database session

9 – Process count exceeds threshold
Basically what it says is that the process counts exceeds a specified threshold, the reason for this may be that you have to many transactions on that hosts that creates too many processes, try reducing the amount of allowed threads on the host so that the system don’t create too many threads. The self throttling will do the exact same as in state 8 and basically reduce the EPM thread count, stop it from creating new and messages will be delayed until the host is back to normal state again. You can monitor this by using the following counters in Performance Monitor

Counter Sub counter
BizTalk:Message Agent Thread count
BizTalk:Message Agent Thread count threshold

Message delivery throttling state

1 – Incoming rate exceeds outgoing rate
This happens if the message incoming rate is greater and exceeds the message outgoing rate, it can be caused by a shortage in resources or by high process complexity. BizTalk perform a self throttling blocking the delivery thread for a computed time until the message incoming rate is at par with the outgoing rate. To monitor this use the following counters in the Performance Monitor

Counter Sub counter
BizTalk:Message Agent Message publishing incoming rate
BizTalk:Message Agent message publishing outgoing rate

3 – in-process delivery exceeds threshold
this state informs us that the a number on in-process messages delivered to a service class exceeds the specified threshold and can be caused by processing high complexity messages, slow outbound adapters or a shortage in resources on the server. BizTalk self throttles and slows down the message delivery to XLANG or adapters, It also reduce the size of the thread pool by the message agent. You can monitor this by looking at the following counters in Performance Monitor

Counter Sub counter
BizTalk:Message Agent In-process message count
BizTalk:Message Agent In-process message count threshold

4 – Process memory exceeds threshold
This one is also covered well in publishing throttling states, it basically tells us that we exceed the specified value of the process memory threshold. It occurs when you have large and complex messages that are being processed, or a high amount of message simultaneously. This sate will not make the system idle so no imidiate action is needed if this is a once in a life-time scenario, however BizTalk will still perform self throttling and slow down deleivery of messages to adapters and XLANG. It may also reduce the memory usage by dehydrating messages and reduce the size of the EPM thread pool sometimes it also force a .NET garbace collect. you can monitor this by using the counters described below;

Counter Sub counter
BizTalk:Message Agent High process memory
Process memory usage (MB)
Process memory usage threshold (MB)
Active instance count
In-process message count

5 -System memory exceeds threshold
Basically the systems memory exceeds the specified threshold and BizTalk will slow down message delivery to adapters and XLANG, it will also try to decrease the memory consumption by dehydrating messages it also reduce the thread pool size of EPM. If you have this issue often you should consider reducing the EPM thread pool to decrease the occurence of this throttling state as it will slow down BizTalk drastically. You can read the monitor for this in Performance Monitor as well by using the following counter;

Counter Sub counter
BizTalk:Message Agent Physical memory usage

9 -Process thread count exceeds threshold
You will encounter this sate if BizTalk has a process thread count that exceeds the specified threshold. BizTalk will automagical throttle to decrease the thread count used by EPM and you should consider changeing the different thread throttling settings for the different hosts. You can monitor this issue too in Performance Monitor by checking the following counters;

Counter Sub counter
BizTalk:Message Agent Thread count
Thread count threshold

I hope you liked this blog post, please leave a comment if you have any questions, or send me an email.

You can download the entire article as a PDF document.
BizTalk Self Throttling.
  • Thanks for your post, it’s very interesting and remember some case I encountered.

    But, when I face with these kind of issue, I modify the parameter, could you give us some example???

    regards

    • Tord Glad Nordahl

      Yes, you can modify the parameter, but it all depends, if it is a temporary DNS issue theres is no need to start manuel throttling. However one of the main issues I see is servers running with much more resources then what is used. I will cover this a lot more in another post (i have started it, but it takes time). The main reason for this is that the throttling will differ from server to server. It’s very environment specific, and it requires a “bettter” knowledge of the current environment in order to throttle corect. As I said, im working on a blog post regarding this and I belive I’ll have it out this year. Stay tunes!

  • Well done. I had to deal with a two severe case 6 -host message size recently so i was keen to see what you wrote about this. I think that someone should write an article about the possible causes of this state. In my case it took quite a while to work out what the root cause was. In one case the counter had got out of sequence and the message box was not be cleaned up properly by the SQL agents. in the second it was a poorly written long running orchestration that would wait forever for asychronous response. In both case the spool was too big but very different root causes.

    • Tord Glad Nordahl

      I’ll see what I can do, but going into the depths of the states, the endless possible scenarios and solutions will in the end become a book. Performance optimizing is extremly relevant to the enviornment, the use of resources, and how to use the corresponding rest. So, making a general suggestion on how to do it isnt really needed, since microsofts automatic throttling and settings is the general one. If you do want to know more regarding throttling of your enviornment feel free to contact me, in the beginning of next year I will provide a deep dive course into health check, performance optimizing and monitoring. I’ll announce this information here when its all done, you can also give me a call and I’ll try to explain and help you with the current issue!

  • Leo

    Seems this is information from http://msdn.microsoft.com/en-us/library/aa559893.aspx article.

    • Tord Glad Nordahl

      Yes, and from my experience. 🙂 happy new year!

  • can i get biztalk server 2006 installed on a virtual machine, so that i can download it and start right away working on it?

    • Tord Glad Nordahl

      Hi Seo.

      Basically you can. However I recommend you to download 2010 Developer edition. Its free. Then you need a SQL server to install the database on. But yes you can all micrsoft vitual machines, and all third-party platforms that are SVVP-Certified can run BizTalk.

BizTalk360
BizTalk Server

Over 500+ customers across
30+ countries depend on BizTalk360

Learn More
Serverless360
Azure

Manage and monitor serverless
components effortlessly

Learn More
Atomicscope
Business Users

Monitor your Business Activity in iPaaS
or Hybrid integration solutions

Learn More

Back to Top