Add attribute_warn_unused_result to ast_taskprocessor_push,
ast_taskprocessor_push_local and ast_threadpool_push. This will help
ensure we perform the necessary cleanup upon failure.
Change-Id: I7e4079bd7b21cfe52fb431ea79e41314520c3f6d
If both channels which should be masqueraded
are in the same serializer:
1st channel will be locked waiting condition 'complete'
2nd channel will be locked waiting condition 'suspended'
On heavy load system a chance that both channels will be in
the same serializer 'pjsip/distibutor' is very high.
To reproduce compile res_pjsip/pjsip_distributor.c with
DISTRIBUTOR_POOL_SIZE=1
Steps to reproduce:
1. Party A calls Party B (bridged call 'AB')
2. Party B places Party A on hold
3. Party B calls Voicemail app (non-bridged call 'BV')
4. Party B attended transfers Party A to voicemail using REFER.
5. When asterisk masquerades calls 'AB' and 'BV',
a deadlock is happened.
This patch adds a suspension indicator to the taskprocessor.
When a session suspends/unsuspends the serializer
it sets the indicator to the appropriate state.
The session checks the suspension indicator before
suspend the serializer.
ASTERISK-26145 #close
Change-Id: Iaaebee60013a58c942ba47b1b4930a63e686663b
When taskprocessors get backed up, there is a good chance that we are
being overloaded and need to defer adding new work to the system.
* Implemented a high/low water alert mechanism for modules to check if the
system is being overloaded and take appropriate action. When a
taskprocessor is created it has default congestion levels set. A
taskprocessor can later have those congestion levels altered for specific
needs if stress testing shows that the taskprocessor is a symptom of
overloading or needs to handle bursty activity without triggering an
overload alert.
* Add CLI "core show taskprocessor" low/high water columns.
* Fixed __allocate_taskprocessor() to not use RAII_VAR(). RAII_VAR() was
never a good thing to use when creating a taskprocessor because of the
nature of how its references needed to be cleaned up on a partial
creation.
* Made res_pjsip's distributor check if the taskprocessor overload alert
is active before placing a message representing brand new work onto a
distributor serializer.
ASTERISK-26088
Reported by: Richard Mudgett
Change-Id: I182f1be603529cd665958661c4c05ff9901825fa
* Add new API call to get a sequence number for use in human friendly
taskprocessor names.
* Add new API call to create a taskprocessor name in a given buffer and
append a sequence number.
Change-Id: Iac458f05b45232315ed64aa31b1df05b875537a9
In practical tests, we have seen certain taskprocessors, specifically
Stasis subscription taskprocessors, cross the recently-added high-water
mark and emit a warning. This high-water mark warning is only intended
to be emitted when things have tanked on the system and things are
heading south quickly. In the practical tests, the Stasis taskprocessors
sometimes had a max depth of 180 tasks in them, and Asterisk wasn't in
any danger at all.
As such, this ups the high-water mark to 500 tasks instead. It also
redefines the SIP threadpool request denial number to be a multiple of
the taskprocessor high-water mark.
Change-Id: Ic8d3e9497452fecd768ac427bb6f58aa616eebce
We have observed situations where the SIP threadpool may become
deadlocked. However, because incoming traffic is still arriving, the SIP
threadpool's queue can continue to grow, eventually running the system
out of memory.
This change makes it so that incoming traffic gets rejected with a 503
response if the queue is backed up too much.
Change-Id: I4e736d48a2ba79fd1f8056c0dcd330e38e6a3816
Performing a directed call pickup resulted in a deadlock when PJSIP
channels were involved.
A masquerade needs to hold onto the channel locks while it swaps channel
information between the two channels involved in the masquerade. With
PJSIP channels, the fixup routine needed to push a fixup task onto the
PJSIP channel's serializer. Unfortunately, if the serializer was also
processing a task that needed to lock the channel, you get deadlock.
* Added a new control frame that is used to notify the channels that a
masquerade is about to start and when it has completed.
* Added the ability to query taskprocessors if the current thread is the
taskprocessor thread.
* Added the ability to suspend/unsuspend the PJSIP serializer thread so a
masquerade could fixup the PJSIP channel without using the serializer.
ASTERISK-24356 #close
Reported by: rmudgett
Review: https://reviewboard.asterisk.org/r/4034/
........
Merged revisions 424471 from http://svn.asterisk.org/svn/asterisk/branches/12
........
Merged revisions 424472 from http://svn.asterisk.org/svn/asterisk/branches/13
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@424473 65c4cc65-6c06-0410-ace0-fbb531ad65f3
........
r399887 | dlee | 2013-09-26 10:41:47 -0500 (Thu, 26 Sep 2013) | 1 line
Minor performance bump by not allocate manager variable struct if we don't need it
........
r400138 | dlee | 2013-09-30 10:24:00 -0500 (Mon, 30 Sep 2013) | 23 lines
Stasis performance improvements
This patch addresses several performance problems that were found in
the initial performance testing of Asterisk 12.
The Stasis dispatch object was allocated as an AO2 object, even though
it has a very confined lifecycle. This was replaced with a straight
ast_malloc().
The Stasis message router was spending an inordinate amount of time
searching hash tables. In this case, most of our routers had 6 or
fewer routes in them to begin with. This was replaced with an array
that's searched linearly for the route.
We more heavily rely on AO2 objects in Asterisk 12, and the memset()
in ao2_ref() actually became noticeable on the profile. This was
#ifdef'ed to only run when AO2_DEBUG was enabled.
After being misled by an erroneous comment in taskprocessor.c during
profiling, the wrong comment was removed.
Review: https://reviewboard.asterisk.org/r/2873/
........
r400178 | dlee | 2013-09-30 13:26:27 -0500 (Mon, 30 Sep 2013) | 24 lines
Taskprocessor optimization; switch Stasis to use taskprocessors
This patch optimizes taskprocessor to use a semaphore for signaling,
which the OS can do a better job at managing contention and waiting
that we can with a mutex and condition.
The taskprocessor execution was also slightly optimized to reduce the
number of locks taken.
The only observable difference in the taskprocessor implementation is
that when the final reference to the taskprocessor goes away, it will
execute all tasks to completion instead of discarding the unexecuted
tasks.
For systems where unnamed semaphores are not supported, a really
simple semaphore implementation is provided. (Which gives identical
performance as the original taskprocessor implementation).
The way we ended up implementing Stasis caused the threadpool to be a
burden instead of a boost to performance. This was switched to just
use taskprocessors directly for subscriptions.
Review: https://reviewboard.asterisk.org/r/2881/
........
r400180 | dlee | 2013-09-30 13:39:34 -0500 (Mon, 30 Sep 2013) | 28 lines
Optimize how Stasis forwards are dispatched
This patch optimizes how forwards are dispatched in Stasis.
Originally, forwards were dispatched as subscriptions that are invoked
on the publishing thread. This did not account for the vast number of
forwards we would end up having in the system, and the amount of work it
would take to walk though the forward subscriptions.
This patch modifies Stasis so that rather than walking the tree of
forwards on every dispatch, when forwards and subscriptions are changed,
the subscriber list for every topic in the tree is changed.
This has a couple of benefits. First, this reduces the workload of
dispatching messages. It also reduces contention when dispatching to
different topics that happen to forward to the same aggregation topic
(as happens with all of the channel, bridge and endpoint topics).
Since forwards are no longer subscriptions, the bulk of this patch is
simply changing stasis_subscription objects to stasis_forward objects
(which, admittedly, I should have done in the first place.)
Since this required me to yet again put in a growing array, I finally
abstracted that out into a set of ast_vector macros in
asterisk/vector.h.
Review: https://reviewboard.asterisk.org/r/2883/
........
r400181 | dlee | 2013-09-30 13:48:57 -0500 (Mon, 30 Sep 2013) | 28 lines
Remove dispatch object allocation from Stasis publishing
While looking for areas for performance improvement, I realized that an
unused feature in Stasis was negatively impacting performance.
When a message is sent to a subscriber, a dispatch object is allocated
for the dispatch, containing the topic the message was published to, the
subscriber the message is being sent to, and the message itself.
The topic is actually unused by any subscriber in Asterisk today. And
the subscriber is associated with the taskprocessor the message is being
dispatched to.
First, this patch removes the unused topic parameter from Stasis
subscription callbacks.
Second, this patch introduces the concept of taskprocessor local data,
data that may be set on a taskprocessor and provided along with the data
pointer when a task is pushed using the ast_taskprocessor_push_local()
call. This allows the task to have both data specific to that
taskprocessor, in addition to data specific to that invocation.
With those two changes, the dispatch object can be removed completely,
and the message is simply refcounted and sent directly to the
taskprocessor.
Review: https://reviewboard.asterisk.org/r/2884/
........
Merged revisions 399887,400138,400178,400180-400181 from http://svn.asterisk.org/svn/asterisk/branches/12
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@400186 65c4cc65-6c06-0410-ace0-fbb531ad65f3
* Add a max_size option for threadpools. Also added a test for this option.
* Fixed comments to be more accurate and have fewer typos.
* Updated copyright dates on new files.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@379375 65c4cc65-6c06-0410-ace0-fbb531ad65f3
* Clarify some documentation
* Change copyright date of taskprocessor files
* Address potential issue of creating taskprocessor with listener if
taskprocessor with that name exists already
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@379124 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Now user data is allocated by the creator of the taskprocessor
listener and that user data is passed into ast_taskprocessor_listener_alloc().
Similarly, freeing of the user data is left up to the user himself. He can
free the data when the taskprocessor shuts down, or he can choose to hold
onto it if it makes sense to do so.
This, unsurprisingly, makes threadpool allocation a LOT cleaner now.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@379120 65c4cc65-6c06-0410-ace0-fbb531ad65f3
* Remove extraneous whitespace
* Bump up debug levels of messages and add identifying info to messages.
* Account for potential failures of ao2_link()
* Add additional test and some more test data
* Add some comments in places where they could be useful
* Make threadpool listeners and their callbacks optional
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@378652 65c4cc65-6c06-0410-ace0-fbb531ad65f3
The documentation for taskprocessors was incorrect with
regards to when a listener's alloc callback was called.
I also made the names of queued function calls in the
threadpool more uniform.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@377802 65c4cc65-6c06-0410-ace0-fbb531ad65f3
The new thread creation test fails because Asterisk locks up
while trying to lock a taskprocessor.
While trying to debug that, I found a race condition during taskprocessor
creation where a default taskprocessor listener could try to operate on
a partially started taskprocessor. This was fixed by adding a new callback
to taskprocessor listeners.
Then while testing that change, I found some bugs in the taskprocessor
tests where I was not properly unlocking when done with a lock. Scoped
locks have spoiled me a bit.
I still have not figured out why the threadpool thread creation test
is locking up.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@377368 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This helps account for the fact that it is unknown just
how many references may exist for a given taskprocessor
listener, so simply unreffing it from the taskprocessor
shutdown function is not enough to convey the gravity
of the situation.
By putting in a shutdown callback, it now becomes clear
to the listener not to try to do any further operations
on the taskprocessor.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@376381 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Taskprocessors are now divided into two units: the task queue
and their listeners.
When a task is added to the queue, the listener is notified and
can take whatever action is desired. This means that taskprocessors
are no longer confined to having their tasks executed within a
single thread.
A default taskprocessor listener has been added that mirrors the
old taskprocessor behavior.
I've tested it by running Asterisk and placing calls. It appears
to work as expected. I'm going to do some cleaning up first and
then write some unit tests to be sure everything works as expected.
git-svn-id: https://origsvn.digium.com/svn/asterisk/team/mmichelson/threadpool@376118 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch adds 'const' tags to a number of Asterisk APIs where they are appropriate (where the API already demanded that the function argument not be modified, but the compiler was not informed of that fact). The list includes:
- CLI command handlers
- CLI command handler arguments
- AGI command handlers
- AGI command handler arguments
- Dialplan application handler arguments
- Speech engine API function arguments
In addition, various file-scope and function-scope constant arrays got 'const' and/or 'static' qualifiers where they were missing.
Review: https://reviewboard.asterisk.org/r/251/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@196072 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Copied from my review board description:
This is a continuation of the API changes documentation started for describing
changes between releases. Most of the API changes were pretty simple needing
only to be brought to attention via the new "Asterisk API Changes" list.
However, if you see anything that needs further explanation feel free to
supplement what is there. The current method of documenting is to add (in the
header file): \version <ver number> <description of changes> and then to add
the function to the change list in doxyref.h on the AstAPIChanges page. I also
made sure all the functions that were newly added were tagged with \since
1.6.1. I think this is a good habit to start both for the historical aspect as
well as for the future ability to easily add a "New Asterisk API" page.
Review: http://reviewboard.digium.com/r/190/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@180719 65c4cc65-6c06-0410-ace0-fbb531ad65f3
A task wraps a callback function pointer and a data pointer and is managed internal to the taskprocessor subsystem. The callback function is responsible for releasing task data.
Taskprocessor API
* ast_taskprocessor_get(..) - returns a reference to a taskprocessor
* ast_taskprocessor_unreference(..) - releases reference to a taskprocessor
* ast_taskprocessor_push(..) - push a task into a taskprocessor queue
Check doxygen for more details
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@115268 65c4cc65-6c06-0410-ace0-fbb531ad65f3
