Since discovering the source of the timeouts, our engineers have implemented some fixes in AdvocateAnywhere to prevent it from affecting the larger AdvocateHub ecosystem. This includes optimizing the way it fetches information about available challenges (a major source of the heavy traffic) and rate limiting individual instances of AdvocateAnywhere so no single person can overload the system with requests.
We have been monitoring since Monday afternoon (our busiest time of the week by far), and have not seen any abnormal behaviour or received any complaints of downtime.
Apr 24, 16:15 UTC
Good morning! Our engineering team has been working hard to optimize the way that AdvocateAnywhere communicates with AdvocateHub so that traffic spikes don't slow down the entire system.
They are testing those optimizations right now and they will hopefully make it out to our production (customer facing) systems soon.
Apr 19, 13:14 UTC
While we still experience periods of instability we have narrowed the source of the issues down to AdvocateAnywhere, our embedded product.
We have been experiencing higher than usual traffic from customers using AdvocateAnywhere that has caused timeouts on AdvocateHub.
We are working now to improve the performance of the APIs serving our embedded products to eliminate the timeouts while investigating the origin of the traffic.
Apr 18, 20:54 UTC
You may have noticed that the AdvocateHub seems to be returning to a state of intermittent instability. We are looking into why this issues has returned right now.
Stay tuned for updates.
Apr 18, 19:06 UTC
We have eliminated the source of the heavy load causing AdvocateHub to become unstable. We will monitor for some time to ensure the application is behaving normally before giving the all clear.
Apr 18, 17:10 UTC
We have identified the source of the heavy load which has been causing AdvocateHub to time out. We are working to remedy the situation now.
Apr 18, 15:46 UTC
Administrators and Advocates in your AdvocateHub may be experiencing increased slowness and timeouts leading to 503 or 504 error pages when trying to access any part of the application.
Our engineers are investigating the cause of these timeouts. For the time being, reloading the page may help temporarily.
Apr 18, 15:24 UTC