0
Welcome Guest! Login
0 items Join Now

503 Error and high server loads

    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • I just need login credentials for super admin, though it would be better to have a separate test site allowing me to play with it.

      We could start by installing debug plugin to the live site and making it visible for admin only.
    • Gantry 5, enjoy!
  • Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • We've created a test website that is a copy of DB and files of live website in another folder that you can play with it.

      I've added the login credentials to Secure tab.

      Thanks for your suppport, we're waiting for your reply.
    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • It looks like that the setup isn't fully working (file permission issues) as I'm not allowed to install extensions:

      JFTP::write: Bad response.

      In addition there are a few issues in the site:

      1) Last time the site has been updated was almost 2 years ago. You really need to keep the software up to date.
      2) If possible, you should not be using FTP mode in Joomla, instead of that you should set up the server so that it can write the files directly.
      3) You should uninstall extensions that you do not use, like Gantry 4, old themes from Joomla 2.5 etc...
      4) You should generally be running the same version of Gantry and Hydrogen theme; for how to make overrides without them getting deleted on updates, there's a good documentation for that.

      PS. Have you migrated the site from Joomla 1.5? I'm not sure, but I know it used to run on Joomla 2.5 at least.


      First thing that comes to my mind is that your Gantry / theme versions are really old and it looks like that we have fixed some issues that may be related to yours. I really recommend you to update at least Gantry, but it would be better if you could keep every extension up to date as there are multiple known security vulnerabilities on the versions of Joomla and extensions you're running. Also (partially because of the previous reason) I recommend cleaning up the site from any extension that you had installed before but which aren't used anymore. And yes, turn off that FTP mode and properly configure your server instead as it just slows down the site and causes all kinds of issues.


      Now that your site has a copy of it, you can easily start testing how to do the the updates, non-FTP mode and removal of unused extensions without causing issues in your live site. Just remember to keep notes on what you should be doing in the live site and maybe also try to do some dry-runs in order to figure out if there are any issues during the process.


      I hope this helps, unfortunately I couldn't do much more as the test site was not properly been set up.
    • Gantry 5, enjoy!
  • Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • Hello Matias,

      Thanks for your support.

      Really sorry for the misconfiguration of FTP for test website, we didn't reconfigure it and It has used the live website (/fa). Now It's ok and you can install any extension.

      Regarding the other problems you've mentioned, Thanks for your kind information, It's true and we're working on it. First of all we should be able to resolve the Server Load issue and after that our plan is to upgrade everything.

      We didn't migrate from J!1.5, We've migrated from J!2.5 to 3.x.

      Please install your plugin and let us know If you could find out what's the problem regarding Gantry that cause high serverloads and 503 error. It's very crucial to know what is causing in Gantry/Template to make long lasting processes and cause high server loads.
    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • Just a quick note after doing some testing last night.

      Both sites (live and test site) stopped working with 503 at the same time, but admin kept on working just fine, including Gantry admin. If there was an issue with high load or high I/O, admin should have also been heavily affected, which makes me think that there is nothing wrong with your server -- in fact, it seems to be very fast (including the very slow and resource intensive cache clear and CSS recompilation tasks).

      I'm starting to think that there's something in the page itself which causes a deadlock of some kind. It cannot be in the filesystem nor in the database either, as the test site does not share either with the live site. The situation seems to go away in about 5 minutes, though I didn't track the time. Deadlock usually happens when 2 processes try to lock 2 different resources at the same time but do that in different order causing both threads to wait for their second resource to be freed until they die. After that, every subsequent request will fail too, as nothing can access the locked resource until it frees up.

      I'm trying to think what could cause this to happen, but it is hard without knowing more about the site. By doing some googling, I found out this can happen with redis cache so changing the cache to apcu might be a good test to make. If it isn't that, the only choice is to start disabling modules and plugins when the 503 happens until it starts working in the test site while the live site is still broken.


      As an unrelated note, I updated Gantry 5 in your test site and it seems to be causing some issues with your modifications in the theme. You may want to look into those before upgrading. :)
    • Last Edit: 5 years 3 months ago by Matias Griese.
    • Gantry 5, enjoy!
  • Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • Hello Matias,

      Thanks for your reply.

      We tested disabling redis but It doesn't have any effect.

      Also, regarding the plugins and modules we tried to disable one by one of plugins and modules but the server load didn't changed.

      Need to mention that when we even try to use Hydrogen default theme that doesn't have any of our plugins and modules again the server load won't decreased. So, the deadlock should be in the gantry itself.

      It's strange that both sites got 503 because the traffic is only send to /fa not /test1. Is there any configuration within gantry that share resources?

      Any help? It's related to Gantry and the only place it could be solved is RocketTheme for sure.

      We currently have to block some our visitors to decrease the visits and server load and It's a bad situation for us.
    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 3 months ago
    • Two Gantry installs share nothing between them, which is why I'm thinking there must be something else that is shared causing both of the sites to stop responding at the same time. This could have been redis caching, but it can also be some extension in the site that tries to access some remote resource, but fails to do so.

      Also if the load in the server isn't like 20-50, it is not high. Load of 1 means that one (virtual) CPU is being used 100% and if the server cannot handle it, load quickly raises fast to very high numbers and the site stops responding.

      The odd thing in this issue is that both sites are affected the same time but still admin runs just fine. Gantry is just a templating engine, it has no other logic than what is needed to render the pages. There's no caching (except for compiling layout and template files into PHP) and there's really nothing which could explain the issue. I would look elsewhere as I cannot think anything which could cause this to happen.

      The reason why changing the template works is that you won't be showing the same modules on the page if you do that. Maybe some module loads content from an external service but fails to get the response in time...? It would work most of the time as the module would cache its contents and maybe the service usually responds fast enough..? I'm just guessing here, but if it was Gantry, only a single site would be affected, not both.

      PS. disabling modules or plugins wouldn't change the server load. The only way to know what it was would be to go to the admin of the test site when the issue happens, verify that the issue happens in the test site, too, and start disabling things until the test site works while the real site doesn't work.
    • Gantry 5, enjoy!
  • Re: 503 Error and high server loads

    Posted 5 years 2 months ago
    • Hello Matias,

      We had lots of testings in this period to be more specific regarding this problem with the help of server admin.

      The only thing was shared between those site was our modules whose job was reading feed of our Forums to display them in our homepage (/fa). We've disabled them but the problem didn't changed solved.

      After contacting the server admin, he said that getting 503 error for frontend is because they had set some scripts to stop some of long processes while server is on load. It's the reason you get error in frontend and don't get it in backend. The frontend process is heavier and longer so It' killed while server is on load. So, your assumption that It's not Gantry because of both sites affected at the right time is not correct because of the Server Management.

      Also, Regarding the disabling plugins and modules, It's not the problem we tested it. The important thing is when we change the template to Default Hydrogen that doesn't have any of our modules, It doesn't help. I mean the server load won't start to decrease and everything is somehow the same when we have enabled our current template. So the modules cannot be lied in Modules and Plugins I think. But when we change the template to a non-gantry one, the server will become better by some minutes and server load is also decreasing.

      So, Three things currently is very important:

      - Some of our processes from frontend is not ended even some minutes after we put the site offline. You've previously said "something is triggering Gantry to recompile CSS which takes a very long time." What can cause It?

      - Sum of our Lstat processes sometimes takes to 32seconds! I think It can be some of the problem or even hint where is the problem.

      - You said it could be deadlock, as the previous assumption regarding external sources was not true because server management, What's your idea right now?
      Also, is feeing of resources in Joomla can be found via Strace? If yes, How?

      I think we should come back to checking Gantry and why some our processes from frontend is not ended normally and finally is terminating.

      Honestly, We're in a big trouble as we have between 200-400 active users in realtime (GA) and we cannot serve them :(

      We want to solve this problem to upgrade Gantry and template.

      We had removed the testing directory. If you need again, Please let us know.

      Thanks.
    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 2 months ago
    • I think I've found reason why you get so many lstat calls. Normally PHP caches the file stat data, but it doesn't do that if you enable open_basedir, see PHP bug 52312 .

      This said, most of the lstat calls from your example aren't coming from Gantry. There were in total of 144636 calls from which 67506 (46.6%) were coming from Gantry, and this is in Development mode. If I exclude the scss file lookups, which happen only in development mode, there would have been 21681 lstat calls less, meaning that 45825 out of 122955 calls (37.2%) would have been from Gantry. Each file stat read pings every parent folder, though, which may bias the total time spent in Gantry vs elsewhere, but it shouldn't really change the fact that Gantry only accounts less than half of the time spent in lstat calls.

      I'm not saying that lstat calls aren't slow in your system, but the performance impact of Gantry shouldn't be more than factor of 2 because of those calls. So clearly there is something else that causes the slowdown. You cannot say that lstat calls made by Joomla are faster than those made by Gantry.

      Please remember that using strace will make application to run 100-1000 times slower, so you should never keep it running in the production environments! This is the basic issue with all debugging tools including xdebug -- those will not find the real causes of the issues unless you find something that gets called way too many times or find out that a single call takes a lot of time.

      What comes to my comment of triggering Gantry to reload SCSS files -- that was the first thing I tested in the server and it wasn't the case. Your site was in development mode which was the reason why I saw lstat calls towards the SCSS files.

      As the result, I still do not think that the issue is caused by Gantry, otherwise your site would be slow all the time. Each request takes as much time (what it comes to Gantry) to process and my debugging tool didn't find anything unusually slow in my tests. The responds I'm getting are around 1-1.5 seconds which is quite usual for a Joomla site. Without lstat cache issue your site would likely be ~0.5 seconds faster (including both Joomla and Gantry lstat calls), but the difference isn't big enough to prove your thinking.

      Right now your site runs great, I have no issues on laoding the pages. It brings me back to my original thinking:
      • Load gets too high and saturates the server on rush hours
      • There's a lock that doesn't get free'd causing all request to stop working
      • There's a deadlock which resolves after the process dies out

      First one should be really easy to recognize, just check the load by using top. Others are really hard to figure out.

      You are the only one who I've heard of having these kinds of issues (after I ruled out SCSS/template recompilation). Usually Gantry 5 themes tend to run about the same speed or slightly faster than other themes with the same content in the page. When cached, Gantry really doesn't do that much.

      I am not sure how I could help you. I don't know where the issue is: are you running out of CPU time or I/O? Is there a deadlock or a long lock in either filesystem or in the database? I can send you a debugging tool, which tells how much time is spent inside of Gantry and how much in Joomla, but I'm not sure how useful it is as the site is fairly fast most of the time.

      Im just guessing here. From the testing in your site, I could see that you have an issue, but there was nothing unusual in your site while it was working. In fact, the server seemed to be rather good fit to your site and it seems to handle the normal load just fine.
    • Gantry 5, enjoy!
    • Matias Griese's Avatar
    • Matias Griese
    • Sr. Rocketeer
    • Posts: 249
    • Thanks: 104
    • Lead Developer

    Re: 503 Error and high server loads

    Posted 5 years 2 months ago
    • Quick idea: please try to temporarily disable open_basedir and make sure you have realpath cache enabled and properly configured in your installation. Default PHP values should work just fine, those are what I'm using and I'm getting sub 100ms response times in my test setup (Helium RL installation without any modifications).

      This test should either confirm or rule out lstat calls being an issue.

      PS. remember to run Gantry in production mode, you can find the option from the theme configuration (first item in Extras dropdown menu). It will radically cut down the lstat calls, both inside Gantry code and Twig. It also should stop all SCSS file checks.
    • Last Edit: 5 years 2 months ago by Matias Griese.
    • Gantry 5, enjoy!

Time to create page: 0.060 seconds