0

We are facing a problem with extremely high data writes to disk and huge amounts of traffic over loopback interface.

Environment: Debian 10.3 (4.19.0.8) - virtualized in hyper-v nginx/1.14.2 php-fpm 7.3 mariadb Ver 15.1 Distrib 10.3.22-MariaDB

The machine is serving 5 php Laravel portals with some dynamic but mostly static content (e-books with a lot of assets - pictures, audio, video material). Having 300-1500 concurrent users alltogether, the disk writes varies all the time from 50 - 200 MB/s (measured with iotop), data read is practically 0. In one day there could be more than 1 TB of data written to disk, having inbound traffic on eth0 of just around 10 GB. Free space on hdd is being practically unchanged. Disk writes correlate to traffic that passes over loopback interface. It looks like the nginx/fpm are all the time "rewriting" the data on the disk.

The machine has dedicated 48 cores, 48 GB of ram, raid-6 SSD, swap partition never touched.

So far we've tried many possible cfgs (switching php-fpm from socket to tcpip, sendfile off/on, tweaking nginx fastcgi_cache) but with no luck. Here is the current cfg: https://pastebin.com/S6rdjbvE

Has anyone experienced such behaviour? We are quite out of ideas..

Thank you. Nejc

Improve this question
2
  • did you triee with nginx to cache and also use memcached? Commented Mar 21, 2020 at 10:23
  • True, the problem was in session files that were being constantly overwritten. Putting them to memcached reduced the amount of data by 80%, thanks a lot! Commented Mar 23, 2020 at 18:32

1 Answer 1

Reset to default
0

Run the below command and try to set the value based on the result in www.conf

ps --no-headers -o "rss,cmd" -C php-fpm7.3 | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"Mb") }'

Calculate the values 

min_spare_servers + (max_spare_servers - min_spare_servers) / 2
Improve this answer
2
  • Thanks, it was not a problem in fpm spare servers but in session files. Storing them to memcached solved the issue, however i still see a lot of writes like this. I persume this is some buffer-cache for fastcgi data. Any chance to have this stored in memory instead of disk? nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)<br /> nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)<br /> nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)<br /> nginx(146977): W /var/lib/nginx/fastcgi/9/14/0000570149 (deleted)<br /> Commented Mar 23, 2020 at 18:33
  • if your not plenty on ram you may create a zram or tmpfs device Commented Mar 27, 2020 at 10:06

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.