Last Updated: 4/21/05
At DynDNS, we're big fans of Open Source technologies. One of the major parts of our system is the MySQL Database Server. MySQL is a wonderful piece of software, and serves us and our customers very well by providing the backend database for every hit on our website.
Running MySQL effectively in a multi-processor environment on FreeBSD 4.x requires the use of LinuxThreads, because FreeBSD 4.x's native pthreads cannot scale across CPUs. MySQL provides binaries that are linked against LinuxThreads, which allows each thread to appear to the OS as a separate process, allowing them to run on separate CPUs.
Several months ago we started running into problems where we couldn't create more than about 700 threads. We would always see the following error:
Can't create a new thread (errno 35); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
We searched and searched and couldn't come up with a solution. Eventually we simply re-tuned our application so it did not need as many threads, largely by turning down the MySQL wait_timeout variable, causing idle threads to timeout more quickly, since our application is smart enough to re-connect a thread that has been disconnected by the server.
Recently, we decided it was time to purchase a MySQL Network subscription, both as a way of contributing back to the development of MySQL, and for the technical support resources that it would make available to us. One of the issues we requested help on was this thread issue.
After some troubleshooting steps back and forth, Sinisa Milivojevic provided us with the solution we needed. There were actually two problems that needed to be corrected. First, it turns out that the LinuxThreads version used in the FreeBSD ports system allocates a static amount of stack memory to every thread that is created, and that this is hard compiled as 2MB per thread! Second, this LinuxThreads version also hard-codes the maximum number of threads per process to 1024.
With this information (and patches) in hand, we made the appropriate changes, re-compiled and re-installed LinuxThreads, re-ran the test code that we had gotten from MySQL, and lo and behold, we were able to create up to 4096 threads without a problem. A quick re-start of our MySQL daemon later, and our test script was able to use the full 2048 maximum connections specified in our configuration file.
The two patches below change the STACK_SIZE to 128K, and the
PTHREAD_THREADS_MAX to 4096, which should be sufficient for most people's
needs. (Note: you may need a larger STACK_SIZE if your threaded
applications are stack-intense - 128K is the default used by MySQL, though,
and generally ought to be enough for most purposes.) These patches are
against the FreeBSD port devel/linuxthreads with version number
2.2.3_16.
To use these patches, do the following (as root, or whatever user on your system has the appropriate privileges to mess with ports):
cd /usr/ports/devel/linuxthreadsmake cleanmake patch (this will download, extract, and run the
port-specific patches on the LinuxThreads distribution)cd work/linuxthreads-2.2.3_16patch -p0 < /path/to/internals.patchpatch -p0 < /path/to/sysdeps.patchcd ../..make allmake installViola, you will now have LinuxThreads installed with the appropriate patches.
Many thanks to Sinisa Milivojevic and Victoria Reznichenko of MySQL for helping us to find and solve this problem, and for giving me permission to post the solution here. We highly recommend MySQL, and if you need top-notch technical support for MySQL, there's no better place to get it than direct from the source through the MySQL Network.