I'd like to get some tips on how to improve the performance of DokuWiki. This includes parsing, searching and everything else you can think of. These are currently just a few loose thoughts.
Think the big thing, with PHP, is being clever about what takes place at runtime (on each page request). If there are operations taking place which are repeated in exactly the same on each request, try to push them to “compile time”.
A prime example would be those language files which are currently being loaded and executed at runtime, for embedding in the HTML output viainc/html.php- the alternative would be to “compile” multiple versions ofinc/html.phpfor each language, when you're preparing a Dokuwiki distribution, embedding the contents of the language files into the PHP source. I've been finding cog very helpful in this - while developing I can execute “uncompiled” PHP source, the cog statements being ignored by PHP then when it comes the distibute, run the script through cog to “compile” it.
Another example (not meant as criticism - just suggestion) is the current utf8 implementation. In there are functions like;
function utf8_strpos($haystack, $needle,$offset=0) { if(!defined('UTF8_NOMBSTRING') && function_exists('mb_strpos')) return mb_strpos($haystack,$needle,$offset,'utf-8'); // etc. code to execute if no mbstring available. }
Working out whethermb_strposis available will happen every time this function is called. An alternative approach to handle this would be a conditional includes at the point the utf8.php script is loaded, e.g.;
if ( extension_loaded('mbstring') ) { require_once(DOKU_INC.'inc/utf8_mbstring.php'); } else { require_once(DOKU_INC.'inc/utf8.php'); }
Otherwise, specific to Dokuwiki, think alot could be achieved, performancewise, by indexing the metadata. Specific to searching, it might be worth considering indexing with something like plucene (as an “add on” for those users that can install and use it) - believe it should be possible to generate indexes PHP can read.
I'm playing around with xdebug to find out what takes most time.
xdebug is an excellent tool for profiling. It's worth storing the output so you can look back at it later. Can also be worth using it on a live site occasionally to see what impact real data is having. FYI another alternative with similar functionality is APD
Might also be worth looking at HTTP conditional requests in PHP - does a good job explaining the options with HTTP caching.
One thing I found is that GeSHi the syntax highlighter is quite slow. Pages with a lot of code examples (such as the parser page) may even exceed the maximum execution time on a slower machine. While I like GeSHi's interface and found it's developer very respondent, I'm thinking about a replacement. But what are the alternatives?
Another one to look at Beautifier - supports alot of languages though output not as “pretty” as GeSHi. Not sure what the performance is like but from glancing at the source, looks like it parses in a single pass and it's written by a Perl coder, if that's any indication
Another thing: the GeSHi Documentation states the following:
Unfortunately, GeSHi is quite load/time intensive for large blocks of code. However, you want speed? Turn off any features you don't like, pre-make a stylesheet and use CSS classes to reduce the amount of output and more - it's easy to strike a balance that suits you.
You may also wish to consult Section 3.8: Disabling Lexics for more information on speeding up GeSHi.
Note to Andreas: GeSHi 1.2 will implement caching of source to save much time and CPU load - compile once, serve 10000 times. And it may end up faster than 1.0.X anyway – nigel [at] geshi [dot] org]
Thanks for the info! I'm looking forward the 1.2 release. However caching will not help here. DokuWiki already does this, so the speed problem is on the first pass only anyway. — Andreas Gohr 2005-05-11 10:29
Another thing is to use a cache. One that I have used is ABcache but it seems to be somewhat dated (last entry: 2002). There are others out there, but I've not used them.
Some of these include:
As I said, I've not used these. Here is an article on PHP caching which has many links to other articles as well.
Here is an article on optimizing PHP, including caching.
With PHP 5.1 and APC 3.0.8, APC tends to break DokuWiki, specifically to do with the plugins. If you enable APC on a dokuwiki site, you will probably want to filter out everything in the lib/plugins directory from caching, at least until the APC crew fixes the bugs.
It appears that version 3.0.9 fixes the problems that were breaking in 3.0.8.
— Jonathan Arkell 2006-02-16 22:44
I have had some problems editing pages with Dokuwiki 2006-11-06, PHP 4.3.10 and APC 3.0.12p2 on Debian GNU/Linux. Filtering out lib/plugins directory from caching fixes them. – frjo 2006-11-24
The following is predicated on my understanding of what dokuwiki is mainly about. I believe it is designed for a wiki whose pages that will read many times for every time they are edited, and should therefore be optimized for the common case.
It seems to me then that the wiki pages themselves don't need to even go through PHP (when they are not being edited), wouldn't it make sense to have them be static html pages and let apache take care of serving them? (obviously there are a few pages like recent changes, and so forth that need special care, but using the cache just for those should work).
Of course, the editing of pages will not be sped up, but that is not (in my impression) the common case for dokuwiki.
C.J.
I think page-level caching is the single most effective change that DokuWiki can make. Combine this with only loading the PHP files needed to load a page from cache and no other files, and you have a level of performance that's hard to beat. Additional PHP would be loaded only when pages need to be generated.
By “page-level caching,” I mean caching the entire resulting output HTML page, as generated by the template files. This approach optimizes for read-only access by non-logged-in users, as you'd have to generate at least a portion of each page for logged in users (to embed the login ID). I'm guessing that the DokuWiki installations having the greatest peak-load requirements are those whose primary user is an anonymous visitor, so this solution would address the situation where peak-load actually becomes an issue.
I'm not sure I understand the current caching approach. Why does the parser cache page instructions and not the entire Parser-generated portion of the page? To support edit buttons, you'd probably cache each page section individually. But why does it cache instructions? Can you make one “syntax mode” dynamic and the next cached? Or is the entire page cached or dynamic according to whether one syntax mode of the page is dyanmic, in which case it seems to me that page sections would be a more efficient caching approach. I'm missing something here. — Spider Joe
Part of the wiki page may be affected by changes to other wiki pages. The primary example of this is internal wiki link colour coding, the parser needs to reprocess the page rendering instructions in order to determine the correct colour to assign to any links on the page. Dokuwiki does cache the html output of the renderer, however, the life-time of such cached pages is comparatively short. In a large, mainly read (rather than update) wiki, the savings could still be significant. — Christopher Smith 2006-02-17 01:49
When thinking about the performance problem, it is helpful to distinguish two kinds of performance: peak (load) performance and off-peak performance. Improving one area of performance may not improve the other.
Off-peak performance can be measured as average page-load time when the server is not being taxed. This performance merely determines how long a client has to wait to get a page. It's a function of both how much work the server has to do to generate the page (including how fast that code is) and how much data the client must download. This performance determines what kinds of clients are suitable for accessing the site, including browser type and bandwidth.
Peak performance can be measured as the maximum sustainable hits per unit time that the server can handle. When the server exceeds its peak performance level, clients may be denied connections, or once connected, may timeout waiting for the server to provide the page. Peak performance is governed by both the resources that the server offers (clock speed, memory, OS, drive speed, etc.) and how DokuWiki uses those resources. It determines what kind of server is suitable for running DokuWiki.
Of the two, peak performance is generally the more critical. The peak performance behavior of DokuWiki governs whether or not DokuWiki can be used to do the job. Off-peak performance merely governs responsiveness, but can be important if responsiveness is worse than people's normal wait threshhold.
My personal experience tells me that DokuWiki's off-peak performance is far worse than it needs to be. I often access my web site from my Handspring Treo, where page size is the major performance factor. Pages that should only include 40K of download are actually downloaded around 200K. Consequently, my DokuWiki site is not usable from my Treo. I'm assuming that DokuWiki is causing unnecessary file downloads, but I haven't looked into this yet.
However, I've seen a few complaints about DokuWiki's peak performance capabilities. These are the kinds of complaints that arise when there's a high hit rate, usually accompanying a large web site. This issue is more critical for DokuWiki, I think, because it limits DokuWiki to either low-traffic installations or to high-power, sophisticated servers.
To solve the peak performance problem we need to address DokuWiki's use of resources. The standard approach is to cache pages, so that all the server does is retrieve and transmit pages — the minimum that any web server can be asked to do. Caching is itself a challenging problem, as the caching solution must take into account the dynamic needs of the server.
And caching does not always solve the problem. For example, some pages may contain dynamic data that needs to be updated on each page load. If these pages are few or only infrequently accessed, their resource requirements are likely negligible. However, if there are many of these pages or they are commonly accessed — as in a site whose users all log in or many of whose users are updating pages — then DokuWiki's resource management becomes absolutely critical.
Resource management is a complicated issue. Memory is the most crucial resource and using less memory on each page access can greatly improve performance. However, rather than just asking, “How can I make a page load use fewer resources?” we have to ask, “How does one page load interact with another concurrent one?” In other words, where will one page block waiting on another page? This includes database locking (not applicable to DokuWiki) and file system access. But it also includes accessing the system heap, which must allocate memory to each page and centrally manage the allocation links.
Again, we're back to memory use as the primary and most likely culprit of peak-load problems on highly dynamic sites. Only the issue isn't just how much memory is being used; it's also the frequency of new allocations. Barring a detailed study of how DokuWiki actually uses its resources and where the bottlenecks are, making more efficient use of memory is probably the best thing we can do, second to page-level caching.
In my experience, memory problems frequently arise when too many objects are being created. Having seen quite a bit of the DokuWiki code, I have a guess about what is being created most excessively: strings. Preg functions generate tons of strings. Each substr() call generates a string. But a hidden source of immense numbers of strings is the simple string concatenation operator. Each string concatenation results in a new string of greater size than the previous, in addition to the clock-cycle hit of copying the contents of both strings into a new string. The minimum per page heap size shoots up, and it goes up in increments so that multiple, possibly-blocking heap allocations must occur. Bus speeds are generally much slower than CPU speeds, and this much data transfer can drop the effective CPU speed to something closer to the bus speed.
Many string concatenations cannot be helped, but I believe that object creation can be reduced by one or two orders of magnitude by having all things that output to the page — particularly the parser — write to a buffered output stream. PHP allows output buffers to nest, so that ob_start() creates a new output buffer not affecting an outer output buffer, the topmost of which is the stream intended for the client. An output buffer typically creates a new object every so many characters (say every 4K), and no copying of previously output data occurs (except for a few rare instances required by buffer management).
I think this can be done without breaking syntax plugins. Plugins output by concatenating to $renderer→doc. We can't eliminate that particular concatenation in existing plugins, but we can eliminate the unnecessary copying of data. We can also hope that PHP is smart enough not to create a new object when cat'ing to an empty string. Eventually plugins can be transitioned to use a renderer function that encapsulates what it means to output, so that even the plugins will be writing to an output buffer without string concatenation.
— Frank Hinkel 2006-08-08 17:29 How about using $renderer→doc[] = $value instead of $renderer→doc .= $value. So we just add the new data to an array instead of copying the entire string. I wrote a simple performance test for this, wich shows great advantages for the array-solution.
Okay, despite all this talk about string management, I still believe that the absolute best solution is to do page-level caching. This will greatly reduce peak performance requirements on all but the few sites that are highly dynamic. We can then help those dynamic sites by addressing resource use, and particularly memory use. And if we want to provide better support for low-bandwidth clients, we'll figure out how to better download only those files that a page needs (but in fairness, I'm at this time just guessing about the cause of the latter problem).
Just FYI: I have some experience with the performance problem. I wrote what may still be the most efficient tree-generating XML parser out there. It was recently compared to numerous other Java-based XML parsers and found to still out-compete them on peak loads. There are more tricks than I've named here, but memory is the big one. However, I have no experience with a regex-based parser. That's a lot of unknown behavior under the hood, not under the control of the application. — Spider Joe