Searchindex Manager

experimental

This admin plugin allows you to rebuild the index used by the fulltext search. This isn’t needed generally as the index builds and updates itself while users browse your wiki. However if you just upgraded, added or removed a lot of files it may be a good idea to cleanup the index.

This Plugin needs a recent Browser as it makes use of modern JavaScript to carry out multiple tasks in the background (using AJAX).

• Download this file: searchindex-plugin-2005-09-04.tgz
• Unpack into the lib/plugins/ directory (you should end up with a lib/plugins/searchindex folder)

After installation there will be a Searchindex Manager link on the Admin page of your wiki. The page displayed will have a 'Rebuild Index' button. Click this to start the rebuilding process. You will be shown an activity bar and a status line listing each page as it is indexed. When the index rebuilding has finished you will be shown a “finished” message.

Please note that recent user experience suggests that this plugin will not work if you are using a template other than the default one. If you are using another template you will need to revert to the default from the Configuration Page before running it. You may also need to add a line to your /inc/indexer.php at line 56;

if (!is_array($links)) { $links = Array($links); }

To circumvent memory and timelimits set by the PHP environment this plugin uses an AJAX interface to call the indexer seperately every wiki page.

The JavaScript functions in script.js are called in the following order:

1. plugin_searchindex_go() by hitting the Interface button
2. plugin_searchindex_cb_pages() by AJAX callback
3. plugin_searchindex_clear()
4. plugin_searchindex_cb_clear() by AJAX callback
5. plugin_searchindex_index()
6. plugin_searchindex_cb_index() by AJAX callback → back to 5 if any pages left

Place these files in plugins/searchindex/lang/ru folder and don't forget about UTF8 encoding

intro.txt

====== Менеджер индекса полнотекстового поиска ======

С помощью этой страницы можно перестроить индекс, используемый для полнотекстового поиска.
Обычно эта операция не требуется, так как индекс автоматически строится и обновляется в то
время, как пользователи ходят по страницам wiki-сайта. Однако если вы только что обновили, 
добавили или удалили большое количество файлов, может оказаться полезным переиндексировать сайт.

Для работы Менеджера требуется новая версия браузера, который поддерживает современную 
версию JavaScript, так как  Менеджер реализован как многопоточное приложение 
(с использованием [[wp>AJAX]]-технологии).

Перестраивание индекса может занять продолжительное время. Для успешного выполнения работы 
Менеджера не закрывайте страницу до тех пор, пока не окончится индексирование.

lang.php

<?php
/**
 *russian language file
 * Alexander Sorkin aka Kibi <kibizoid [at] gmail [dot] com>
 */
 
// settings must be present and set appropriately for the language
$lang['encoding']   = 'utf-8';
$lang['direction']  = 'ltr';
 
// for admin plugins, the menu prompt to be displayed in the admin menu
// if set here, the plugin doesn't need to override the getMenuText() method
$lang['menu'] = 'Менеджер индекса полнотекстового поиска'; 
 
// custom language strings for the plugin
//Setup VIM: ex: et ts=4 enc=utf-8 :

— Kibi 18.06.2006

I've changed whole source code to apply thoroughly translation, including JavaScript AJAX part of the plugin. You may download it from my site : searchindex_i18n.zip — Kibi 19.06.2006

intro.txt

====== Suchindex Manager ======
 
Mit dieser Seite können Sie den Index neu erstellen, der von der Volltextsuche benutzt wird.
Normalerweise ist das nicht nötig, da sich der Index von selbst erstellt und aktualisiert,
während Besucher das Wiki benutzen. Aber falls Sie gerade eine große Menge Seiten aktualisiert,
hinzugefügt oder entfernt haben, kann es hilfreich sein, den Suchindex neu zu erstellen.
 
Dieser Manager benötigt einen aktuellen Browser, da er modernes JavaScript benutzt,
um mehrere Aufgaben im Hintergrund mit [[wp>AJAX]] aus zu führen.
 
Den Index neu zu erstellen dauert lange. Sie sollten diese Seite so lange
geöffnet lassen, bis die Indizierung beendet ist.

lang.php

<?php
/**
 * german language file
 */
 
// einstellungen muessen vorhenden und der sprache angemessen gesetzt sein
$lang['encoding']   = 'utf-8';
$lang['direction']  = 'ltr';
 
// die menu-ausgabe fuer das admin-plugin
// wenn dies hier gesetzt wird, muss das plugin die funktion getMenuText()
// nicht ueberschreiben
$lang['menu'] = 'Suchindex Manager'; 
 
// custom language strings for the plugin
 
//Setup VIM: ex: et ts=4 enc=utf-8 :
?>

— Andy Pillip 2007-08-17 11:55

Does this plugin work with the new 2006 dokuwiki releases?

It seems to be working — Kibi 2006-06-19 01:25

It doesn't seem to work with the 2006-11-06 release. I get the following errors;

Warning: array_keys(): The first argument should be an array in D:\dokuwiki-2006-11-06\inc\indexer.php on line 56

Warning: join(): Bad arguments. in D:\dokuwiki-2006-11-06\inc\indexer.php on line 56

and the progressbar below those errors never gets beyond 1, although it keeps attempting to run. Line 56 of indexer.php is;

      $tmp = join(' ',array_keys($links));                // make a single string

— DGM2 2007-01-03 17:58

I found my own answer to the above problem. The current version requires PHP 5+. Upgrading that is never fun or easy, but uninstalling PHP 4.4, completely, then running the Windows Installer for PHP 5.2 got me back up and running. — DGM2 2007-01-03 20:02

I got the same problem with indexing, but I tried this with php 4.4.6 and some 5.2.x (both XAMPP installations within Win XP). I got no success with both. On the other hand I managed to get this to work in the office with exactly the same XAMPP and/or php version. I will try further and report on success or failure! — forcemaker 2007-05-17 18:15

Yes, it turns out to be fickle. When it wouldn't run for me this time, even though I had PHP 5+ AND I had set Allow Debug to true (which turns out to be necessary), it was also necessary to modify the indexer.php to include a new line of code at line 56, as shown here.

if (!is_array($links)) { $links = Array($links); }
$tmp = join(' ',array_keys($links));

After that, it ran beautifully. I run a modified version of Christopher Smith's Sidebar template, and it works fine without switching back to the default template as some people say is necessary. — DGM2 2007-08-17 17:11

Despite the claim, above, that “the index builds and updates itself while users browse your wiki”, I found that not to be the case. My 2005-09-22 release wiki didn't update at all, no matter how many times pages were viewed, or by whom. This wiki was not an upgrade of an existing installation, which is listed as a source of problems; it was a clean, new install. Where it comes close to being an upgrade is the data. I copied the data/pages/* folders from my older installation to the new one. There are no .idx files in previous versions, so copying those wasn't an option. And, no amount of browsing/editing those pages caused the index to be updated, so it appears to me that browsing does not update indices (or indexes, if you insist) as documented. Only during the creation of pages does the index get updated.

But, the searchindex plugin did work for me, with no problems and no tweaking of any files required. (2005-09-22 release, running on Windows 2K with IIS) — DGM2 2006-03-17 22:45

As noted above, I have run into several problems with the Index Manager since posting the previous comment. — DGM2 2007-08-17 17:25

When I run this, I get the Found - The document was moved here text, and a progressbar. There is a link to a page that is in fact an 500 error on the web server. Is this normal?

I get the following error: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, webmaster@xx.xx and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request. Any ideas?

  I've the same problem (but no problem with Macos X/Apache/PHP 5 in localhost)

Plugin disappeared after first successful use.

I needed to add

print '<script language="javascript" type="text/javascript" charset="utf-8" src="/dokuwiki/lib/plugins/searchindex/script.js"></script>';

in admin.php to make it work…

The correct line should be:

print '<script language="javascript" type="text/javascript" charset="utf-8" src="lib/plugins/searchindex/script.js"></script>';

In case your wiki is not in ”/dokuwiki”

Is there a tool that can be run periodically from crontab to rebuild the index? I often copy text files directly into my wiki, and would like to automate a method to update the index with these new files.

Yes, the latest development version of DW has a PHP command line tool, that does exactly that. You can get the file bin/indexer.php from the DW Darcs repository. — Robert Rackl 2006-01-03 17:01

It kept throwing the error:

Fatal error: Maximum execution time of 60 seconds exceeded in /var/www/localhost/htdocs/dokuwiki/inc/indexer.php on line 190

until I added:

set_time_limit(600);

near the top of the file. PHP wizards may know this, but noobs like me didn't :)

Cheers, bt

On Linux (or any permission-based system), you generally need to run this as the web-server's user. For me, this was as simple as:

$ sudo -u apache php indexer.php -c

Tyler Bletsch 2006-10-16

Sorry if this is a stupid question, but once I have untarred the package into lib/plugins, what should I do?

Your wiki needs to have ACL enabled and you need to be the superuser. When you are logged in as the superuser on an ACL enabled wiki, an admin button will appear on the right hand side of the bottom button bar. Click that button to be shown the admin menu. If you have installed this plugin one of the options will be “Searchindex Manager”. Click that link and follow the on screen instructions. — Christopher Smith 2005-10-28 10:54

I didn't think it was stupid. You're not the only one. That kind of info should appear at the top of the page for each plugin, in a how-to-use section. new user 2006-01-25

Ok, now I have the searchindex directory in the lib/plugins directory. When I go to the Admin page I don't see the plugin listed. I am using Dokuwiki 1.1 in Xoops. Any idea why I can't see the link? Is there another way for me to run the indexer?Chris 2006-06-21

the index rebuilding gets stuck at “clearing index…”, any idea why this could be? are there special permissions the different .idx files need to have? because the plugin works perfectly on my local test site, but gets stuck on the online, full version hosted somewhere else.

After I call this plugin from the Admin Page, it shows a button “Rebuild Index” with a running bar to its right. No matter if i click the button and wait, or only wait, nothing happens. Does this plugin work at all??? I tried to look at the soure for a long time now, but didn't help. What should happen, once all pages are indexed? Should the page show anything? — Robert Rackl 2006-02-28 12:54

very poor english: after i enabled the debug mode (configuration) and saved the configuration, it shows a button “Rebuild Index” WITHOUT a running bar. then i clicked the button and it runs. perhaps this will help you. the crazy thing afterwards: after i disabled the debug mode (configuration) and saved the configuration, i can do a lot of button-clicks “Rebuild Index” - and it runs?! where's the bug?

thanks for this curious trick : it works !

After I upgraded to the 20060309 version I had the same problem, but the debug mode was already enabled. After disabling this mode, “Rebuild Index” worked correctly.(???)

I experienced that I have to perform a refresh (press F5) of that page with my browser (Firefox), otherwise nothing will happen.

Using Windows XP and Firefox 2.0.0.11, I have the same problem—a running progress bar, but no action. Neither refreshing the page—via pressing F5—nor clicking the “Rebuild Index” invokes any action. However, IE6 works.

on my case, go back to the default theme if you are using a customized theme. Then, searchindex will now work. After that, go back to your customized theme.

In my case, the plugin worked only after shifting to the default template. ?????

1. Install the plugin to ~/www/dokuwiki/lib/plugins/
2. Ensure theme is set to DEFAULT
3. Add the following to line 56 of /inc/indexer.php) if (!is_array($links)) { $links = Array($links); } (re moreupgradedetailhosted)
4. Visit the plugin [do=admin&page=searchindex] and rebuild the index.
5. All is now fine even after an update/upgrade of dokuwiki.

I've been searching for a way to index the content uploaded as well. Some people on my Wik upload documents (Word, PDF, etc) instead of creating pages. Thanks. Apologies if this is posted in the wrong spot. Just starting to pick this up so I'm flying a little blind.

Assuming your wiki is public: Have you tried the Google replacement plugin for search? Google indexes word and PDF files. — Viktor 2006-10-16

> For an internal DokuWiki deployment, is there a way to index the content of the uploaded files? — Joe 2007-01-16

If the wiki is integrated in joomla 1.0.11, the text “Direct Access to this location is not allowed.” appears in the box. Simply horrible.

I had the same problem, here is the solution that worked for me

in the admin.php that goes inside the zip file for this plugin, you have to redefine the constants:

if(!defined('DOKU_INC')) define('DOKU_INC',$mosConfig_absolute_path.'/components/com_jd-wiki/');
if(!defined('DOKU_PLUGIN')) define('DOKU_PLUGIN',DOKU_INC.'lib/plugins/');

Same thing with ajax.php from the zip file, but not the same way to define the constants, because here we lost all the context:

define('_VALID_MOS', 1 );
if(!defined('DOKU_INC')) define('DOKU_INC','./../../../');
if(!defined('DOKU_PLUGIN')) define('DOKU_PLUGIN',DOKU_INC.'lib/plugins/');
require_once(DOKU_INC.'../../configuration.php');

Note that you are also loading the configuration.php from joomla.

Still in ajax.php, get rid of this security check that is too difficult to pass in the joomla version: transform into comments like this

//we only work for admins!
/*if (auth_quickaclcheck($conf['start']) < AUTH_ADMIN){
  die('access denied');
}*/

Moreover, there is a bad instruction on the line 123 of this same file (ajax.php). Replace

io_saveFile(metaFN($id,'.indexed'),'');

with

io_saveFile(metaFN($_POST['page'],'.indexed'),'');

Then you have to modify the file components/com_jd-wiki/inc/auth.php: at line 35 replace

auth_login($_REQUEST['u'],$_REQUEST['p'],$_REQUEST['r']);

with

auth_login(null,null,null);

The parameters in this line were not accessible in joomla version. Other parameters are called in the auth_login function, so don't worry your security will not be weaker.

And finally you should modify your file components/com_jd-wiki/inc/indexer.php. Just to make it cleaner: line 136 replace

idx_writeIndexLine($tmp,$line,$pid,$words[$lno]);

with

if(isset($words[$lno])){idx_writeIndexLine($tmp,$line,$pid,$words[$lno]);}
else{idx_writeIndexLine($tmp,$line,$pid,false);}

line 153 replace

idx_writeIndexLine($tmp,'',$pid,$words[$lno]);

with

if(isset($words[$lno])){idx_writeIndexLine($tmp,'',$pid,$words[$lno]);} 
else{idx_writeIndexLine($tmp,'',$pid,false);}

Hope this help — Sébastien Lamy 2006-10-19 17:27

After upgrading to the 2006-11-06 release and re-installing this plugin, I am getting the following error inside the box continuously when it attempts to create the new index.

Warning: array_keys(): The first argument should be an array in /home/zyberdog/public_html/wiki/inc/indexer.php on line 56

Warning: join(): Bad arguments. in /home/zyberdog/public_html/wiki/inc/indexer.php on line 56

A temporary fix/hack for this is to insert the following on line 56 in /inc/indexer.php:

      if (!is_array($links)) { $links = Array($links); }

— Zyberdog 2006-11-07 10:59

I didn't read know of this workaround, so I just added an '@' before the join-function on line 56 and it did the trick.

$tmp = @join(' ',array_keys($links));

Works fine. — konstantin baierer 2007-05-02 16:29

I can verify / duplicate the issue; Thanx for the workaround Zyberdog.

— John 2006-11-08 23:12

Same here, works great with the workaround. Cool thing is that editing the file while the error is occurring fixes it, and allows indexing to continue. — Ronald Bruintjes 2007-01-22 16:05

Hi, when I press “rebuild index” I get the “clearing index…” message, then I get an “indexer is locked.” message, which goes back to the “clearing index” message and there appears to be no progress. What causes this?

— Loki 2007-03-08 11:44

This error seems to show up and disappear randomly, I just don't understand it. I eventually did get the index rebuilt, though, and both backlinks and search work fine now. — Loki 2007-03-08 11:44

Hi, I manage to rebuild index for about 1700 out of my 1800 pages, and then it says “indexer is locked”. Does anyone know what it means and what I can do to fix this? I have spent 3 hours rebuilding the index, and I don't want to redo it, just to find out that it generates the same problem again.

— Olivia 2007-04-02 14:33

Okay, I got past the above “indexer is locked” problem (only overwrote the entire installation; no biggie), but now I'm getting the original error:

Warning: fopen([wiki path]data/cache/page.idx) [function.fopen]: failed to open stream: Permission denied in [wiki path]inc/indexer.php on line 134

I think this is because allow_url_fopen is off by default in the configuration. So I edited .htaccess in the dokuwiki folder to add the following line:

php_flag allow_url_fopen on

And it's still giving me the same error. Did I misdiagnose the problem, or is the fix wrong?

This turned out to be a file permission issue with page.idx. Also word.idx, according to indexer.php's bug report. Fixed the file permissions on those two files and the rebuilding went fine. Maybe the “indexer locked” message above had something to do with the permission issues, I have no idea. — Loki

I had trouble with “left over” items in the index, which was causing me to find pages with phantom text. (I assume this may have to do with recent changes in the indexing method, as I'm using DokuWiki 2007-06-26, but I never checked further.) So I modified the following:

• in /lib/plugins/searchindex/ajax.php, line 76

//clear all index files //io_saveFile($conf['cachedir'].'/word.idx',''); //io_saveFile($conf['cachedir'].'/page.idx',''); //io_saveFile($conf['cachedir'].'/index.idx',''); $indexfiles = scandir($conf['indexdir']); foreach($indexfiles as $indexfile) { if(substr($indexfile,-4) == '.idx') { io_saveFile($conf['indexdir'].'/'.$indexfile,''); } } — Todd Augsburger 2007-07-12 10:56