Thank you for your hard effort gnawing at the roots in the burrow! Veronica-2 is one of the key tidbits that make gopher so practicable and fun.<br><br>On Wednesday, 23 December 2015, <<a href="mailto:gopher-project-request@lists.alioth.debian.org">gopher-project-request@lists.alioth.debian.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Gopher-Project mailing list submissions to<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'gopher-project@lists.alioth.debian.org')">gopher-project@lists.alioth.debian.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'gopher-project-request@lists.alioth.debian.org')">gopher-project-request@lists.alioth.debian.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'gopher-project-owner@lists.alioth.debian.org')">gopher-project-owner@lists.alioth.debian.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Gopher-Project digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Changes to Veronica-2 (and VISHNU's present retirement)<br>
(Cameron Kaiser)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 22 Dec 2015 10:46:03 -0800 (PST)<br>
From: Cameron Kaiser <<a href="javascript:;" onclick="_e(event, 'cvml', 'spectre@floodgap.com')">spectre@floodgap.com</a>><br>
To: <a href="javascript:;" onclick="_e(event, 'cvml', 'gopher-project@lists.alioth.debian.org')">gopher-project@lists.alioth.debian.org</a><br>
Subject: [gopher] Changes to Veronica-2 (and VISHNU's present<br>
retirement)<br>
Message-ID: <<a href="javascript:;" onclick="_e(event, 'cvml', '201512221846.tBMIk3fE14286980@floodgap.com')">201512221846.tBMIk3fE14286980@floodgap.com</a>><br>
Content-Type: text/plain; charset=US-ASCII<br>
<br>
Veronica-2's internals have been substantially rewritten (again). Currently,<br>
as the database now carries close to 4 million selectors -- most of which are,<br>
to my delight, perfectly valid -- certain keyword sets cause big pulls on the<br>
database and some queries will not return. For example, a relatively<br>
innocuous search for "debian linux" will pull about 1.67 million selectors<br>
that need to be evaluated and scored. This query completes and is highly<br>
accurate, but not within the one minute maximum timeout for queries sent<br>
by outside clients. Some pathological ones I investigated took as long as<br>
ten minutes. This problem will only get worse as Gopher slowly expands.<br>
<br>
Google solves this by throwing hardware at the problem and sharding the heck<br>
out of everything, but I can't afford to do anything much like that<br>
(<a href="http://gopher.floodgap.com" target="_blank">gopher.floodgap.com</a> is a commercial-grade server with fast storage, but<br>
its 2-way POWER6 CPU is showing its age, comparatively speaking). Although I<br>
will probably unthrottle the CPU at some point and eat the additional power<br>
usage cost, I wanted to see what I could wring out of it right now.<br>
<br>
The current version now has a lot more predictive logic and even more<br>
aggressive results-stage caching. If the predictor indicates that a query<br>
is likely to go Cartesian, it then takes the most impactful keywords (as<br>
determined by a tunable internal heuristic) and runs them against a second<br>
cache that uses statistical sampling to pull a representative set, using<br>
the more specific keywords' complete individual results for scoring purposes.<br>
Since building this secondary cache is somewhat expensive, it does not do<br>
so "live" (it takes about 30 minutes currently to analyze and generate the<br>
extracts), but we're trying to rely on cached data more anyway, so this is<br>
necessity turned into virtue.<br>
<br>
tl;dr: Some queries will still be slow, but almost all should complete<br>
within the one minute timeout, some queries will now be substantially<br>
faster, and the majority will still return useful and relevant results.<br>
Please report all weird and unexpected behaviour.<br>
<br>
As a consequence, though, VISHNU is now removed from the V-2 menu. Because<br>
it requires accessing and pulling the entire dataset to be relevant, it is<br>
not compatible with the statistical sampling technique V-2 is now using as<br>
the most useful queries for VISHNU tend to be the least scalable. It really<br>
just needs a full redesign from scratch and I'm not sure of the best<br>
approach yet.<br>
<br>
If you were using VISHNU previously, it will still respond to queries<br>
(assuming they work), though it is no longer publicly exposed on any Floodgap<br>
menu. It was a nice idea at the time, but that's what experiments are for.<br>
<br>
--<br>
------------------------------------ personal: <a href="http://www.cameronkaiser.com/" target="_blank">http://www.cameronkaiser.com/</a> --<br>
Cameron Kaiser * Floodgap Systems * <a href="http://www.floodgap.com" target="_blank">www.floodgap.com</a> * <a href="javascript:;" onclick="_e(event, 'cvml', 'ckaiser@floodgap.com')">ckaiser@floodgap.com</a><br>
-- Feeling a little blue in January is normal. -- Marilu Henner ---------------<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Gopher-Project mailing list<br>
<a href="javascript:;" onclick="_e(event, 'cvml', 'Gopher-Project@lists.alioth.debian.org')">Gopher-Project@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/gopher-project</a><br>
<br>
------------------------------<br>
<br>
End of Gopher-Project Digest, Vol 68, Issue 1<br>
*********************************************<br>
</blockquote><br><br>-- <br> <br>