[gopher] New Gopher Proxy
Brian Koontz
brian at pongonova.net
Sat Dec 5 22:22:33 UTC 2009
On Sat, Dec 05, 2009 at 02:38:07PM -0800, Cameron Kaiser wrote:
> If those people are really interested in
> gopher, then they need to be spidering gopherspace directly.
Why bother with an HTTP proxy to gopherspace if we want to prevent
HTTP-based spiders from accessing?
What might be useful is to figure out a way to serve up robot.txt to
HTTP-based spiders to prevent crawling of certain gopherspace pages
(as you brought up in your previous e-mail).
--Brian
--
Don't have gopher? Visit the world's first wiki-based gopher proxy!
http://www.pongonova.org/gopherwiki
More information about the Gopher-Project
mailing list