[cap-talk] Toolbars and unguessable URLs

[Tyler Close]

On Wed, Sep 24, 2008 at 5:51 PM, Jasvir Nagra <jas at nagras.com> wrote:
> On Wed, Sep 24, 2008 at 5:17 PM, Tyler Close <tyler.close at gmail.com> wrote:
>> On Tue, Sep 23, 2008 at 1:30 PM, ♘ stay <stay at google.com> wrote:
>>> Browser toolbars from internet search companies routinely capture URLs
>>> that users go to and then index them.  This seems like a Very Bad
>>> Thing with respect to URLs as capabilities.
>>
>> Will they behave if given an appropriate robots.txt file?
>
> Whether a toolbar should honor robots.txt aside, an appropriate
> robots.txt file might be hard to write given that the robots.txt
> consensus spec suggests that a "#" starts a comment and doesn't
> provide a way to escape characters.

I was thinking the robots.txt file would just say don't index anything
in the URI hierarchy that contains the web-keys. I was hoping that if
the toolbar handed the URI off to the search index crawler, that the
crawler would obey the robots.txt, and so the URI would not end up in
the public search index. In this case, the damage would be limited to
the transmission of the visited URL back to the mothership, and the
carnage would end there.

--Tyler