[iDC] Alternatives to black-box page-rank algorithm (was conference summary part 2: the internet as playground and factory)

Sat Nov 21 18:40:15 UTC 2009

I learned a great deal from your book *Search Engine Society*, Alex, and
thanks for the comments below.

The concept of a public option in search is a great idea.  I promoted it at
a conference earlier this year at Yale on the future of libraries, in a talk
called "From Managed Care to Managed Knowledge."  It's archived here, along
with a very good talk from Brewster Kahle:
http://www.youtube.com/watch?v=pcktP5jz7mc&feature=youtube_gdata

I have also promoted it in a book chapter that will come out next year; I'll
excerpt some parts of it below:

The recent Google Book Search settlement negotiations have led Siva
Vaidhyanathan to characterize Google’s archive project as evidence of a
“public failure.” Whereas government intervention is often necessary in
cases of “market failure,” Vaidhyanathan argues that the reverse can occur:
market actors can step into a vacuum where government should have been.  In
the case of digitized books, the problem is presented starkly: why has the
Library of Congress failed to require *digital *deposit of books, instead of
merely accepting paper copies?  We can debate when such a requirement became
plausible; however, had the government required such deposit as soon as it
became feasible, the problematic possibility of a Google monopoly here would
be much less troubling.  If digital deposit ever is adopted, the government
could license its corpus to alternative search services.  There is no good
reason why the company that is best capable of reproducing books should have
a monopoly on search technologies used to organize and distribute them.

More ambitiously, an NGO or quasi-administrative NGO (QANGO) could undertake
to index and archive the web, licensing opportunities to search and organize
it to various entities that promise to maintain open standards for ranking
and rating websites and other internet presences.  Wikipedia, Slashdot, and
eBay all suggest methods of evaluating relevance and authority that could be
employed by open search engines.  If such a search engine became at least
somewhat popular (or popular within a given niche), it could provide an
important alternative source of information and metadata on ranking
processes.

* *

The need for a public option in search becomes even more apparent when we
consider the waste and inefficiency causes by opaque intermediaries in other
fields.  Like private health insurers, Google is a middleman, standing
between consumers and producers of knowledge. In programs like Book Search,
it will effectively collaborate with copyright owners to determine what
access knowledge consumers get, how much they have to pay, and on what
terms.  In the health field, providers and private insurers are often
concentrated industries in the US, and consumers (i.e., the businesses and
individuals who buy insurance plans) are not. Insurers and providers also
jealously guard the secrecy of many pricing decisions.  That is one key
reason why the US spends so much more on health care than other
industrialized nations, without getting consistently better results, access,
or quality.

* *

In contemporary debates on health reform in the US, reformists split into
two camps: those who believe that regulation of middlemen like insurers can
bring about fair results, and those who believe that only a public option
can serve as a benchmark for judging the behavior of private insurers.  Even
if the public option in health care falters now, it will remain a policy
option in the future if new regulations on private insurers fail to improve
their efficiency and quality, or to expand access to care.  If search
engines block or frustrate measures to increase their accountability,
governments should seriously consider funding public alternatives.

*from The Troubling Trend Toward Trade Secrecy in Search Engine Rankings and
Ratings*, forthcoming in The Law and Theory of Trade Secrecy: A Handbook of
Contemporary Research (Rochelle C. Dreyfuss and Katherine J. Strandburg,
eds., Edward Elgar Publishing, forthcoming 2010).

As the US considers a second stimulus, let's hope that some visionaries
promote a "Cyber WPA) that could carry out some of the ideas mentioned
above.  A small investment now could pay great dividends in the future.
Advances in search technology should not be shackled to ownership of the
most comprehensive database of content and search behavior (even if that
ownership were monopolized via entirely legitimate means).
All best,
--Frank

On Thu, Nov 19, 2009 at 2:10 PM, Alexander Halavais <halavais at gmail.com>wrote:

> I'm sorry to say I also missed Frank Pasquale's presentation...
>
> The "security through obscurity" argument that drives the black boxing
> of Google's algorithm doesn't eliminate "gaming" the system, it
> professionalizes it. If the government requires Coke to give us some
> indication of what goes into their products, isn't it fair to ask
> something similar of those who are "organizing the world's
> information"? There is a middle ground between fully disclosing the
> algorithm and providing some indication of the policy choices that
> have gone into its formation.
>
> A robust public option seems to be the most interesting alternative
> here. The (chiefly French & German) state effort in this direction
> seems to be dead in the water. There are ways this might be done in a
> distributed fashion, but not the collective will to do so yet. It will
> take some much clearer outrages on the part of Google before that
> changes...
>
> - Alex
>
> On Thu, Nov 19, 2009 at 3:08 AM, Zbigniew Lukasiak <zzbbyy at gmail.com>
> wrote:
>  > Hi there,
> >
> > I have not been at the conference and I don't know if this point was
> > raised, if it was then - please forgive me.
> >
> > On Wed, Nov 18, 2009 at 6:28 AM, nathan jurgenson
> > <nathanjurgenson at gmail.com> wrote:
> >> Frank Pasquale forcefully called on Google to be more transparent. Given
> >> what was discussed above, as well as Google’s central status in our
> >> day-to-day knowledge-seeking life, Pasquale leaves us with questions to
> >> ponder: should its page-rank algorithm be public? Should Google be
> allowed
> >> to up-rank or down-rank links based their relationship to the company?
> >> Should Google be able to simply remove pages from its listings? Should
> >> Google be forced to let us know when they do these things? ~nathan
> >
> > I am also more and more afraid of the kafquesque world of Google
> > government of our information sources - but they do have a valid point
> > for the secrecy of page-rank: this is about defending against those
> > that try to game the system.  If the page-rank algorithm was public it
> > would be analysed and effective ways to game it would be found and we
> > would drown under the deluge of spam.  Now there are still people and
> > companies that try to analyse the black-box - but at least their
> > actions cannot be very effective.
> >
> > If we are to be constructive in our criticism Google for the black-box
> > algorithm we should also propose some alternative.   Most probably
> > there is no alternative that Google could unilaterally deploy - most
> > probably this would require a complex web of law, social norms and
> > technical changes.  This would be an interesting project.
> >
> > Cheers,
> > Zbigniew Lukasiak
> > http://brudnopis.blogspot.com/
> > http://perlalchemy.blogspot.com/
> > _______________________________________________
> > iDC -- mailing list of the Institute for Distributed Creativity (
> distributedcreativity.org)
> > iDC at mailman.thing.net
> > https://mailman.thing.net/mailman/listinfo/idc
> >
> > List Archive:
> > http://mailman.thing.net/pipermail/idc/
> >
> > iDC Photo Stream:
> > http://www.flickr.com/photos/tags/idcnetwork/
> >
> > RSS feed:
> > http://rss.gmane.org/gmane.culture.media.idc
> >
> > iDC Chat on Facebook:
> > http://www.facebook.com/group.php?gid=2457237647
> >
> > Share relevant URLs on Del.icio.us <http://del.icio.us/> by adding the
> tag iDCref
>
>
>
> --
> --
> //
> // This email is
> // [x] assumed public and may be blogged / forwarded.
> // [ ] assumed to be private, please ask before redistributing.
> //
> // Alexander C. Halavais, ciberflâneur
> // http://alex.halavais.net
>  //
> _______________________________________________
> iDC -- mailing list of the Institute for Distributed Creativity (
> distributedcreativity.org)
> iDC at mailman.thing.net
> https://mailman.thing.net/mailman/listinfo/idc
>
> List Archive:
> http://mailman.thing.net/pipermail/idc/
>
> iDC Photo Stream:
> http://www.flickr.com/photos/tags/idcnetwork/
>
> RSS feed:
> http://rss.gmane.org/gmane.culture.media.idc
>
> iDC Chat on Facebook:
> http://www.facebook.com/group.php?gid=2457237647
>
> Share relevant URLs on Del.icio.us <http://del.icio.us/> by adding the tag
> iDCref
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.thing.net/pipermail/idc/attachments/20091121/4a467b06/attachment-0001.htm 