Search log analysis

Discussion articles

  • Better search engine design: beyond algorithms
    "A useful search engine is more than a search algorithm. This article explains how to create a search query analysis tool, a best bets feature, and a basic controlled vocabulary".
    (Peter Van Dijck - O'Reilly Media)

  • How search can help you understand your audience
    "It seems to me that many of the metrics with which we measure user interaction with the web are deeply flawed, and provide ample evidence that the internet was invented by physicists and technologists, not marketing and advertising executives. However, the action of typing a word or phrase into a search box is a very direct interaction between the user and the site. We can safely assume that when the user typed in that word or phrase they expected 'something' back from the site."
    (Martin Belam - currybetdotnet)

  • Internal site search analysis: simple, effective, life altering
    "Understanding of your site visitors’ intent is one of the most delightful parts of web data analysis. In this article, we’ll learn five ways to analyze your internal site-search data—data that’s easy to get, to understand, and to act on."
    (Avinash Kaushik - A List Apart)

  • Intranet search reports
    "A range of statistics are typically gathered on intranet usage, but of these, search engine reports are by far the most useful. This briefing explores two key search engine reports that should be implemented on all intranets, and looks at how they can be used to improve the effectiveness of the site".
    (James Robertson - Step Two Designs)

  • Search log analysis and the long trail
    "Addressing the top 500 queries makes good sense because it could improve the user experience for 40% of those who search our sites. But what about all those other queries? There has been quite a lot of discussion recently about the value of the "long tail" in e-commerce and in many other contexts. I wonder if we're missing out on important opportunities by ignoring the long tail of search?"
    (Lou Rosenfeld - Bloug)

  • Site search analytics from the bottom up
    "We'll explore a 'bottom-up' approach that relies on pattern analysis and failure analysis to help you understand your users' intent in qualitative ways that complement the top-down approach."
    (Lou Rosenfeld - A List Apart)

  • Testing search for relevancy and precision
    "Despite the fact that site search often receives the most traffic, it’s also the place where the user experience designer bears the least influence. Few tools exist to appraise the quality of the search experience, much less strategize ways to improve it. But relevancy testing and precision testing offer hope. These are two tools you can use to analyze and improve the search user experience."
    (John Ferrera - A List Apart)

Research papers

  • Understanding web searching and navigation patterns
    "We describe a model for log data of user search sessions obtained from a trail-based search and navigation documentation system. The model elicits interesting patterns that can be used to better understand Web user search and navigation behaviour. Our study shows that such log data reveals interesting patterns beyond the typical statistical query terms analysis".
    (Mazlita Mat-Hassan, Mark Levene - University of London)

Case studies

  • A Day in the Life of BBCi Search
    Since BBCi launched in November 2001, its improved search offering has been collecting data on the way that BBC website users search both the BBC's website, and through its homepage Websearch, the whole wide web. This article shows how the search team analyse their search logs in order to understand user search behaviour.

  • Analysing user behaviour: a case study
    "Data was analysed from the web log files generated from four web-enabled databases hosted by The National Archives, using an in-house word analysis program written in the Perl programming language, using dictionaries of personal and place names built in-house. The initial analysis concentrated on determining the frequency of keywords per search. Keyword frequency analysis revealed that since 2001, users have increased the number of terms they use in a search - there appears to have been a shift from single word searches to two word searches."
    (Chris Kutler, Ray Devaney - FreePint Newsletter)

Books and book reviews

  • Search Analytics For Your Site
    "Any organization that has a searchable web site or intranet is sitting on top of hugely valuable and usually under-exploited data: logs that capture what users are searching for, how often each query was searched, and how many results each query retrieved. Search queries are gold: they are real data that show us exactly what users are searching for in their own words. This book shows you how to use search analytics to carry on a conversation with your customers: listen to and understand their needs, and improve your content, navigation and search performance to meet those needs."
    (Rosenfeld Media)

Presentations