Benutzer:Atlasowa/multilingual search

Wikipedia needs a multilingual search portal/feature.

Why?

• Because most Wikipedia readers read more than 1 language Wikipedia (72% !) and
• because Wikipedia readers want better Wikipedia search to find Wikipedia articles on the topics and
• because Wikipedia readers shouldn't be forced to "google Wikipedia" (and thereby reveal their Wikipedia requests/search history) and
• because Wikipedia readers can't "google Wikipedia" from Wikipedia apps/ Wikipedia Zero.

What do we need?

• simultaneous multilingual Wikipedia search, especially bilingual (first language WP + language WP with broader coverage: zh/en, ar/fr, kz/ru, ...)
• reader-optimized search portal, no clutter, to be linked at from language Wikipedia search/redlink pages
• www.wikipedia.org Portal and www.m.wikipedia.org for mobile
• www.wikipedia.org|ru|en permalinks for bilingual search, customizable (also for browser search plugin)
• search results with snippets, with image ("Seitenbild", mw:Extension:PageImages), with short text description (mw:Extension:TextExtracts)
• en:incremental search
• future enhancement: Wiktionary search integration
• future enhancement: image search [1]
• future enhancement: search result location on OpenStreetMap

Readers of Wikipedia need better search

• meta:Research:Wikipedia Readership Survey 2011/Results#b. Search is the most desired improvement to Wikipedia

• 
  Readers likely to use Wikipedia more if search was improved - by country. (meta:Research:Wikipedia Readership Survey 2011/Results)
• 
  Readers very likely to use Wikipedia more if search was improved - by language (meta:Research:Wikipedia Readership Survey 2011/Results)
• 
  Slightly over half of our readers (51%) said they specifically look for Wikipedia in search engine results like google (WMF Blog post)
• 
  Wikipedia access comparison across devices (C6. How do you access Wikipedia from the following devices? (note: different bases for desktop/laptop/mobile))
• 
  meta:Research:Characterizing Wikipedia Reader Behaviour
  - Why are you reading this article today?
  I am reading this article to … [Information Need]
  Prior to visiting this article … [Prior Knowledge].
  I am reading this article because … [Motivation] (microsurvey2015)

• Wikipedia search survey powered by Qualtrics "What kind of feedback do you have about search on Wikipedia?" (June 2015) meta:Research:Measuring User Search Satisfaction by Oliver Keyes

Readers of Wikipedia are multilingual

• 
  European countries and requested WP language versions, 2010.
• 
  Countries in Northern Africa and the Middle East and requested WP language versions, 2010.
• 
  Q2a. Percent who read the listed language Wikipedia / Q2b. Percent who primarily read the listed language Wikipedia (n=4,930).
• 
  Q1a. Percent who contribute to the listed language Wikipedia / Q1b. Percent who primarily contribute to the listed language Wikipedia (n=4,930).
• 
  Q2a. Number of Wikipedia languages read (n=4,930) meta:Editor Survey 2011/Location & Language
• 
  Q1a. Number of Wikipedia languages edited (n=4,930)

Wikipedia full text search page

Full text search page

Recherche multicatégorie Wikipedia

• "incategory:" search
• "deepcat:" search
• "intitle:" search

• MediaWiki-l: An "advanced search" page for CirrusSearch?

Interwiki search

• mw:Search#Interwiki search, boxes on the side of the search results page where matching pages from sister projects are shown: so you can search a word on Wikipedia and get the link to an entry in Wiktionary.
• The feature originally existed for all wikis around 2009 but was later disabled; as of now there isn't a timeline for re-enabling it by default.
• T46420: Restore interwiki (sister projects) results in search queries (Closed, Resolved)
• T87631: Interwiki search results should come after local ones in HTML
• T87632: Enable interwiki search on Meta-Wiki (Open)
• T96881: Don't show the "results from other projects" box if empty (Open)
• restored on italian WP: example search
• mailinglist January 2017: "secondary search results are now available over the API. This means that automated language detection (provided by TextCat <https://www.mediawiki.org/wiki/TextCat>) and query forwarding can now be used by API consumers."

Portals www.wikipedia.org and www.wikipedia.de

• 
  Wikipedia search portal at www.wikipedia.org, screenshot of 2021
• 
  Search portal at www.wikipedia.de, screenshot of 2015
• 
  ComScore trend data on WMF Sites, as of Jan 2010

• http://www.wikipedia.org
• meta:Talk:Www.wikipedia.org_template
• T5665 Auto-detect interface language for anonymous users (2005!)
  • tweet #wikipedia #what? wikipedia.DE: "Wozu Englisch, wenn man sich zwischen Obersorbisch und Saterfriesisch entscheiden kann?"

• T26767 - Multilingual search on project portals (e.g. www.wikipedia.org)
• T90835 - Investigate www.wikipedia.org traffic % ([2][3])
• T98076 Does portal traffic come from zero? (No)
  • Fwd: Traffic to the portal from Zero providers Oliver Keyes
  • Reader use of the Wikipedia home page Oliver Keyes, 2015-05-01
  • discovery: Portal A/B test announcements Oliver Keyes, Dec 21 2015 "initial test to identify if a more prominent search box would improve the rate at which users clicked through from the portal to our various projects (at the moment, only a third of users do so)."
• T104984 - Add a search option "in any language" to the multi-lingual search on www.wikipedia.org
• T71489 Expose mwgrep functionality on-wiki
  • Cross-wiki search API?

Prominent place for readers, traffic to portal

Autosuggest search word (by article names)

search in all Wikipedia languages

no simultaneous multilingual Wikipedia search

cluttered search page

no search results page, directs to Wikipedia Special:Search

Mobile Wikipedia search

• 
  Wikipedia Android app search with descriptions
• 
  Wikipedia Beta search on Android 4.4.4 2015-02-09 (incremental search?)
• 
  WIKIPEDIA MOBILE READER TOPLINE 2011
• 
  Malaysia WP Zero landing page

• meta:Research:Wikipedia_Mobile_User_Research#Searches for Wikipedia content start in Google Wikipedia Mobile User Research 2011: 20-22% want multilingual Wikipedia search

• Bug 30389 - making http://www.wikipedia.org/ mobile friendly
• T30815 - wikipedia.org should detect mobile browsers and local language, and forward to xx.m.wp

• Wikimedia-l/Wikitech-l: Re: Making www.wikipedia.org mobile friendly 2012
• WikimediaMobile: Re: Where to redirect m.wikipedia.org? 2013
• mw:Search/Design
• WikimediaMobile Mailinglist: Simplified search? It's possible! How good is our search? Here's some data! 2014

clean search interface, no clutter

autosuggest for search

Search results page with pictures

Search results with WP article lede sentences

no simultaneous multilingual Wikipedia search

Search tools by Wikipedians

Bong by Magnus Manske

Search-bong-tools-wmflabs-magnus lagunita

• Bong: Bing-like Wikimedia search interface. By Magnus Manske @tools.wmflabs
• https://tools.wmflabs.org/magnus-toolserver/bong.php?doit=1&mode=pedia&q=lagunita
• Bong image search (basic, small)

Clean search interface

Search results page with pictures

Shows WP article lede sentences

Supports several wikimedia projects (commons, wiktionary,...)

No search snippets

No simultaneous multilingual search, only english

No autosuggest for search

No incremental search

GlobalWPsearch by Aka

Globalwpsearch for lagunita

• http://vs.aka-online.de/cgi-bin/globalwpsearch.pl
• option: multi line search field (Please enter one article name per line.)
• http://www.similarweb.com/website/vs.aka-online.de : 68.12% of total Vs.aka-online.de desktop traffic in the last 3 months came from referrals. Top Referring Sites:
  • fr.wikipedia.org
  • it.wikipedia.org
  • hi.wikipedia.org
  • en.wikipedia.org
  • zh.wikipedia.org
  • tl.wikipedia.org
  • az.wikipedia.org
  • de.wikipedia.org
  • es.wikipedia.org
  • mr.wikipedia.org
• http://www.alexa.com/siteinfo/vs.aka-online.de : Which sites did people visit immediately before this site? 1. wikipedia.org 66.7%

Benutzer_Diskussion:Aka/GlobalWPSearch since 2005!

simultaneous multilingual search! (~50 Wikipedias)

prominently linked in several Wikipedia language versions:

IT it:MediaWiki:Noarticletext: "Cerca nell'enciclopedia se esiste un titolo simile. tutte"

FR fr:Nick_Helm: "Recherchez cet article dans d’autres langues, grâce à l’outil Global Wikipedia Article Search (en anglais)."

HI [4]

No search snippets, no article ledes

No images

No autosuggest for search

No incremental search

Wikimedia-Search by Kolossos

Wikimedia-Search by Kolossos

• https://toolserver.org/~kolossos/tree/search.php?pro=wikipedia&langauswahl=en dead with toolserver
• Wikipedia:Kurier/Archiv_2005/2: "Auf dem Toolserver befindet sich die von Kolossos entwickelte Suchmaschine Wikimedia-Search für alle Projekte und Sprachen der Wikimedia. Dieses Werkzeug verfügt über eine so genannte "Suggest"-Funktion, errät also welchen Artikel man sucht, dabei werden lange Artikel bevorzugt. Es sollen damit auch die Server in den USA entlastet werden, da Fehleingaben vermieden werden. Mit einem Mausklick lässt sich das Programm in die Sidebar des Browsers integrieren, und ist somit schnell aufrufbar.(Kolossos 22.12.)"

Autosuggest article names for search

Search in several WP languages

No simultaneous multilingual search

Offline, dead.

Wdsearch by Magnus Manske

• 
  Wdsearch script screenshot enwiki
• 
  Wdsearch screenshot frwiki 2014-04-24
• 
  Wdsearch at occitan Wikipedia

• magnusmanske.de November 6, 2013
• en:MediaWiki:Wdsearch.js
• en:MediaWiki_talk:Wdsearch.js
• Wikitech mailinglist: Re: searching across all languages, November 2014

simultaneous multilingual Wikipedia search !

search in several Wikimedia projects

No WP search snippets

No images in search results

cluttered search interface

hack for a Knowledge Engine by Magnus Manske

• https://tools.wmflabs.org/magnustools/ke.html
• MagnusManske 15. Feb. 2016: "My cheapo “knowledge engine” searches across Wikimedia projects (single language, apart from @wikidata)"

Wikipedia search at external sites

instapedia

Screenshot of instapedia.com wikipedia search for "weltanschauung"-Text excerpts from Wikipedia articles

• http://web.archive.org/web/*/http://www.instapedia.com/
• instapedia = super fast multilingual wikipedia search with autocomplete.
• Instapedia - Mobile: Auto-complete style multi-language simultaneous Wikipedia search

simultaneous multilingual Wikipedia search!

Autocomplete search

Incremental search

Clean interface

Shows result with article lede

For mobile and non-mobile

No images in search results

Not integrated to WP, dump search?

Offline, dead.

WikiWand

Wikiwand on mobile

• en:WikiWand
• www.wikiwand.com/
  • www.wikiwand.com/en/Ebola_virus_epidemic_in_Liberia
• Wikiwand app for iOS
  • "Multi-language search (up to 3 languages simultaneously)"
  • "Amazingly accurate search results with pictures"

Autocomplete search

Incremental search

For mobile and non-mobile

Clean interface

Search results with images

Simultaneous multilingual Wikipedia search (up to 3 languages)

External site, Wikipedia mirrors

buk.io

• http://buk.io/ (by Minsu Kang)
• Wikipedia in unseen UI ycombinator

Fast en:Incremental search: search as you type

For mobile and non-mobile

Clean interface

Shows article lede as search results

Search results with images

search in several Wikipedia languages (english, korean, german)

No simultaneous multilingual Wikipedia search

External site, Wikipedia mirrors

wiksearch.com

• http://wiksearch.com
  • "Type a topic above to see links to Wikipedia articles that match — English only, for now. As you type, the green bar below the input box will fill with matching Wikipedia links sorted by how popular they are."
  • "This service uses the Datamuse API on a vocabulary of approximately 10 million Wikipedia titles and redirects." [5]
  • "The titles come from the publicly available Wikiedpedia page view counts for the last 7 days."
  • "Privacy: This website saves no data other than a count of the number of queries made to it. The API calls and the Wikipedia links use HTTPS. The API server does not log or remember your queries."
• by Doug Beeferman - 16.06.2015 (http://www.dougb.com) [6][7]

Wikiwix

• http://www.wikiwix.com/ = http://wikipediasearch.net/
• search language switching

search in several Wikipedia languages

search in several Wikimedia projects (Wikipedia / Wikisource / Wiktionary / Wikiquote / Wikibooks / Wikispecies / Wikiversity / Wikinews / Commons)

No simultaneous multilingual Wikipedia search

External site, Wikipedia mirrors

Wikilinks / WikiWeb App

• 
  Wikilinks / WikiWeb App

• WikiWeb App (wikilinks)
• WikiWeb "developed by Baltimore-based design startup Friends of the Web, is an alternative iPad or iPhone reader that provides a visual and interactive way to explore all knowledge as described at Wikipedia." [8]
• wikilinks3 @itunes: "Fast Wikipedia article search in all the languages of your choice at once."
• video, example search for "Federer"

Autosuggest search

Clean interface

Wiktionary search

Simultaneous multilingual Wikipedia search

External site, Wikipedia mirrors

Google multilingual search

Slightly over half of our readers (51%) said they specifically look for Wikipedia in search engine results like google (WMF Blog post)

• Google drives traffic to Wikipedia, but half of readers look for Wikipedia content WMF Blog Ayush Khanna on October 26, 2011
• Who reads which Wikipedia? The WMF's surprising stats Signpost/2013-04-01

• 2012, Google Inc. United States Patent 8,224,836 - "Searching in multiple languages".
• search results in multilingual format in google, (June 11th, 2012)
• http://www.2lingual.com/about.html
• en:Bilingual search engine
• "wiki" is a top search term! https://www.google.nl/trends/explore#q=wikipedia%2C%20wiki&cmpt=q

Stats

Search api analysis, Okeyes (WMF), 2015

• http://searchdata.wmflabs.org/ SearchData Discovery Dashboards
  • http://searchdata.wmflabs.org/metrics/

• meta:Research:Measuring User Search Satisfaction

• External Search API usage Oliver Keyes, 2015-05-20 ("Conclusions: (...) Drop in replacements for WP's interface potentially a big market now.")

Backlinks to Wikipedia

• Backlinks TO Wikipedia for search [9], www.0pii.com
  • 2013 wikilinks-corpus
  • https://commoncrawl.org/ is the biggest open one
  • http://80legs.com/giant-web-crawl.html is perhaps the biggest commercial one.

Search and Discovery (WMF)

Initial report: Wikipedia Portal Usage. By Oliver Keyes, April 2015

First Portal Test: Portal Search Box Improvements Test. Oliver Keyes et al., January 20, 2016

• mw:Discovery: "The Discovery department of Wikimedia Engineering is building the anonymous path of discovery to a trusted and relevant source of knowledge."
• mw:Search/Old/Design ("old" since 30 April 2015‎)
• wikimedia-search mailinglist started April 2015 - renamed September 2015:
  • wikimedia-discovery mailinglist (wikimedia-discovery @gmane)

• https://suggesty.wmflabs.org/suggest.html Completion suggester experiment
  • http://en-suggesty.wmflabs.org/wiki/Special:Search (using page rank)

• mw:User:TJones (WMF)/Notes
  • mw:User:TJones (WMF)/Notes/Cross Language Wiki Searching
  • mw:User:TJones (WMF)/Notes/Why People Use Search Engines
• LanguageSupportBrainstorming

• Discovery Dashboards:
  • http://discovery.wmflabs.org/portal/ #clickthrough_breakdown
  • http://discovery.wmflabs.org/external/ #traffic_summary #traffic_by_engine (it breaks down pageviews and shows how external search engines influence the traffic that hits wikis. Both a simple count of search-referred pageviews versus other pageviews, and a breakdown of how much traffic is coming from what specific search engines, is included.[10])
  • http://discovery.wmflabs.org/metrics/

• https://people.wikimedia.org/~jgirault/ (Top 10 based on navigator languages, Prototypes using React) [11]
  • proxy: http://portal.wmflabs.org
  • https://people.wikimedia.org/~jgirault/react-top10-fs/

See also

• THE WIKIPEDIA CORPUS by en:Mark Davies (linguist), Brigham Young University:

"This corpus contains the full text of the English version of Wikipedia, and it contains 1.9 billion words in more than 4.4 million articles. But this corpus allows you to search Wikipedia in a much more powerful way than is possible with the standard interface. You can search by word, phrase, part of speech, and synonyms. You can also find collocates (nearby words), and see re-sortable concordance lines for any word or phrase."

• Corpex - Corpora Explorer @tools.wmflabs:

Corpex ermöglicht das schnelle Durchsuchen aller Wörter der Wikipedia. Es zeigt die Umgebung von Buchstabenketten und Wörtern innerhalb des Textes einer Wikipedia Sprachversion. info, Tool: Corpex – Wikipedia Corpora Explorer by RENDER, video Screencast: Corpex

• Artikeltitel-Grep (Beta) by Jarry1250's:

"Dieses Werkzeug zeigt alle Artikeltitel, die einem regulären Ausdrucksmuster (Erklärung) entsprechen. Das Suchen mit einem regulären Ausdruck ist ressourcenintensiv. Falls überhaupt möglich, vergiss nicht die Präfix- und Suffix-Indikatoren (^ bzw. $)."

• Edit summary search by sigma:

"This tool searches through a user's contribution history and returns the edits made by that user if the edit summary contains the specified string."