From rainer.blome at gmx.de Tue Jul 1 10:06:13 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Tue, 01 Jul 2008 12:06:13 +0200 Subject: [search-ui] Loading tables on demand (2) In-Reply-To: References: <20080630160646.5960@gmx.net> <48692FFA.1040606@gmx.de> Message-ID: <486A0195.2050103@gmx.de> When loading files via HTTP, the HTTP status code check was not quite in the right place. This should affect only users with an elaborate locale name, for example "pt-BR". Fixed in attached i18n.js. Jeremie Miller wrote: > I love this stuff, thanks Rainer, being able to dynamically load any > language seems so futuristic, yet it's here :) :-) Aren't there other web applications out there that do this? i18n.js is quite independent, it could be used elsewhere. At first glance, there's only var url= search_server_to_use+"js/i18n/"+lang+".js"; // Should instead use the "dirname" of the i18n.js URL - how to? and var pages= ["search.html", "profile.html", "index.html"]; // Should instead detect a flag (property) set by the html. Rainer From rainer.blome at gmx.de Tue Jul 1 10:12:12 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Tue, 01 Jul 2008 12:12:12 +0200 Subject: [search-ui] Loading tables on demand (2) In-Reply-To: <486A0195.2050103@gmx.de> References: <20080630160646.5960@gmx.net> <48692FFA.1040606@gmx.de> <486A0195.2050103@gmx.de> Message-ID: <486A02FC.3070603@gmx.de> Forgot to attach the file :-/ For the same users, there was the possibility of an infinite loop, fixed as well. Rainer Rainer Blome wrote: > When loading files via HTTP, the HTTP status code check was not quite in > the right place. This should affect only users with an elaborate locale > name, for example "pt-BR". Fixed in attached i18n.js. -------------- next part -------------- A non-text attachment was scrubbed... Name: i18n.js.gz Type: application/postscript Size: 6672 bytes Desc: not available Url : http://lists.wikia.com/pipermail/search-ui/attachments/20080701/4a6aa5ef/attachment.eps From rainer.blome at gmx.de Tue Jul 1 10:39:35 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Tue, 01 Jul 2008 12:39:35 +0200 Subject: [search-ui] Determining the user's preferred content language Message-ID: <486A0967.8040109@gmx.de> Browsers allow the users to choose the content languages preferred. For example Firefox: Tools -> Options -> Advanced -> Languages. The top language in that list should be the default Wikia Search content language, and also the default UI language. How can we determine this? Is there a way to access this list from a script (I found none)? Another way would be to send a cheap request to a cooperating web server which sends back the request headers (or just the Accept-Languages header). There are such servers out there, I think, but re.search should use its own. Could the "now" kt-request be used for this purpose? Rainer From jeremie at jabber.org Tue Jul 1 15:18:09 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Tue, 1 Jul 2008 10:18:09 -0500 Subject: [search-ui] Determining the user's preferred content language In-Reply-To: <486A0967.8040109@gmx.de> References: <486A0967.8040109@gmx.de> Message-ID: I looked into this exact thing a while ago and came to the same conclusion, it's really sad that the browsers don't properly expose these preferences in the DOM. Yeah, one of the json requests will have to also return the languages header data, I'm not sure which is the best one, but now.js is a great candidate. Jer On Jul 1, 2008, at 5:39 AM, Rainer Blome wrote: > Browsers allow the users to choose the content languages preferred. > For example Firefox: Tools -> Options -> Advanced -> Languages. > > The top language in that list should be the default Wikia Search > content > language, and also the default UI language. > > How can we determine this? Is there a way to access this list from a > script (I found none)? > > Another way would be to send a cheap request to a cooperating web > server > which sends back the request headers (or just the Accept-Languages > header). > There are such servers out there, I think, but re.search should use > its > own. Could the "now" kt-request be used for this purpose? > > Rainer > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From phil.nelson at gmail.com Wed Jul 2 16:52:31 2008 From: phil.nelson at gmail.com (Christian Nelson) Date: Wed, 2 Jul 2008 12:52:31 -0400 Subject: [search-ui] Bookmarking (or permlinks) Message-ID: One thing I've had a problem with, and heard many times from other people, is the inability to link to a particular part of a result set in Wikia Search due to the infinite scrolling results list. I realize the results for any given query are not predictable or set in stone (due to the very nature of this project), but there still needs to be a way to link another person to a certain result range. Is there any work being done on this right now? My first idea would be to provide a link on the left side of the page every X results (user definable, default to 20?) that says something like "link to the following 20 results" or similar. I'm not sure how this would jibe with our use of the # to seperate the query string from the URL, but I guess that's for people smarter than I to figure out. Thoughts? - Phil Nelson http://philnelson.name From jeremie at jabber.org Wed Jul 2 23:02:27 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Wed, 2 Jul 2008 18:02:27 -0500 Subject: [search-ui] Bookmarking (or permlinks) In-Reply-To: References: Message-ID: Yeah, this has definitely come up... it's actually been stuck in the backlog of todo stuff for a while to have a floating div stuck in the corner somewhere when you're scrolling down with a few tools in it, like showing where you are at, jumping back to the top, making a permalink, etc. The permalinks to scrolled-down will probably have a funkier-looking url, I don't know what syntax would be best to use but it might just be looking for an optional special first character and using something like search.html#?key=val&foo=bar query string style... TBD :) Jer On Jul 2, 2008, at 11:52 AM, Christian Nelson wrote: > One thing I've had a problem with, and heard many times from other > people, is the inability to link to a particular part of a result set > in Wikia Search due to the infinite scrolling results list. I realize > the results for any given query are not predictable or set in stone > (due to the very nature of this project), but there still needs to be > a way to link another person to a certain result range. Is there any > work being done on this right now? > > My first idea would be to provide a link on the left side of the page > every X results (user definable, default to 20?) that says something > like "link to the following 20 results" or similar. I'm not sure how > this would jibe with our use of the # to seperate the query string > from the URL, but I guess that's for people smarter than I to figure > out. > > Thoughts? > > - Phil Nelson > http://philnelson.name > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From rainer.blome at gmx.de Sun Jul 6 09:42:20 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Sun, 06 Jul 2008 11:42:20 +0200 Subject: [search-ui] Determining the user's preferred content language In-Reply-To: References: <486A0967.8040109@gmx.de> Message-ID: <4870937C.3020404@gmx.de> Jeremie Miller wrote: > On Jul 1, 2008, at 5:39 AM, Rainer Blome wrote: >> send... request... which sends back ... Accept-Languages header. > Yeah, one of the json requests will have to also return the languages > header data, For MSIE and search_server_to_use="file:...", using an external request header echo server ("http://pgl.yoyo.org/http/browser-headers.php") works. For other configurations, the request is treated as cross-site and fails as it should. In this case, during testing, a hard-coded example string is used in place of a response. For production, a URL beginning with search_server_to_use would be used so that the request should succeed. However, I am unsatisfied with the complicated control flow. The additional asynchronous request adds another "layer" of continuation (closure) calling, which makes the code hard to debug, understand, get right and maintain. In addition, I would like to iterate over all languages given by Accept-Languages (currently, only the first one is considered). This would make the control flow even harder to follow. Currently I'm trying to remedy this using an event queue and a top-level dispatcher. This problem has probably been already solved by others. There may even be code for this in the current application. Any pointers? Rainer From jeremie at jabber.org Mon Jul 7 00:54:03 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Sun, 6 Jul 2008 19:54:03 -0500 Subject: [search-ui] Determining the user's preferred content language In-Reply-To: <4870937C.3020404@gmx.de> References: <486A0967.8040109@gmx.de> <4870937C.3020404@gmx.de> Message-ID: <241D0874-D0F8-4297-8923-B734CD1A06BF@jabber.org> I've not seen anything else work exactly like this w.r.t. dynamic language detection/loading... I think it's just going to be tough, and best to encapsulate it as much as possible with the hopes that some more modern browsers offer better shortcuts to do it w/o a roundtrip. And yes, whatever returns the A-L header should encode the whole value, not just the top-most preference. There's a whole batch of new features in SVN for the KT service and maybe we can get this one in there for the now.js too. Jer On Jul 6, 2008, at 4:42 AM, Rainer Blome wrote: > Jeremie Miller wrote: >> On Jul 1, 2008, at 5:39 AM, Rainer Blome wrote: >>> send... request... which sends back ... Accept-Languages header. >> Yeah, one of the json requests will have to also return the languages >> header data, > > For MSIE and search_server_to_use="file:...", using an external > request > header echo server ("http://pgl.yoyo.org/http/browser-headers.php") > works. For other configurations, the request is treated as cross-site > and fails as it should. In this case, during testing, a hard-coded > example string is used in place of a response. For production, a URL > beginning with search_server_to_use would be used so that the request > should succeed. > > However, I am unsatisfied with the complicated control flow. The > additional asynchronous request adds another "layer" of continuation > (closure) calling, which makes the code hard to debug, understand, get > right and maintain. In addition, I would like to iterate over all > languages given by Accept-Languages (currently, only the first one is > considered). This would make the control flow even harder to follow. > > Currently I'm trying to remedy this using an event queue and a top- > level > dispatcher. This problem has probably been already solved by others. > There may even be code for this in the current application. Any > pointers? > > Rainer > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From rainer.blome at gmx.de Mon Jul 7 11:28:06 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 07 Jul 2008 13:28:06 +0200 Subject: [search-ui] Minimizing the number of sites where scripts are pulled from Message-ID: <4871FDC6.10908@gmx.de> There is an interesting forum question in the wiki: http://search.wikia.com/wiki/Forum:Pull_javascript_from_one_domain_only%3F From rainer.blome at gmx.de Mon Jul 7 11:39:24 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 07 Jul 2008 13:39:24 +0200 Subject: [search-ui] UI internationalization To Do list Message-ID: <4872006C.9020405@gmx.de> Please note that the UI internationalization To Do list now resides at http://search.wikia.com/wiki/Talk:UI/Internationalization Anyone can now add, comment and prioritize. Rainer From rainer.blome at gmx.de Mon Jul 7 16:47:13 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 07 Jul 2008 18:47:13 +0200 Subject: [search-ui] Determining the user's preferred content language In-Reply-To: <241D0874-D0F8-4297-8923-B734CD1A06BF@jabber.org> References: <486A0967.8040109@gmx.de> <4870937C.3020404@gmx.de> <241D0874-D0F8-4297-8923-B734CD1A06BF@jabber.org> Message-ID: <48724891.7090606@gmx.de> Jeremie Miller wrote: > I've not seen anything else work exactly like this w.r.t. dynamic > language detection/loading... > On Jul 6, 2008, at 4:42 AM, Rainer Blome wrote: >> This problem has probably been already solved by others. I was referring to... >> ...event queue and a top-level dispatcher. >> There may even be code for this in the current application. Any >> pointers? Rainer From rainer.blome at gmx.de Wed Jul 9 08:07:18 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Wed, 09 Jul 2008 10:07:18 +0200 Subject: [search-ui] A built-in way to create new mini articles Message-ID: <487471B6.7050403@gmx.de> As far as I can see, in the UI there is currently no built-in way to add a new mini article. What is the plan regarding mini articles? Are they going to stay? If so (and I would like that), a way to add a new one should be built in. In cool/js/ui.js, replacing the "mini article not found" branch (sr.length == 0) of function maRender() with the below code will display 'Write a Mini Article about "the query"'. Rainer function maRender(j) { Q.ma = j; var sr = (j.title ? j.title: ""); if(sr.length == 0) { // Unfortunately, when there is no exiting mini // article, j is an empty object ("{}"), so we can not simply // use j.editurl. var query= document.location.hash; if ( undefined === query || ""===query) { $("mini-article-container").hide(); return; } if (query.charAt(0) === "#") { query= query.substring(1); } var article= query.replace(/ /g, "_"); // Need to uppercase the first char? var editurl= "http://search.wikia.com/index.php?title=Mini:"+ article+"&action=edit"; var in_new_window='target="_blank"'; //var in_new_window='rel="external"'; // See below $("mini-article-container").innerHTML = gettext("Write a Mini Article about")+' "'+ ''+query+'"'; $("mini-article-container").show(); return; } ... } // Here's one way to get 'rel="external"' to work, // but I did not bother, since the HTML is generated anyway: function activate_external_links() { if ( ! document.getElementsByTagName) { return; } var anchors= document.getElementsByTagName("a"); for (var i= anchors.length-1; i >= 0; --i) { var a= anchors[i]; if (a.getAttribute("href") && a.getAttribute("rel") === "external") { a.target= "_blank"; } } } //activate_external_links(); From jeremie at jabber.org Wed Jul 9 20:00:06 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Wed, 9 Jul 2008 15:00:06 -0500 Subject: [search-ui] A built-in way to create new mini articles In-Reply-To: <487471B6.7050403@gmx.de> References: <487471B6.7050403@gmx.de> Message-ID: <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> There's been a lot of chatter (over IM and some 1-1 emails) about what we can do to make a generalized "widget" framework that would be able to host the functionality of a mini article and many other kinds of general widgets (like calculator, weather, the typical "meta" kinds of things that there can be general answers for and that the mini article fits into). So since there is the outstanding desire to "do it right" and not just make a special case of the mini articles, the MA add isn't getting as much love until the rest of it gets scoped out. So on this thread, what other kinds of general-answer types of things should a widget framework try to support? Jer On Jul 9, 2008, at 3:07 AM, Rainer Blome wrote: > As far as I can see, in the UI there is currently no built-in way to > add > a new mini article. > What is the plan regarding mini articles? Are they going to stay? > If so (and I would like that), a way to add a new one should be > built in. > > In cool/js/ui.js, replacing the "mini article not found" branch > (sr.length == 0) of function maRender() with the below code will > display > 'Write a Mini Article about "the query"'. > > Rainer > > function maRender(j) > { > Q.ma = j; > var sr = (j.title ? j.title: ""); > if(sr.length == 0) { > // Unfortunately, when there is no exiting mini > // article, j is an empty object ("{}"), so we can not simply > // use j.editurl. > var query= document.location.hash; > if ( undefined === query || ""===query) { > $("mini-article-container").hide(); > return; > } > if (query.charAt(0) === "#") { query= query.substring(1); } > var article= query.replace(/ /g, "_"); > // Need to uppercase the first char? > var editurl= "http://search.wikia.com/index.php?title=Mini:"+ > article+"&action=edit"; > var in_new_window='target="_blank"'; > //var in_new_window='rel="external"'; // See below > $("mini-article-container").innerHTML = > gettext("Write a Mini Article about")+' "'+ > ''+query+'"'; > $("mini-article-container").show(); > return; > } > ... > } > > // Here's one way to get 'rel="external"' to work, > // but I did not bother, since the HTML is generated anyway: > function activate_external_links() { > if ( ! document.getElementsByTagName) { return; } > var anchors= document.getElementsByTagName("a"); > for (var i= anchors.length-1; i >= 0; --i) { > var a= anchors[i]; > if (a.getAttribute("href") && > a.getAttribute("rel") === "external") { > a.target= "_blank"; > } > } > } > //activate_external_links(); > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From phil.nelson at gmail.com Wed Jul 9 20:04:25 2008 From: phil.nelson at gmail.com (Phil Nelson) Date: Wed, 9 Jul 2008 16:04:25 -0400 Subject: [search-ui] A built-in way to create new mini articles In-Reply-To: <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> References: <487471B6.7050403@gmx.de> <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> Message-ID: <3bcb4e010807091304q4d131f69jc7a778d531b875a6@mail.gmail.com> Movie and music showtimes and/or "ratings"? Maybe also tap into upcoming.org but that brings with it some privacy concerns, naturally. If the topic has a related Wikia wiki, maybe grab that? I assume that is in the works... - Phil ---- http://philnelson.name On Wed, Jul 9, 2008 at 4:00 PM, Jeremie Miller wrote: > So on this thread, what other kinds of general-answer types of things > should a widget framework try to support? From jeremie at jabber.org Wed Jul 9 20:09:09 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Wed, 9 Jul 2008 15:09:09 -0500 Subject: [search-ui] A built-in way to create new mini articles In-Reply-To: <3bcb4e010807091304q4d131f69jc7a778d531b875a6@mail.gmail.com> References: <487471B6.7050403@gmx.de> <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> <3bcb4e010807091304q4d131f69jc7a778d531b875a6@mail.gmail.com> Message-ID: *nod*, exactly the right line of thinking :) On Jul 9, 2008, at 3:04 PM, Phil Nelson wrote: > Movie and music showtimes and/or "ratings"? Maybe also tap into > upcoming.org but that brings with it some privacy concerns, naturally. > If the topic has a related Wikia wiki, maybe grab that? I assume that > is in the works... > > - Phil > ---- > http://philnelson.name > > > On Wed, Jul 9, 2008 at 4:00 PM, Jeremie Miller > wrote: >> So on this thread, what other kinds of general-answer types of things >> should a widget framework try to support? > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From rainer.blome at gmx.de Wed Jul 9 22:45:00 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Thu, 10 Jul 2008 00:45:00 +0200 Subject: [search-ui] A built-in way to create new mini articles In-Reply-To: <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> References: <487471B6.7050403@gmx.de> <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> Message-ID: <48753F6C.5060306@gmx.de> Jeremie Miller wrote: > [design of] a generalized "widget" framework that would be able > to host the functionality of a mini article and many other kinds of > general widgets ... MA add > isn't getting as much love until the rest of it gets scoped out. I take that as a "Yes, mini articles are going to stay, but MA add will not be done right now." Sounds like this is a while off :-(. I fear that by then the Mini Article community will have dissolved (if it hasn't already) and need to start afresh. Rainer From rainer.blome at gmx.de Wed Jul 9 22:58:07 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Thu, 10 Jul 2008 00:58:07 +0200 Subject: [search-ui] A built-in way to create new mini articles In-Reply-To: <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> References: <487471B6.7050403@gmx.de> <8F891B94-4F2F-4D87-A7DC-FE86953D5323@jabber.org> Message-ID: <4875427F.40906@gmx.de> Jeremie Miller wrote: > So on this thread, what other kinds of general-answer types of things > should a widget framework try to support? o dictionaries o thesauri o zip codes o Man pages :-) o news o by media type (images, videos, audio) o generic "link-through" to other search services - "open search" it is called, I think? This would make the "See results from" just another widget. Enable the users to invent their own ones. Make it user-extensible. Rainer From newsmarkie at googlemail.com Fri Jul 18 19:35:15 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Fri, 18 Jul 2008 20:35:15 +0100 Subject: [search-ui] Widgets Message-ID: I know i havent been to active round here recently (been really busy, sorry :-( ) but one thing that i have noticed on the mailing lists that really interested me is the widgets ideas. IMO these are really key things that, shouldnt :-p, be too hard to implement. Ive had a brief look at one of the widgets which i look for alot, which is weather (although i dont know why, it always seems to be rain :-p) NWS / NOAA provide an XML output of weather for all US cities and most international cities, and with it being a US gov org all its content is therefore PD and totally free to use and copy etc (not a lawyer though). this we could then implement easily using a JS call from the search var, with or without looking for a "weather: XXX" keyword and then output next to the search results, maybe below the adverts. this could also be JS implemented to be hide able etc with more widgets....... the problem with NOAA IMO is that it requires calls to use lat/long co-ordinates for the calls, thus required more geo-location stuff which makes this harder and harder. so then i looked for alternative APIs which could be used and came up with weather.com's feed. this requires registration, but is then free to use but requires 5 links back and a logo, which may or maynot be acceptable to us?? this however does give the ease of being able to call by city name, removing the need for geo-locations stuff. however the balance needs to come between coding and technical possibility and the "freeness" of the content. also im guessing there are many different ideas for widgets so also please feel free to give thoughts on this one, and any others that spring to mind as being wanted. my thoughts were: Open Street Map contents when searching locations (ie a map of the location in the world/country/are when searching for a city) or cinema times for jimbo etc etc regards (and apologies for the lengthy message) mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080718/b16c0d24/attachment.html From thindil2 at gmail.com Fri Jul 18 19:54:24 2008 From: thindil2 at gmail.com (Bartek Jasicki) Date: Fri, 18 Jul 2008 21:54:24 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages Message-ID: <20080718215424.11a4b091@vallheru.vallheru> Hi Jeremie push me to write this mail here so this all his fault ;) On all Wikia Search pages is problem with show non-Latin characters (problem was reported by few people). I think problem is with strange character encoding in files - its good shown here: http://people.swlabs.org/~bartek/websvn/filedetails.php?repname=SearchUI&path=%2Fcool%2Fjs%2Fi18n%2Fi18n.js or if you all characters are ok, look at this screenshot: http://people.swlabs.org/~bartek/images/i18n.js.png This is view of file with set UTF-8 encoding. I don't know German, but im sure that this weird symbols are not German letters ;) I try open this file under GNU Emacs and Gedit (both are graphical editors) - both show weird characters than normal (but Nano and editor in Midnight Commander show all ok - both are console editors). So maybe change encoding in files fix this problem? I mean encoding in which file are saved on disk, not setting for browsers. Regards Bartek Jasicki From newsmarkie at googlemail.com Fri Jul 18 20:39:51 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Fri, 18 Jul 2008 21:39:51 +0100 Subject: [search-ui] i18n errors Message-ID: http://c.imagehost.org/0228/untitled_4.jpg button is borked. but i am loving the on the fly translations :-) regards mark -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080718/941c29bb/attachment.html From jeremie at jabber.org Sat Jul 19 19:18:17 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Sat, 19 Jul 2008 14:18:17 -0500 Subject: [search-ui] Y Combinator: Startup Ideas We'd Like to Fund Message-ID: <3739242E-7F4F-4ED0-9F69-D01565292C77@jabber.org> http://ycombinator.com/ideas.html A fun list by itself, but idea #16 is just begging at us to do something fun and interesting here :) 16. A form of search that depends on design. Google doesn't have a lot of weaknesses. One of the biggest is that they have no sense of design. They do the next best thing, which is to keep things sparse. But if there were a kind of search that depended a lot on design, a startup might actually be able to beat Google at search. I don't know if there is, but if you do, we'd love to hear from you. Jer PS: Is idea #23 not also what we're doing, social search? From rainer.blome at gmx.de Sat Jul 19 23:12:51 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Sun, 20 Jul 2008 01:12:51 +0200 Subject: [search-ui] i18n errors In-Reply-To: References: Message-ID: <488274F3.9030002@gmx.de> Mark (Markie) wrote: > http://c.imagehost.org/0228/untitled_4.jpg button is borked. Fixed, but not ready for public consumption. Jer, you offered a SVN account with checkin capability. Does this offer still stand? That would allow for individual fixes independent of bigger transformations. > but i am loving the on the fly translations :-) The breakage makes Jeff's technique used to implement on the fly translation visible :-). In this case, it is incorrectly applied. Rainer From jeremie at jabber.org Sat Jul 19 23:37:36 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Sat, 19 Jul 2008 18:37:36 -0500 Subject: [search-ui] i18n errors In-Reply-To: <488274F3.9030002@gmx.de> References: <488274F3.9030002@gmx.de> Message-ID: <54141F07-BB31-4F62-B61D-163F1C74B276@jabber.org> > Jer, you offered a SVN account with checkin capability. > Does this offer still stand? Definitely, just ping me via xmpp:jer at jabber.org and I'll get you set up. Jer From rainer.blome at gmx.de Sun Jul 20 08:12:37 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Sun, 20 Jul 2008 10:12:37 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <20080718215424.11a4b091@vallheru.vallheru> References: <20080718215424.11a4b091@vallheru.vallheru> Message-ID: <4882F375.3000103@gmx.de> Summary: o For an out-of-the-box browser, things should just work. o The encoding to be used by the browser is specified in the HTML files. As far as I can see, the HTML specifies the correct encoding (UTF-8). o Browser-side selection of the encoding can interfere. Turn off "automatic guessing" of the encoding (the default should be "off"). Disclaimer: I am no expert in this area. If you think anything is wrong with what I wrote, it may very well be. In that case, please point it out. In this mail, I will try to briefly give direct answers. More information can be found at http://search.wikia.com/index.php?title=UI/Internationalization Bartek Jasicki wrote: > On all Wikia Search pages is problem with show non-Latin characters Sigh. There are many things that can go wrong. I assume that you mean non-ASCII characters. For the record, when I point the browser to any Wikia Search page, for example http://re.search.wikia.com/search.html , it displays the user interface text correctly, no matter with which browser. In particular, when I open the language dialog, the native language names in the drop-down list are displayed in the native scripts. Note that I can not verify that the language names are displayed correctly (since I can not read most languages :-). If the search.html page is displayed with *multiple* "funny characters" where there should be only one character, it is most likely that the browser does not interpret the content as UTF-8-encoded. > (problem was reported by few people). Where can we see the reported issues? > I think problem is with strange character encoding in files - What do you mean by "strange"? We need to use Unicode. See below. > - its good shown here: > http://people.swlabs.org/~bartek/websvn/filedetails.php?repname=SearchUI&path=%2Fcool%2Fjs%2Fi18n%2Fi18n.js You mean that the problem is visible on that page? (Your English is sometimes confusing. It's "it's", not "its", when you mean "it is". It's "its" when you mean "it belongs to it". ) That URL shows a view of the file i18n.js, generated by WebSVN. The view appears broken. Somehow, the view does not appear to show the content of the file. It may be that WebSVN has a false understanding of the file's encoding. Whether this view is displayed OK or not is not relevant to whether the file works when deployed. Better load the file itself: http://re.search.wikia.com/js/i18n/i18n.js Using Mozilla or Safari, the code can be viewed directly in the browser. Using Internet Explorer 6, it must be saved to a file in order to be viewed. Even loading the file explicitly is not the same as loading it from a script tag. What counts is whether it works when loaded from the context of an HTML page. (As an aside, IE offers to either save the file or interpret the code in the file. It does not offer to view the code. The other browsers don't offer any options, they simply display the file's content. Why? The mime-type displayed by SeaMonkey's Page Info dialog is "application/x-javascript", does this play a role? ) > or if you all characters are ok, "If you all characters are OK"??? :-) > look at this screenshot: > http://people.swlabs.org/~bartek/images/i18n.js.png > > This is view of file with set UTF-8 encoding. "View of file", yes. "UTF-8 encoding", no. According to the URL visible in the screenshot, it shows the same "apparently broken WebSVN view" of the file, not the file itself. > I don't know German, but im sure that this weird symbols are not > German letters ;) Correct :^) > I try open this file under GNU Emacs and Gedit (both > are graphical editors) - both show weird characters than normal (but > Nano and editor in Midnight Commander show all ok - both are console > editors). The editor needs to both support Unicode and use the right encoding for the file. GNU Emacs 22 supports Unicode, version 21 doesn't. Gedit I don't know. Your console editors apparently support Unicode encoding. Do they display correctly all characters in the language list? > So maybe change encoding in files fix this problem? I mean > encoding in which file are saved on disk, not setting for browsers. It may be that there is an encoding that works in more cases, in particular an encoding that works when auto-detection of the encoding has been switched on. Rainer From thindil2 at gmail.com Sun Jul 20 09:22:09 2008 From: thindil2 at gmail.com (Bartek Jasicki) Date: Sun, 20 Jul 2008 11:22:09 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <4882F375.3000103@gmx.de> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> Message-ID: <20080720112209.60094c52@vallheru.vallheru> 2008-07-20, 10:12:37 Rainer Blome wrote: > > Bartek Jasicki wrote: > > On all Wikia Search pages is problem with show non-Latin characters > > Sigh. There are many things that can go wrong. I assume that you mean > non-ASCII characters. > > For the record, when I point the browser to any Wikia Search page, for > example http://re.search.wikia.com/search.html , it displays the user > interface text correctly, no matter with which browser. In > particular, when I open the language dialog, the native language names > in the drop-down list are displayed in the native scripts. Note that > I can not verify that the language names are displayed correctly > (since I can not read most languages :-). > > If the search.html page is displayed with *multiple* "funny > characters" where there should be only one character, it is most > likely that the browser does not interpret the content as > UTF-8-encoded. > If you mean interface texts - yes, this all is ok (i even switch to German language - everything looks good). But if you mean texts inserted by users (walls, url's edits) there most of non-Latin (or as you prefer non-ASCII ;) ) characters are weird. You can see it on my profile: http://re.search.wikia.com/profile.html?user=Thindil Interesting - ealier (around 3.6.2008) all works good (earlier posts on my wall). > > (problem was reported by few people). > > Where can we see the reported issues? > I saw it on http://search.wikia.com/wiki/Special:ProblemReports Sorry for me is hard to find something there thus i cannot give you accurate links. > > I think problem is with strange character encoding in files - > > What do you mean by "strange"? We need to use Unicode. See below. > > > - its good shown here: > > > http://people.swlabs.org/~bartek/websvn/filedetails.php?repname=SearchUI&path=%2Fcool%2Fjs%2Fi18n%2Fi18n.js > > You mean that the problem is visible on that page? > > (Your English is sometimes confusing. It's "it's", not "its", when > you mean "it is". It's "its" when you mean "it belongs to it". ) > I plan use Urdu - probably been more clear ;) But seriously: sorry for that. > That URL shows a view of the file i18n.js, generated by WebSVN. > The view appears broken. > Somehow, the view does not appear to show the content of the file. > It may be that WebSVN has a false understanding of the file's > encoding. > > Whether this view is displayed OK or not is not relevant to whether > the file works when deployed. > Better load the file itself: > > http://re.search.wikia.com/js/i18n/i18n.js > > Using Mozilla or Safari, the code can be viewed directly in the > browser. Using Internet Explorer 6, it must be saved to a file in > order to be viewed. > Yes, you have right, my mistake, sorry. > > > or if you all characters are ok, > > "If you all characters are OK"??? :-) > Ok, ok, I back to school ;) > > The editor needs to both support Unicode and use the right encoding > for the file. GNU Emacs 22 supports Unicode, version 21 doesn't. > Gedit I don't know. Your console editors apparently support Unicode > encoding. Do they display correctly all characters in the language > list? > All of this editors support Unicode without problem (Emacs ver 22.2.1, Gedit ver 2.22.1) - tested with Polish and German characters. For me this is very strange situation. > > So maybe change encoding in files fix this problem? I mean > > encoding in which file are saved on disk, not setting for browsers. > > It may be that there is an encoding that works in more cases, in > particular an encoding that works when auto-detection of the encoding > has been switched on. > Some time ago i saw similar problem when file was first saved with one character encoding and later saved with other after edit. But maybe this is wrong way to solution. There is problem with inserted by users texts - maybe is something wrong with record to or read from database? Problem was even with old interface (this from 7.1.2008) Bartek From avatar at wikia.com Sun Jul 20 10:09:51 2008 From: avatar at wikia.com (Tim Bartel) Date: Sun, 20 Jul 2008 12:09:51 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <20080720112209.60094c52@vallheru.vallheru> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> Message-ID: Hi, 2008/7/20 Bartek Jasicki : > If you mean interface texts - yes, this all is ok (i even switch to > German language - everything looks good). Nope. While * http://re.search.wikia.com/recent.html * http://re.search.wikia.com/fun/fun.html * http://re.search.wikia.com/about/about.html * http://re.search.wikia.com/help/help.html and * http://re.search.wikia.com/invite.html appear to show a correct menu bar * http://re.search.wikia.com/legal/privacy.html and * http://re.search.wikia.com/legal/legal.html have broken umlauts. Bye, Tim. From rainer.blome at gmx.de Sun Jul 20 20:40:35 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Sun, 20 Jul 2008 22:40:35 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> Message-ID: <4883A2C3.5060907@gmx.de> Tim Bartel wrote: > * http://re.search.wikia.com/legal/privacy.html and > * http://re.search.wikia.com/legal/legal.html > have broken umlauts. > The encoding spec in the HTML source is missing for these pages. This line needs to be in the header: Note that the closing slash is missing from those pages that already have the Content-Type line. Rainer From rainer.blome at gmx.de Sun Jul 20 22:11:05 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 21 Jul 2008 00:11:05 +0200 Subject: [search-ui] XHTML? Message-ID: <4883B7F9.9070103@gmx.de> Trying to find out whether the Content-Type meta-tag should be terminated with a slash or not, I fed http://re.search.wikia.com/search.html to an XHTML validator: http://validator.w3.org/check?uri=http%3A%2F%2Fre.search.wikia.com%2Fsearch.html&charset=%28detect+automatically%29&doctype=Inline&group=0 The result is, err, hilarious? :-) The first error is in line 2, can you spot the error?: Hint: the interesting character is after the 2nd "x". Is this intentional, perhaps a way to force lenient browser behavior? ;-) Seriously, are the deviations from the standard intentional? Rainer From phil.nelson at gmail.com Sun Jul 20 22:47:11 2008 From: phil.nelson at gmail.com (Christian Nelson) Date: Sun, 20 Jul 2008 18:47:11 -0400 Subject: [search-ui] XHTML? In-Reply-To: <4883B7F9.9070103@gmx.de> References: <4883B7F9.9070103@gmx.de> Message-ID: <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> Good catch, Rainer. Looks like a simple brainfart to me ; ) - Phil On Jul 20, 2008, at 6:11 PM, Rainer Blome wrote: > Seriously, are the deviations from the standard intentional? From rainer.blome at gmx.de Sun Jul 20 23:25:42 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 21 Jul 2008 01:25:42 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <20080720112209.60094c52@vallheru.vallheru> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> Message-ID: <4883C976.3040002@gmx.de> Bartek Jasicki wrote: > 2008-07-20, 10:12:37 > Rainer Blome wrote: >> Bartek Jasicki wrote: >>> On all Wikia Search pages is problem with show non-Latin characters > If you mean interface texts - yes, this all is ok Wellll... your reference to i18n.js certainly got me on the wrong track then. > (i even switch to German language - everything looks good). Glad to hear that. > But if you mean texts inserted > by users (walls, url's edits) there most of non-Latin (or as you prefer > non-ASCII ;) ) characters are weird. You can see it on my profile: > > http://re.search.wikia.com/profile.html?user=Thindil Yes, I've noticed that, too. Somewhere on the way from user input in the browser to kt server and back to display in the browser, the encoding is messed up. Note that any of the below questions may be meanigless, based on wrong assumptions. o How does character input in the browser work? o How does the browser produce a string from the characters input by the user? o Does a browser expect a particular encoding being input? If so, which one? > Interesting - ealier (around 3.6.2008) all works good (earlier posts on > my wall). Is this maybe because the Content-Type meta-tag has been added, changing the default encoding (whichever that was) to UTF-8? Assuming that the default input encoding is not UTF-8, but, say, UTF-16, it may be that UTF-16 input is now interpreted as UTF-8. >> > (problem was reported by few people). >> >> Where can we see the reported issues? >> > > I saw it on http://search.wikia.com/wiki/Special:ProblemReports > Sorry for me is hard to find something there thus i cannot give you > accurate links. When I go to that URL, no reports are shown. >>> I think problem is with strange character encoding in files - What do you mean by "strange"? > All of this editors support Unicode without problem (Emacs ver 22.2.1, > Gedit ver 2.22.1) - tested with Polish and German characters. For me > this is very strange situation. What do you mean by "strange"? Please be more specific. Please have a look at http://search.wikia.com/wiki/UI/Internationalization#Editing_translation_tables and see if that helps. > Some time ago i saw similar problem when file was first saved with one > character encoding and later saved with other after edit. When a file is loaded into an editor with the wrong encoding, then the encoding is switched to the correct one, and then the files is saved, the file will be messed up. It happened to me once. Back up often! > But > maybe this is wrong way to solution. There is problem with inserted by > users texts - maybe is something wrong with record to or read from > database? Problem was even with old interface (this from 7.1.2008) Currently both the user interface texts and the search results texts are affected by the encoding selected by the browser. Mabe it is possible to use different encodings for different parts of a page? Otherwise we need to agree on a common encoding used by all parts of the pages. I tried to change the encoding of search.html and i18n.js to UTF-16. However, Firefox then can not read the other .js files (prototype.js etc.). Unfortunately, this seems to be an all-or nothing situation. My impression is that without any other information, the browser expects all .js files to be encoded using the same encoding as the one specified in search.html . So I would need to convert all .js files to UTF-16 to properly test this approach. For this, some form of batch converter would be nice. Anyone knows hoe to do that? However, I think that it would not work even then, because the content served by the ISC servers is probably not encoded as UTF-16. Can the desired encoding be specified in the KT requests? Rainer From thindil2 at gmail.com Mon Jul 21 11:37:27 2008 From: thindil2 at gmail.com (Bartek Jasicki) Date: Mon, 21 Jul 2008 13:37:27 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <4883C976.3040002@gmx.de> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> Message-ID: <20080721133727.553b687d@vallheru.vallheru> 2008-07-21, 01:25:42 Rainer Blome wrote: > Bartek Jasicki wrote: > > 2008-07-20, 10:12:37 > > Rainer Blome wrote: > >> Bartek Jasicki wrote: > >>> On all Wikia Search pages is problem with show non-Latin > >>> characters > > If you mean interface texts - yes, this all is ok > > Wellll... your reference to i18n.js certainly got me on the wrong > track then. > Yes, my mistake, but there is a problem with encoding in this file - downloaded by browser (Firefox 3.0.1) and by svn checkout. GNU Emacs and Gedit don't autodetect file encoding - even if earlier UTF-8 is set as default. Reproduction on GNU Emacs: 1. Download http://re.search.wikia.com/js/i18n/i18n.js by browser 2. Open GNU Emacs and set (if not is set) UTF-8 encoding 3. Open downloaded file. Character encoding automatically change to 'nil' (unknown? ASCII?) This is problem - you can test for example pl.po file http://svn.swlabs.org/grubng/trunk/csharp/Grubngtranslations/pl.po All editors detect correct character encoding in this file. > > But if you mean texts inserted > > by users (walls, url's edits) there most of non-Latin (or as you > > prefer non-ASCII ;) ) characters are weird. You can see it on my > > profile: > > > > http://re.search.wikia.com/profile.html?user=Thindil > > Yes, I've noticed that, too. Somewhere on the way from user input in > the browser to kt server and back to display in the browser, the > encoding is messed up. > > Note that any of the below questions may be meanigless, based on > wrong assumptions. > > o How does character input in the browser work? > > o How does the browser produce a string from the characters input by > the user? > > o Does a browser expect a particular encoding being input? If so, > which one? > I think, this is wrong way to resolve problem - if on other pages all is ok (i mean other pages on net, not on Wikia Search, but for example Wiki works good) then problem is in files not in browser. Browser don't expect any particular encoding on input (often use system default) but if page does not include information about character encoding, browser must make a blind assumption for show that page. > > >> > (problem was reported by few people). > >> > >> Where can we see the reported issues? > >> > > > > I saw it on http://search.wikia.com/wiki/Special:ProblemReports > > Sorry for me is hard to find something there thus i cannot give you > > accurate links. > > When I go to that URL, no reports are shown. > Yes, i know, this bugtrack is little annoying ;) I thought that you (or someone other) have more options to browse reports. I can only browse bug by bug (if of course i know number) - example: http://search.wikia.com/wiki/Special:ProblemReports/11163 > >>> I think problem is with strange character encoding in files - > > What do you mean by "strange"? > > > All of this editors support Unicode without problem (Emacs ver > > 22.2.1, Gedit ver 2.22.1) - tested with Polish and German > > characters. For me this is very strange situation. > > What do you mean by "strange"? Please be more specific. Please have a > look at > http://search.wikia.com/wiki/UI/Internationalization#Editing_translation_tables > and see if that helps. > Strange = i cannot detect what exactly encoding is used - i try UTF-8, ISO, Windows and Mac encoding - all are incorrect. > > But > > maybe this is wrong way to solution. There is problem with > > inserted by users texts - maybe is something wrong with record to > > or read from database? Problem was even with old interface (this > > from 7.1.2008) > > Currently both the user interface texts and the search results texts > are affected by the encoding selected by the browser. Mabe it is > possible to use different encodings for different parts of a page? > > Otherwise we need to agree on a common encoding used by all parts of > the pages. > > I tried to change the encoding of search.html and i18n.js to UTF-16. > However, Firefox then can not read the other .js files (prototype.js > etc.). Unfortunately, this seems to be an all-or nothing situation. > My impression is that without any other information, > the browser expects all .js files to be encoded > using the same encoding as the one specified in search.html . > So I would need to convert all .js files to UTF-16 to properly test > this approach. For this, some form of batch converter would be nice. > Anyone knows hoe to do that? > If you want convert to any character encoding, all files must have this same encoding, without exception. Maybe not all Wikia Search files are in UTF-8? For character conversion tools: http://en.wikipedia.org/wiki/Character_encoding#Character_conversion_tools Bartek From david.pean at gmail.com Mon Jul 21 15:38:00 2008 From: david.pean at gmail.com (David Pean) Date: Mon, 21 Jul 2008 11:38:00 -0400 Subject: [search-ui] XHTML? In-Reply-To: <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> References: <4883B7F9.9070103@gmx.de> <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> Message-ID: Hrm, not sure how that happened...we are fixing it on dev and will have it deployed live Dave On Sun, Jul 20, 2008 at 6:47 PM, Christian Nelson wrote: > Good catch, Rainer. Looks like a simple brainfart to me ; ) > > - Phil > > On Jul 20, 2008, at 6:11 PM, Rainer Blome wrote: > >> Seriously, are the deviations from the standard intentional? > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From jeremie at jabber.org Mon Jul 21 18:16:34 2008 From: jeremie at jabber.org (Jeremie Miller) Date: Mon, 21 Jul 2008 13:16:34 -0500 Subject: [search-ui] Widgets In-Reply-To: References: Message-ID: <21A55959-1867-4F94-A510-33F9534E8B12@jabber.org> My response is pretty much a UI thing so I thought I'd switch it to this list :) For me, the in-my-head debate has been about how with our social/ editable results page, to best serve up widget-able data, and I think I've convinced myself that any widget results should be shown exactly like a normal search result (with a title, sumamry, link, and optional sub-annotations). It also has to be subject to all the normal editing/ ranking/deletion/etc for any keyword it shows up under. The alternative is that they are a special case kind of like how other search engines show them when you search for weather+zip (in it's own UI above the results)... while I think this may be "prettier" I really think it's important to make these results be part of the same social framework as everything else. Maybe there will be multiple weather widgets even, for instance? I just wanted to put these ideas out here before we started implementing anything for widgets, as it's a pretty strategic thing to make them all return a search result instead of some special case UI for each. Jer On Jul 18, 2008, at 2:35 PM, Mark (Markie) wrote: > I know i havent been to active round here recently (been really > busy, sorry :-( ) but one thing that i have noticed on the mailing > lists that really interested me is the widgets ideas. IMO these are > really key things that, shouldnt :-p, be too hard to implement. Ive > had a brief look at one of the widgets which i look for alot, which > is weather (although i dont know why, it always seems to be rain :-p) > > NWS / NOAA provide an XML output of weather for all US cities and > most international cities, and with it being a US gov org all its > content is therefore PD and totally free to use and copy etc (not a > lawyer though). this we could then implement easily using a JS call > from the search var, with or without looking for a "weather: XXX" > keyword and then output next to the search results, maybe below the > adverts. this could also be JS implemented to be hide able etc with > more widgets....... the problem with NOAA IMO is that it requires > calls to use lat/long co-ordinates for the calls, thus required more > geo-location stuff which makes this harder and harder. > > so then i looked for alternative APIs which could be used and came > up with weather.com's feed. this requires registration, but is then > free to use but requires 5 links back and a logo, which may or > maynot be acceptable to us?? this however does give the ease of > being able to call by city name, removing the need for geo-locations > stuff. > > however the balance needs to come between coding and technical > possibility and the "freeness" of the content. > > also im guessing there are many different ideas for widgets so also > please feel free to give thoughts on this one, and any others that > spring to mind as being wanted. my thoughts were: Open Street Map > contents when searching locations (ie a map of the location in the > world/country/are when searching for a city) or cinema times for > jimbo etc etc > > regards (and apologies for the lengthy message) > > mark From rainer.blome at gmx.de Mon Jul 21 20:42:58 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Mon, 21 Jul 2008 22:42:58 +0200 Subject: [search-ui] XHTML? In-Reply-To: <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> References: <4883B7F9.9070103@gmx.de> <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> Message-ID: <4884F4D2.1030903@gmx.de> Christian Nelson wrote: > Good catch, Rainer. Looks like a simple brainfart to me ; ) The "xmtml" is just a typo, all right. What's with the 56 other validation errors? The documents are all marked as "strict", yet do not validate. Marking the documents as "non-strict" ("transitional"?) would at least declare that the documents do not strictly conform to the standard. Are there intentional deviations from the XHTML standard (in which I am no expert, mind you), for example for the sake of browser compatibility? If so, which deviations are intentional? If not, some of the deviations might be fixed as we go along... Rainer > - Phil > > On Jul 20, 2008, at 6:11 PM, Rainer Blome wrote: >> http://validator.w3.org/check?uri=http%3A%2F%2Fre.search.wikia.com%2Fsearch.html&charset=%28detect+automatically%29&doctype=Inline&group=0 >> >> Seriously, are the deviations from the standard intentional? > From aaron.wright at gmail.com Mon Jul 21 21:27:58 2008 From: aaron.wright at gmail.com (Aaron Wright) Date: Mon, 21 Jul 2008 17:27:58 -0400 Subject: [search-ui] XHTML? In-Reply-To: <4884F4D2.1030903@gmx.de> References: <4883B7F9.9070103@gmx.de> <45C62208-8EA9-445F-BA0F-814D76A14676@zerolives.org> <4884F4D2.1030903@gmx.de> Message-ID: I'll try to eliminate the validation errors. If I can't, I'll mark the documents as "non-strict", give me a couple of days. --Aaron On Mon, Jul 21, 2008 at 4:42 PM, Rainer Blome wrote: > Christian Nelson wrote: >> Good catch, Rainer. Looks like a simple brainfart to me ; ) > The "xmtml" is just a typo, all right. > > What's with the 56 other validation errors? > > The documents are all marked as "strict", yet do not validate. > Marking the documents as "non-strict" ("transitional"?) would at least > declare > that the documents do not strictly conform to the standard. > > Are there intentional deviations from the XHTML standard > (in which I am no expert, mind you), > for example for the sake of browser compatibility? > If so, which deviations are intentional? > If not, some of the deviations might be fixed as we go along... > > Rainer >> - Phil >> >> On Jul 20, 2008, at 6:11 PM, Rainer Blome wrote: >>> http://validator.w3.org/check?uri=http%3A%2F%2Fre.search.wikia.com%2Fsearch.html&charset=%28detect+automatically%29&doctype=Inline&group=0 >>> >>> Seriously, are the deviations from the standard intentional? >> > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From rainer.blome at gmx.de Mon Jul 21 22:37:17 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Tue, 22 Jul 2008 00:37:17 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <20080721133727.553b687d@vallheru.vallheru> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> <20080721133727.553b687d@vallheru.vallheru> Message-ID: <48850F9D.6090900@gmx.de> Bartek Jasicki wrote: > 2008-07-21, 01:25:42 > Rainer Blome wrote: >> Bartek Jasicki wrote: >>> 2008-07-20, 10:12:37 >>> Rainer Blome wrote: > there is a problem with encoding in this file - > downloaded by browser (Firefox 3.0.1) and by svn checkout. GNU Emacs > and Gedit don't autodetect file encoding The file's content is correctly encoded. If the right encoding is used to open the file (and the fonts are available), the correct characters are shown. Did you follow the link I posted? Here it is again: >> Please have a look at >> http://search.wikia.com/wiki/UI/Internationalization#Editing_translation_tables >> and see if that helps. Unless you open a file with "M-x find-file-literally", Emacs performs character decoding when it loads a file. The currently deployed version of i18n.js has no explicit encoding spec. So Emacs guesses an encoding or just uses a default value, and ends up with a wrong one. It then uses that wrong encoding to decode the file. At this time, it is already too late, the contents in the buffer are messed up. If you change the encoding now, Emacs will happily "recode" the garbage characters in the buffer, leading to even worse garbage. If a file does not properly declare its encoding, as in this case, you need to tell the editor *right before loading the file* which encoding it should use: Ctrl-x RET c utf-8 RET Ctrl-x Ctrl-f i18n.js RET (Does prolonged use of Emacs give brain damage? Who cares ... ;-) > - even if earlier UTF-8 is set as default. > Reproduction on GNU Emacs: > 1. Download http://re.search.wikia.com/js/i18n/i18n.js by browser > 2. Open GNU Emacs and set (if not is set) UTF-8 encoding How did you set that? The character encoding is buffer-specific. By buffer-specific I mean when you use set-buffer-file-coding-system, it applies to the current buffer, not to the next buffer opened for a different file. (set-default 'buffer-file-coding-system 'utf-8) unfortunately does not apply when I open i18n.js . See above for a way to get it to work. > 3. Open downloaded file. Character encoding automatically change to > 'nil' (unknown? ASCII?) At this point, it's too late. See above. > This is problem - you can test for example pl.po file > http://svn.swlabs.org/grubng/trunk/csharp/Grubngtranslations/pl.po > All editors detect correct character encoding in this file. That's probably because the file explicitly specifies its encoding: "Content-Type: text/plain; charset=UTF-8\n" I assume that the editors pick up this info. >>> texts inserted by users (walls, url's edits) there most of >>> non-ASCII characters are weird >> Somewhere on the way from user input in >> the browser to kt server and back to display in the browser, the >> encoding is messed up. >> >> o How does character input in the browser work? >> >> o How does the browser produce a string from the characters input by >> the user? >> >> o Does a browser expect a particular encoding being input? If so, >> which one > I think, this is wrong way to resolve problem - if on other pages all > is ok (i mean other pages on net, not on Wikia Search, but for example > Wiki works good) then problem is in files not in browser. If only it were that simple :-). The problem as I see it now does not affect whole files, it affects strings generated by scripts. The scripts in turn get their data from a kt server. For edited result texts (which we are talking about), the kt server gets its data from other scripts running in the browser of the user that has edited the result text. > Browser don't > expect any particular encoding on input (often use system default) Upon input, the browser doesn't expect an encoding, it imposes one. The user types keys. The OS and browser convert the keystrokes to characters and finally to a string value, which is processed by a script. This input string value has an encoding. It may be that the input string is encoded using the same encoding that the page specified, but if it doesn't, that could be the source of our problem. > but if page does not include information about character encoding, browser > must make a blind assumption for show that page. search.html does specify its encoding, UTF-8. If it did not, non-ASCII characters in the header would not be correctly displayed. The way I see it, we are talking about bits of pages, in particular search result texts, not about whole pages. > Yes, i know, this bugtrack is little annoying ;) I thought that you (or > someone other) have more options to browse reports. I can only browse > bug by bug (if of course i know number) - example: > http://search.wikia.com/wiki/Special:ProblemReports/11163 You're kidding me? Seriously, even the alpha bug report pages were better. >> What do you mean by "strange"? Please be more specific. > > Strange = i cannot detect what exactly encoding is used - i try UTF-8, > ISO, Windows and Mac encoding - all are incorrect. On search.html, that's because different parts (strings) of the same page have been encoded with different encodings. So whatever page-wide encoding you use, it is wrong for some part of the page. >>> There is problem with >>> inserted by users texts - maybe is something wrong with record to >>> or read from database? Problem was even with old interface (this >>> from 7.1.2008) We need to specify the encodings used by the kt interface: What encoding does it expect for requests (contributions)? What encoding does it use for responses (result texts)? >> Currently both the user interface texts and the search results texts >> are affected by the encoding selected by the browser. Mabe it is >> possible to use different encodings for different parts of a page? >> >> Otherwise we need to agree on a common encoding used by all parts of >> the pages. >> > If you want convert to any character encoding, all files must have this > same encoding, without exception. Maybe not all Wikia Search files are > in UTF-8? Currently, they are all UTF-8, even though they do not all specify this. Except for i18n.js, they are all effectively ASCII. This makes it easy to convert them to UTF-8: just declare them UTF-8, and you're done. > For character conversion tools: > > http://en.wikipedia.org/wiki/Character_encoding#Character_conversion_tools Thanks for the pointer. Rainer From rainer.blome at gmx.de Mon Jul 21 23:09:49 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Tue, 22 Jul 2008 01:09:49 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> Message-ID: <4885173D.30400@gmx.de> Tim Bartel wrote: > * http://re.search.wikia.com/legal/privacy.html and > * http://re.search.wikia.com/legal/legal.html > have broken umlauts. > The encoding specs have been added to these files in the Subversion repository on June 26th (revision 273). That was before i18n.js was added. Apparently, these changes have been missed during deployment. copyright.html however was indeed lacking the encoding spec line, I just added it (my first checkin :-). Rainer From thindil2 at gmail.com Tue Jul 22 11:42:44 2008 From: thindil2 at gmail.com (Bartek Jasicki) Date: Tue, 22 Jul 2008 13:42:44 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <48850F9D.6090900@gmx.de> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> <20080721133727.553b687d@vallheru.vallheru> <48850F9D.6090900@gmx.de> Message-ID: <20080722134244.3331cd13@vallheru.vallheru> 2008-07-22, 00:37:17 Rainer Blome wrote: > The file's content is correctly encoded. > If the right encoding is used to open the file (and the fonts are > available), > the correct characters are shown. > Yes, but i simply curious why this editors can open and autodetect most of files without problem and have problem with this one (yes, for me this is problem ;) ) > Did you follow the link I posted? Here it is again: > Yes, i read it, try this way and file looks correct, but this is workaround problem, not solution. > Unless you open a file with "M-x find-file-literally", > Emacs performs character decoding when it loads a file. > The currently deployed version of i18n.js has no explicit encoding > spec. So Emacs guesses an encoding or just uses a default value, and > ends up with a wrong one. > It then uses that wrong encoding to decode the file. > At this time, it is already too late, the contents in the buffer are > messed up. > If you change the encoding now, Emacs will happily "recode" the > garbage characters in the buffer, leading to even worse garbage. > > If a file does not properly declare its encoding, as in this case, > you need to tell the editor *right before loading the file* > which encoding it should use: > > Ctrl-x RET c utf-8 RET > Ctrl-x Ctrl-f i18n.js RET > > (Does prolonged use of Emacs give brain damage? Who cares ... ;-) > Unless you don't need exit without saving by command: Ctrl-x kiss RET my Ctrl-a ;) You can change encoding in loaded file by Ctrl-x RET r utf-8 RET I think this is simpler (or by menu in graphical mode: Options -> Mule -> Set Coding System -> For Reverting This File Now). > > > This is problem - you can test for example pl.po file > > http://svn.swlabs.org/grubng/trunk/csharp/Grubngtranslations/pl.po > > All editors detect correct character encoding in this file. > > That's probably because the file explicitly specifies its encoding: > > "Content-Type: text/plain; charset=UTF-8\n" > > I assume that the editors pick up this info. > Then how it guess encoding in plain text files? ;) Both editors based on this same encoding function from GTK (if i good saw). Thus maybe problem is here? > > If only it were that simple :-). > The problem as I see it now does not affect whole files, > it affects strings generated by scripts. > The scripts in turn get their data from a kt server. > For edited result texts (which we are talking about), > the kt server gets its data from other scripts running in the browser > of the user that has edited the result text. > Yes, i wrote it earlier - problem is with strings from database not from files. German interface looks good, but sometimes non-ASCII characters in search results (thus in strings from database) are incorrect. In search results it looks like incorrect encoding (weird symbols). In users walls it looks like ASCII code of characters (%u0144 = U+0144 LATIN SMALL LETTER N WITH ACUTE) > > Browser don't > > expect any particular encoding on input (often use system default) > > Upon input, the browser doesn't expect an encoding, it imposes one. > > The user types keys. The OS and browser convert the keystrokes to > characters and finally to a string value, which is processed by a > script. This input string value has an encoding. It may be that the > input string is encoded using the same encoding that the page > specified, but if it doesn't, that could be the source of our problem. > Maybe add security check - every text inserted by user, scripts been convert automatically on utf-8 before adding it to database? > > but if page does not include information about character encoding, > browser > > must make a blind assumption for show that page. > > search.html does specify its encoding, UTF-8. If it did not, > non-ASCII characters in the header would not be correctly displayed. > The way I see it, we are talking about bits of pages, in particular > search result texts, not about whole pages. > I notice that re.search.wikia.com not send informations about character encoding: curl -I http://re.search.wikia.com/js/i18n/i18n.js HTTP/1.1 200 OK Date: Tue, 22 Jul 2008 11:29:14 GMT Server: PWS/1.3.17 X-Px: ms fra-ti2-n40 (fra-ti2-n21), ht fra-ti2-n21 ETag: "3b091-5e06-450931bd1ee00" Content-Length: 24070 Content-Type: application/x-javascript Last-Modified: Thu, 26 Jun 2008 15:04:24 GMT Cache-Control: max-age=288558 Expires: Fri, 25 Jul 2008 19:38:32 GMT Connection: keep-alive curl -I http://re.search.wikia.com HTTP/1.1 200 OK Date: Tue, 22 Jul 2008 11:41:55 GMT Server: PWS/1.3.17 X-Px: rf fra-ti2-n14 (fra-ti2-n31) ETag: "3b095-acb-450f961d0df40" Content-Length: 2763 Content-Type: text/html Last-Modified: Tue, 01 Jul 2008 17:05:25 GMT Cache-Control: max-age=600 Expires: Tue, 22 Jul 2008 11:51:55 GMT Connection: keep-alive Grub page: curl -I http://grub.org HTTP/1.1 200 OK Date: Tue, 22 Jul 2008 11:28:58 GMT Server: Apache/2.2.8 (Debian) PHP/5.2.5-3 with Suhosin-Patch X-Powered-By: PHP/5.2.5-3 Set-Cookie: SESS5e2dff1a9b8e0c758d51cd824dfee2df=afe32fed09a643098d52d541b65222e6; expires=Thu, 14 Aug 2008 15:02:18 GMT; path=/; domain=.grub.org Expires: Sun, 19 Nov 1978 05:00:00 GMT Last-Modified: Tue, 22 Jul 2008 11:28:58 GMT Cache-Control: store, no-cache, must-revalidate Cache-Control: post-check=0, pre-check=0 Content-Type: text/html; charset=utf-8 And Google Poland: curl -I http://www.google.pl HTTP/1.1 200 OK Cache-Control: private, max-age=0 Date: Tue, 22 Jul 2008 11:33:45 GMT Expires: -1 Content-Type: text/html; charset=ISO-8859-2 Set-Cookie: PREF=ID=dac424d3407e820f:TM=1216726425:LM=1216726425:S=CihERyqbbwRoZte9; expires=Thu, 22-Jul-2010 11:33:45 GMT; path=/; domain=.google.pl Server: gws Content-Length: 0 > > Yes, i know, this bugtrack is little annoying ;) I thought that > > you (or someone other) have more options to browse reports. I can > > only browse bug by bug (if of course i know number) - example: > > http://search.wikia.com/wiki/Special:ProblemReports/11163 > > You're kidding me? > Seriously, even the alpha bug report pages were better. > I try be polite ;) > >> What do you mean by "strange"? Please be more specific. > > > > Strange = i cannot detect what exactly encoding is used - i try > > UTF-8, ISO, Windows and Mac encoding - all are incorrect. > > On search.html, that's because different parts (strings) of the same > page have been encoded with different encodings. So whatever > page-wide encoding you use, it is wrong for some part of the page. > > >>> There is problem with > >>> inserted by users texts - maybe is something wrong with record to > >>> or read from database? Problem was even with old interface (this > >>> from 7.1.2008) > > We need to specify the encodings used by the kt interface: > > What encoding does it expect for requests (contributions)? > > What encoding does it use for responses (result texts)? > See above - proposition about security encoding check and informations send in headers by server. Bartek From jwales at wikia.com Mon Jul 21 16:08:38 2008 From: jwales at wikia.com (Jimmy Wales) Date: Mon, 21 Jul 2008 12:08:38 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: Message-ID: <4884B486.5040405@wikia.com> I am 100% focussed on widgets as the "next big thing" for Wikia search. We have a lot of bugfixes and refinements going on... and the toolbar is starting to look pretty amazing too. But I share Markie's view that the widget framework is going to be a killer app. The basic concept, I think I need help with some technical details, most of which will have to come from internal people like Jer. But I also have some general thoughts and questions which I think a lot of people will be able to help with. Basically, what I am wanting to see is a very very simple api that will let boring crappy programmers like me come up with some neat ideas and actually code them up quickly. The idea is: you match on certain regular expressions (though for performance reasons we probably can not support full regex) and return an "object" which might be a specialized search box, a grid of data, whatever. For example: '92109 weather' 'weather 92109' 'san diego weather' 'weather san diego' should return... something, the best and most free thing we can return. And 'SFO to LAX' should return a specialized search object, like what google does... The idea is this: 1. Ordinary programmers can upload triggers (either lists of keywords or... ??? how broad and computational can we make this?) 2. And those triggers call subroutines which return "objects" 3. And the community/staff decide which of these are returned "by default" for all users... but anyone can also go into advanced preferences and add the widgets experimentally that we collectively think are "not ready for prime time". --Jimbo From jwales at wikia.com Mon Jul 21 16:10:35 2008 From: jwales at wikia.com (Jimmy Wales) Date: Mon, 21 Jul 2008 12:10:35 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <355a36af0807182131l34ec2b62id534f6f56935d987@mail.gmail.com> References: <355a36af0807182131l34ec2b62id534f6f56935d987@mail.gmail.com> Message-ID: <4884B4FB.4030508@wikia.com> Aerik Sylvan wrote: > Hmm... couldn't you use a Google Maps API call (JS) to get the lat lang, > then call the NOAA feed? I haven't looked at the NOAA feed at all, > but... Then we cache the lat/lang in a cookie or something, to avoid a > gazillion calls. Might be a little tricky with pure javascript, if the > NOAA feed is only xml and not JSON, but... I am 100% all for this, and I am also a free software and free data fanatic, and would love to see a way to make all of this available under a free license. So calling external APIs to proprietary servcies to make life better is great -- we are doing this now, calling Yahoo BOSS when we have no results -- but it is only great as long as we keep firmly focussed on: how can we also replace this with something really open and free. From jwales at wikia.com Mon Jul 21 16:11:16 2008 From: jwales at wikia.com (Jimmy Wales) Date: Mon, 21 Jul 2008 12:11:16 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <3ae3aa420807181405g353886bek21f94904a089b1cc@mail.gmail.com> References: <3ae3aa420807181405g353886bek21f94904a089b1cc@mail.gmail.com> Message-ID: <4884B524.80208@wikia.com> Linas Vepstas wrote: > 2008/7/18 Mark (Markie) : > >> interested me is the widgets ideas. IMO these are really key things that, > > Maybe this has been talked to death already, but I started using > yahoo search recently, and I really like the keyword-refinement > widget they have -- e.g. if you search for perl, it offers up a bunch > of perl topics to pick from. I'm guessing that picking from this > list improves both accuracy and speed. Can you send us some links to sample searches to look at this? How hard would it be to replicate this functionality in free software? From newsmarkie at googlemail.com Tue Jul 22 13:45:57 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Tue, 22 Jul 2008 14:45:57 +0100 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <4884B486.5040405@wikia.com> References: <4884B486.5040405@wikia.com> Message-ID: excellent, nice to see that we are totally along the same lines here :-p youve pretty much summed up what im thinking, but failed to put down in text, so the next stage i would suppose would to actually start working on it. im planning to start some mockups per jers comments, on getting the layouts and GUI sorted for all widgets before we start actually doing them, and i hope to do these soon :-) regards (a happy) mark On Mon, Jul 21, 2008 at 5:08 PM, Jimmy Wales wrote: > > I am 100% focussed on widgets as the "next big thing" for Wikia search. > We have a lot of bugfixes and refinements going on... and the toolbar > is starting to look pretty amazing too. But I share Markie's view that > the widget framework is going to be a killer app. > > The basic concept, I think I need help with some technical details, most > of which will have to come from internal people like Jer. But I also > have some general thoughts and questions which I think a lot of people > will be able to help with. > > Basically, what I am wanting to see is a very very simple api that will > let boring crappy programmers like me come up with some neat ideas and > actually code them up quickly. > > The idea is: you match on certain regular expressions (though for > performance reasons we probably can not support full regex) and return > an "object" which might be a specialized search box, a grid of data, > whatever. > > For example: > > '92109 weather' > 'weather 92109' > 'san diego weather' > 'weather san diego' > > should return... something, the best and most free thing we can return. > > And 'SFO to LAX' should return a specialized search object, like what > google does... > > The idea is this: > > 1. Ordinary programmers can upload triggers (either lists of keywords > or... ??? how broad and computational can we make this?) > > 2. And those triggers call subroutines which return "objects" > > 3. And the community/staff decide which of these are returned "by > default" for all users... but anyone can also go into advanced > preferences and add the widgets experimentally that we collectively > think are "not ready for prime time". > > --Jimbo > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080722/419170ba/attachment.html From newsmarkie at googlemail.com Tue Jul 22 16:00:40 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Tue, 22 Jul 2008 17:00:40 +0100 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <4884B486.5040405@wikia.com> References: <4884B486.5040405@wikia.com> Message-ID: a very quick mockup of how widgets could look @ http://people.swlabs.org/~markie/widgetmockup.png this is purely my input but i welcome more input on this regards mark On Mon, Jul 21, 2008 at 5:08 PM, Jimmy Wales wrote: > > I am 100% focussed on widgets as the "next big thing" for Wikia search. > We have a lot of bugfixes and refinements going on... and the toolbar > is starting to look pretty amazing too. But I share Markie's view that > the widget framework is going to be a killer app. > > The basic concept, I think I need help with some technical details, most > of which will have to come from internal people like Jer. But I also > have some general thoughts and questions which I think a lot of people > will be able to help with. > > Basically, what I am wanting to see is a very very simple api that will > let boring crappy programmers like me come up with some neat ideas and > actually code them up quickly. > > The idea is: you match on certain regular expressions (though for > performance reasons we probably can not support full regex) and return > an "object" which might be a specialized search box, a grid of data, > whatever. > > For example: > > '92109 weather' > 'weather 92109' > 'san diego weather' > 'weather san diego' > > should return... something, the best and most free thing we can return. > > And 'SFO to LAX' should return a specialized search object, like what > google does... > > The idea is this: > > 1. Ordinary programmers can upload triggers (either lists of keywords > or... ??? how broad and computational can we make this?) > > 2. And those triggers call subroutines which return "objects" > > 3. And the community/staff decide which of these are returned "by > default" for all users... but anyone can also go into advanced > preferences and add the widgets experimentally that we collectively > think are "not ready for prime time". > > --Jimbo > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080722/82c9d598/attachment.html From jwales at wikia.com Tue Jul 22 18:12:09 2008 From: jwales at wikia.com (Jimmy Wales) Date: Tue, 22 Jul 2008 14:12:09 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: <4884B486.5040405@wikia.com> Message-ID: <488622F9.3050307@wikia.com> Mark (Markie) wrote: > a very quick mockup of how widgets could look @ > http://people.swlabs.org/~markie/widgetmockup.png > > this is purely my input but i welcome more input on this http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search Google puts them right at the top of the page. My thinking is that what we do is allow the uploading of "hooks" (either regex's, restricted regex's, or in the simplest case just lists of keywords) which trigger a function to print the object. This works best for the kinds of searches where the best answer is not a list of links, but a smarter object of some kind... either weather pictures or a searchbox for a different kind of site. I think that a community can easily best google at this aspect of things... after all, some fairly obvious "hooks" are overlooked by google: http://www.google.com/search?hl=en&safe=off&q=SFO+weather&btnG=Search Google knows patterns for weather but not for weather A community can easily notice that and fix it... if given the right tools. --Jimbo From newsmarkie at googlemail.com Tue Jul 22 18:28:26 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Tue, 22 Jul 2008 19:28:26 +0100 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <488622F9.3050307@wikia.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> Message-ID: On Tue, Jul 22, 2008 at 7:12 PM, Jimmy Wales wrote: > Mark (Markie) wrote: > > a very quick mockup of how widgets could look @ > > http://people.swlabs.org/~markie/widgetmockup.png > > > > this is purely my input but i welcome more input on this > > http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search > http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search > > Google puts them right at the top of the page. my thinking on using the space at the side was that by putting (possibly multiple) widgets at the top of the page shifts down the results, but this can be good and bad as the widget hopefully provides them with the best answer. however to me it seemed to be a big waste of space on the right so i thought it could be filled, but i dont feel that strongly as to its position. > > > My thinking is that what we do is allow the uploading of "hooks" (either > regex's, restricted regex's, or in the simplest case just lists of > keywords) which trigger a function to print the object. > > This works best for the kinds of searches where the best answer is not a > list of links, but a smarter object of some kind... either weather > pictures or a searchbox for a different kind of site. > > I think that a community can easily best google at this aspect of > things... after all, some fairly obvious "hooks" are overlooked by google: > http://www.google.com/search?hl=en&safe=off&q=SFO+weather&btnG=Search > > Google knows patterns for > weather > but not for > weather > > A community can easily notice that and fix it... if given the right tools. yup, this sounds like a great idea :-) maybe there could be a functions/hooks table on a wiki like page, so that we could change/edit the hooks and patterns to be recognised with their feedbacks, but that gets difficult so maybe not :-p regards mark > > > --Jimbo > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080722/73bfbc2d/attachment.html From jwales at wikia.com Tue Jul 22 19:07:20 2008 From: jwales at wikia.com (Jimmy Wales) Date: Tue, 22 Jul 2008 15:07:20 -0400 Subject: [search-ui] Widgets In-Reply-To: <21A55959-1867-4F94-A510-33F9534E8B12@jabber.org> References: <21A55959-1867-4F94-A510-33F9534E8B12@jabber.org> Message-ID: <48862FE8.5080903@wikia.com> I think that: http://www.google.com/search?hl=en&q=SFO+weather&btnG=Google+Search is a significantly inferior search result as compared to: http://www.google.com/search?hl=en&safe=off&q=94101+weather&btnG=Search I am not sure what "editable" would mean in this context, exactly. :) ------ In the "vaporware" documentation I wrote (and if someone has a copy of it... please send to me, I lost it in my hard drive crash), I tried to distinguish between some things. For example, this: http://www.google.com/search?hl=en&safe=off&q=SFO+to+LAX&btnG=Search This result can be generated without ANY external call to data providers. It's just a simple formulaic fancy-box result (which is nonetheless quite fabulous). All you need to do is more or less a "one time" thing - generate a matrix of " TO " as triggers, and then plug in those codes in the relevant place in the search result template, along with some dates generated based on today's date, and there you go. That one would be perfectly good more or less permanently with only minor changes now and then as airports are created or go out of business. That's different from the weather one, where the answer changes in realtime depending on the weather. --Jimbo Jeremie Miller wrote: > My response is pretty much a UI thing so I thought I'd switch it to > this list :) > > For me, the in-my-head debate has been about how with our social/ > editable results page, to best serve up widget-able data, and I think > I've convinced myself that any widget results should be shown exactly > like a normal search result (with a title, sumamry, link, and optional > sub-annotations). It also has to be subject to all the normal editing/ > ranking/deletion/etc for any keyword it shows up under. > > The alternative is that they are a special case kind of like how other > search engines show them when you search for weather+zip (in it's own > UI above the results)... while I think this may be "prettier" I really > think it's important to make these results be part of the same social > framework as everything else. Maybe there will be multiple weather > widgets even, for instance? > > I just wanted to put these ideas out here before we started > implementing anything for widgets, as it's a pretty strategic thing to > make them all return a search result instead of some special case UI > for each. > > Jer > > On Jul 18, 2008, at 2:35 PM, Mark (Markie) wrote: > >> I know i havent been to active round here recently (been really >> busy, sorry :-( ) but one thing that i have noticed on the mailing >> lists that really interested me is the widgets ideas. IMO these are >> really key things that, shouldnt :-p, be too hard to implement. Ive >> had a brief look at one of the widgets which i look for alot, which >> is weather (although i dont know why, it always seems to be rain :-p) >> >> NWS / NOAA provide an XML output of weather for all US cities and >> most international cities, and with it being a US gov org all its >> content is therefore PD and totally free to use and copy etc (not a >> lawyer though). this we could then implement easily using a JS call >> from the search var, with or without looking for a "weather: XXX" >> keyword and then output next to the search results, maybe below the >> adverts. this could also be JS implemented to be hide able etc with >> more widgets....... the problem with NOAA IMO is that it requires >> calls to use lat/long co-ordinates for the calls, thus required more >> geo-location stuff which makes this harder and harder. >> >> so then i looked for alternative APIs which could be used and came >> up with weather.com's feed. this requires registration, but is then >> free to use but requires 5 links back and a logo, which may or >> maynot be acceptable to us?? this however does give the ease of >> being able to call by city name, removing the need for geo-locations >> stuff. >> >> however the balance needs to come between coding and technical >> possibility and the "freeness" of the content. >> >> also im guessing there are many different ideas for widgets so also >> please feel free to give thoughts on this one, and any others that >> spring to mind as being wanted. my thoughts were: Open Street Map >> contents when searching locations (ie a map of the location in the >> world/country/are when searching for a city) or cinema times for >> jimbo etc etc >> >> regards (and apologies for the lengthy message) >> >> mark > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > > From jwales at wikia.com Tue Jul 22 19:32:43 2008 From: jwales at wikia.com (Jimmy Wales) Date: Tue, 22 Jul 2008 15:32:43 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> Message-ID: <488635DB.5000807@wikia.com> Mark (Markie) wrote: > yup, this sounds like a great idea :-) maybe there could be a > functions/hooks table on a wiki like page, so that we could change/edit > the hooks and patterns to be recognised with their feedbacks, but that > gets difficult so maybe not :-p Absolutely! The more flexible the system is, the better it will be. The more that non-programming editors can affect/fix it, the better. --Jimbo From david.pean at gmail.com Tue Jul 22 20:57:24 2008 From: david.pean at gmail.com (David Pean) Date: Tue, 22 Jul 2008 16:57:24 -0400 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <20080722134244.3331cd13@vallheru.vallheru> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> <20080721133727.553b687d@vallheru.vallheru> <48850F9D.6090900@gmx.de> <20080722134244.3331cd13@vallheru.vallheru> Message-ID: I took a look at the board example and was able to fix it -- We were missing the encodeURIComponent() in the javascript in sending the message text through the form. The url description and titles could have the same related problem -- what is getting sent to KT is wrapped in the encode function, but the conversion to JSON could be throwing it off? I tried to do a simple test to see if the umlats would show up. http://re.search.wikia.com/search.html#test -- this looks like it comes in correctly (I edited the description of the first result) -- I have seen some weird characters in recent changes, so maybe only certain languages are failing to convert? On Tue, Jul 22, 2008 at 7:42 AM, Bartek Jasicki wrote: > 2008-07-22, 00:37:17 > Rainer Blome wrote: > >> The file's content is correctly encoded. >> If the right encoding is used to open the file (and the fonts are >> available), >> the correct characters are shown. >> > > Yes, but i simply curious why this editors can open and autodetect most > of files without problem and have problem with this one (yes, for me > this is problem ;) ) > >> Did you follow the link I posted? Here it is again: >> > > Yes, i read it, try this way and file looks correct, but this is > workaround problem, not solution. > >> Unless you open a file with "M-x find-file-literally", >> Emacs performs character decoding when it loads a file. >> The currently deployed version of i18n.js has no explicit encoding >> spec. So Emacs guesses an encoding or just uses a default value, and >> ends up with a wrong one. >> It then uses that wrong encoding to decode the file. >> At this time, it is already too late, the contents in the buffer are >> messed up. >> If you change the encoding now, Emacs will happily "recode" the >> garbage characters in the buffer, leading to even worse garbage. >> >> If a file does not properly declare its encoding, as in this case, >> you need to tell the editor *right before loading the file* >> which encoding it should use: >> >> Ctrl-x RET c utf-8 RET >> Ctrl-x Ctrl-f i18n.js RET >> >> (Does prolonged use of Emacs give brain damage? Who cares ... ;-) >> > > Unless you don't need exit without saving by command: > Ctrl-x kiss RET my Ctrl-a ;) > > You can change encoding in loaded file by > > Ctrl-x RET r utf-8 RET > > I think this is simpler (or by menu in graphical mode: Options -> Mule > -> Set Coding System -> For Reverting This File Now). > >> >> > This is problem - you can test for example pl.po file >> > http://svn.swlabs.org/grubng/trunk/csharp/Grubngtranslations/pl.po >> > All editors detect correct character encoding in this file. >> >> That's probably because the file explicitly specifies its encoding: >> >> "Content-Type: text/plain; charset=UTF-8\n" >> >> I assume that the editors pick up this info. >> > > Then how it guess encoding in plain text files? ;) > Both editors based on this same encoding function from GTK (if i good > saw). Thus maybe problem is here? > >> >> If only it were that simple :-). >> The problem as I see it now does not affect whole files, >> it affects strings generated by scripts. >> The scripts in turn get their data from a kt server. >> For edited result texts (which we are talking about), >> the kt server gets its data from other scripts running in the browser >> of the user that has edited the result text. >> > > Yes, i wrote it earlier - problem is with strings from database not > from files. German interface looks good, but sometimes non-ASCII > characters in search results (thus in strings from database) are > incorrect. In search results it looks like incorrect encoding (weird > symbols). In users walls it looks like ASCII code of characters > (%u0144 = U+0144 LATIN SMALL LETTER N WITH ACUTE) > >> > Browser don't >> > expect any particular encoding on input (often use system default) >> >> Upon input, the browser doesn't expect an encoding, it imposes one. >> >> The user types keys. The OS and browser convert the keystrokes to >> characters and finally to a string value, which is processed by a >> script. This input string value has an encoding. It may be that the >> input string is encoded using the same encoding that the page >> specified, but if it doesn't, that could be the source of our problem. >> > > Maybe add security check - every text inserted by user, scripts been > convert automatically on utf-8 before adding it to database? > >> > but if page does not include information about character encoding, >> browser >> > must make a blind assumption for show that page. >> >> search.html does specify its encoding, UTF-8. If it did not, >> non-ASCII characters in the header would not be correctly displayed. >> The way I see it, we are talking about bits of pages, in particular >> search result texts, not about whole pages. >> > > I notice that re.search.wikia.com not send informations about character > encoding: > > curl -I http://re.search.wikia.com/js/i18n/i18n.js > HTTP/1.1 200 OK > Date: Tue, 22 Jul 2008 11:29:14 GMT > Server: PWS/1.3.17 > X-Px: ms fra-ti2-n40 (fra-ti2-n21), ht fra-ti2-n21 > ETag: "3b091-5e06-450931bd1ee00" > Content-Length: 24070 > Content-Type: application/x-javascript > Last-Modified: Thu, 26 Jun 2008 15:04:24 GMT > Cache-Control: max-age=288558 > Expires: Fri, 25 Jul 2008 19:38:32 GMT > Connection: keep-alive > > curl -I http://re.search.wikia.com > HTTP/1.1 200 OK > Date: Tue, 22 Jul 2008 11:41:55 GMT > Server: PWS/1.3.17 > X-Px: rf fra-ti2-n14 (fra-ti2-n31) > ETag: "3b095-acb-450f961d0df40" > Content-Length: 2763 > Content-Type: text/html > Last-Modified: Tue, 01 Jul 2008 17:05:25 GMT > Cache-Control: max-age=600 > Expires: Tue, 22 Jul 2008 11:51:55 GMT > Connection: keep-alive > > Grub page: > > curl -I http://grub.org > HTTP/1.1 200 OK > Date: Tue, 22 Jul 2008 11:28:58 GMT > Server: Apache/2.2.8 (Debian) PHP/5.2.5-3 with Suhosin-Patch > X-Powered-By: PHP/5.2.5-3 > Set-Cookie: SESS5e2dff1a9b8e0c758d51cd824dfee2df=afe32fed09a643098d52d541b65222e6; expires=Thu, 14 Aug 2008 15:02:18 GMT; path=/; domain=.grub.org > Expires: Sun, 19 Nov 1978 05:00:00 GMT > Last-Modified: Tue, 22 Jul 2008 11:28:58 GMT > Cache-Control: store, no-cache, must-revalidate > Cache-Control: post-check=0, pre-check=0 > Content-Type: text/html; charset=utf-8 > > And Google Poland: > > curl -I http://www.google.pl > HTTP/1.1 200 OK > Cache-Control: private, max-age=0 > Date: Tue, 22 Jul 2008 11:33:45 GMT > Expires: -1 > Content-Type: text/html; charset=ISO-8859-2 > Set-Cookie: PREF=ID=dac424d3407e820f:TM=1216726425:LM=1216726425:S=CihERyqbbwRoZte9; expires=Thu, 22-Jul-2010 11:33:45 GMT; path=/; domain=.google.pl > Server: gws > Content-Length: 0 > >> > Yes, i know, this bugtrack is little annoying ;) I thought that >> > you (or someone other) have more options to browse reports. I can >> > only browse bug by bug (if of course i know number) - example: >> > http://search.wikia.com/wiki/Special:ProblemReports/11163 >> >> You're kidding me? >> Seriously, even the alpha bug report pages were better. >> > > I try be polite ;) > >> >> What do you mean by "strange"? Please be more specific. >> > >> > Strange = i cannot detect what exactly encoding is used - i try >> > UTF-8, ISO, Windows and Mac encoding - all are incorrect. >> >> On search.html, that's because different parts (strings) of the same >> page have been encoded with different encodings. So whatever >> page-wide encoding you use, it is wrong for some part of the page. >> >> >>> There is problem with >> >>> inserted by users texts - maybe is something wrong with record to >> >>> or read from database? Problem was even with old interface (this >> >>> from 7.1.2008) >> >> We need to specify the encodings used by the kt interface: >> >> What encoding does it expect for requests (contributions)? >> >> What encoding does it use for responses (result texts)? >> > > See above - proposition about security encoding check and informations > send in headers by server. > > Bartek > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > From rainer.blome at gmx.de Wed Jul 23 09:45:37 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Wed, 23 Jul 2008 11:45:37 +0200 Subject: [search-ui] Widgets In-Reply-To: <21A55959-1867-4F94-A510-33F9534E8B12@jabber.org> References: <21A55959-1867-4F94-A510-33F9534E8B12@jabber.org> Message-ID: <4886FDC1.1040209@gmx.de> For the record, I think that we should use some more accurate and less overloaded term than "widget", for example "infolet", "searchlet", "querylet" or "resultlet" (my favorite). Even "portlet" would be more accurate. Jeremie Miller wrote: > any widget results should be shown exactly > like a normal search result (with a title, sumamry, link, and optional > sub-annotations). If the summary and annotations already provide all the info that a "resultlet"/"widget" has to offer, it may not make sense to offer a link. In general, it may be a good idea to separate the presentation (formatting) of results from the ranking. > It also has to be subject to all the normal editing/ > ranking/deletion/etc for any keyword it shows up under. Yes, leave the placement decision to the users. > The alternative is that they are a special case kind of like how other > search engines show them when you search for weather+zip (in it's own > UI above the results)... while I think this may be "prettier" > it's important to make these results be part of the same social > framework as everything else. Maybe there will be multiple weather > widgets even, for instance? Pretty presentation and and social ranking do not have to be exclusive. The social ranking is the more important of the two, I'd say. Custom presentation might take more effort to allow for. The "results pop up menu" might need to be custom, too, as some "resultlets" might offer more or less options. > I just wanted to put these ideas out here before we started > implementing anything for widgets, as it's a pretty strategic thing to > make them all return a search result instead of some special case UI > for each. Maybe it is already possible to achieve custom presentation with just HTML in the description (summary). Another, safer way might be to add a custom HTML field after the summary, for *all* results. The custom field would be empty for ordinary (index) results, and have generated content for the "widgets" alias "resultlets". Rainer From rainer.blome at gmx.de Wed Jul 23 10:21:13 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Wed, 23 Jul 2008 12:21:13 +0200 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> Message-ID: <48870619.7050608@gmx.de> Mark (Markie) wrote: > On Tue, Jul 22, 2008 at 7:12 PM, Jimmy Wales > wrote: > > Mark (Markie) wrote: > > a very quick mockup of how widgets could look @ > > http://people.swlabs.org/~markie/widgetmockup.png > > > > > this is purely my input but i welcome more input on this > > http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search > > http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search > > > Google puts them right at the top of the page. > > > my thinking on using the space at the side was that by putting > (possibly multiple) widgets at the top of the page shifts down the > results, but this can be good and bad as the widget hopefully provides > them with the best answer. however to me it seemed to be a big waste > of space on the right so i thought it could be filled, but i dont feel > that strongly as to its position. Free space is often a good thing. It makes a page easier to use. Once there are more "resultlets" than would fit on one page, even in the "minimized" form shown in your mock-up, you'll long for the days when the results page looked relatively clean :). And note that you "cheated": the space you used for the "widgets" is often filled with about three ads :-). Jer's idea of ranking "widgets" like ordinary results looks to me like the right answer to the question of which and how much space to give to "resultlets". On the other hand, when there are so many star-rated results (which currently always appear before non-rated results, correct?) that new results and "resultlets" do not show up at all on the first page, they will be hard to see. There may even be "resultlets" which generate more than one result, for example postal codes for multiple countries, making the competition even harder. In order to give new "resultlets" a chance (and maybe new ordinary results as well?), they might be "advertised" in a dedicated area (portlet) on the right. For users who don't want to see this, that area should be collapsible. Rainer From rainer.blome at gmx.de Wed Jul 23 10:37:28 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Wed, 23 Jul 2008 12:37:28 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> <20080721133727.553b687d@vallheru.vallheru> <48850F9D.6090900@gmx.de> <20080722134244.3331cd13@vallheru.vallheru> Message-ID: <488709E8.7000904@gmx.de> David Pean wrote: > I took a look at the board example and was able to fix it -- We were > missing the encodeURIComponent() in the javascript in sending the > message text through the form. > One down, great! > The url description and titles could have the same related problem -- > what is getting sent to KT is wrapped in the encode function, but the > conversion to JSON could be throwing it off? I tried to do a simple > test to see if the umlats would show up. > > http://re.search.wikia.com/search.html#test -- this looks like it > comes in correctly (I edited the description of the first result) -- I > have seen some weird characters in recent changes, so maybe only > certain languages are failing to convert? > A few days ago I did a similar test, with "German (de)" in effect, showing that some umlauts, but not all are displayed correctly. Switch to "German" using the language dialog, *then reload*. You should see a result for "test.de" (*not* www.test.de), with a summary with a bunch of umlauts, accented characters and the native name of "Japanese". The first three characters should have been '???' ("A"O"U), but for me display as six question marks instead. Rainer From newsmarkie at googlemail.com Wed Jul 23 13:41:05 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Wed, 23 Jul 2008 14:41:05 +0100 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <48870619.7050608@gmx.de> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> Message-ID: On Wed, Jul 23, 2008 at 11:21 AM, Rainer Blome wrote: > Mark (Markie) wrote: > > On Tue, Jul 22, 2008 at 7:12 PM, Jimmy Wales > > wrote: > > > > Mark (Markie) wrote: > > > a very quick mockup of how widgets could look @ > > > http://people.swlabs.org/~markie/widgetmockup.png > > > > > > > > this is purely my input but i welcome more input on this > > > > > http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search > > < > http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search> > > http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search > > < > http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search> > > > > Google puts them right at the top of the page. > > > > > > my thinking on using the space at the side was that by putting > > (possibly multiple) widgets at the top of the page shifts down the > > results, but this can be good and bad as the widget hopefully provides > > them with the best answer. however to me it seemed to be a big waste > > of space on the right so i thought it could be filled, but i dont feel > > that strongly as to its position. > Free space is often a good thing. It makes a page easier to use. > Once there are more "resultlets" than would fit on one page, > even in the "minimized" form shown in your mock-up, > you'll long for the days when the results page looked relatively clean :). > And note that you "cheated": the space you used for the "widgets" is > often filled with about three ads :-). darn caught :-p > > > Jer's idea of ranking "widgets" like ordinary results looks to me like > the right answer to the question of which and how much space to give to > "resultlets". > > On the other hand, when there are so many star-rated results > (which currently always appear before non-rated results, correct?) > that new results and "resultlets" do not show up at all on the first page, > they will be hard to see. > > There may even be "resultlets" which generate more than one result, for > example postal codes for multiple countries, making the competition even > harder. > > In order to give new "resultlets" a chance > (and maybe new ordinary results as well?), > they might be "advertised" in a dedicated area (portlet) on the right. > For users who don't want to see this, that area should be collapsible. hmm seems like a good idea to me: get them noticed then move them into the "normal/main" :-) regards mark > > > Rainer > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080723/79cb5ce0/attachment-0001.html From thindil2 at gmail.com Wed Jul 23 15:35:30 2008 From: thindil2 at gmail.com (Bartek Jasicki) Date: Wed, 23 Jul 2008 17:35:30 +0200 Subject: [search-ui] Bad characters encoding on Wikia Search pages In-Reply-To: <488709E8.7000904@gmx.de> References: <20080718215424.11a4b091@vallheru.vallheru> <4882F375.3000103@gmx.de> <20080720112209.60094c52@vallheru.vallheru> <4883C976.3040002@gmx.de> <20080721133727.553b687d@vallheru.vallheru> <48850F9D.6090900@gmx.de> <20080722134244.3331cd13@vallheru.vallheru> <488709E8.7000904@gmx.de> Message-ID: <20080723173530.46e4a642@vallheru.vallheru> 2008-07-23, 12:37:28 Rainer Blome wrote: > David Pean wrote: > > I took a look at the board example and was able to fix it -- We were > > missing the encodeURIComponent() in the javascript in sending the > > message text through the form. > > > One down, great! > Probably fixed, on re.search.wikia.com is old version of board.js ;) Bartek From adatta02 at gmail.com Wed Jul 23 23:42:27 2008 From: adatta02 at gmail.com (adatta02 at gmail.com) Date: Wed, 23 Jul 2008 19:42:27 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> Message-ID: <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> Since the chat this morning I've been looking around at "widgets" and thinking about what we discussed. I think that building the framework as a service where users submit Perl/PHP/Python/ect that will be run on one of our servers is a dangerous path to go down. The coice seems to be between security vs. application power. The big concern with allowing server side code is security. From the chat, it sounds like the plan is to hand review any code that will be run on the server and reject "dangerous" code. This seems like a daunting if not down right impossible task as any reasonably complicated piece of code is going to be extremely hard to certify as "safe". There are various examples of sneaking nefarious code into a seemingly safe code block famously http://en.wikipedia.org/wiki/Underhanded_C_Contest >From a developer's standpoint - the big gain of being able to run server side code is the ability to interact with the OS layer (files, sockets, databases, ect) and libraries (GD, standard libraries, ect) . However, it seems like there is no way we would be able to provide access to any OS features while simultanously gaurenteeing security. It also seems like providing persistent storage through this framework would require significant infastructure. What I'd propose is to construct a framework that mirrors the Google igoogle system. Following from this, developers would be able to write javascript blocks that run on some set of triggers (either regex matches or simple in_string()) Next, on the search results page, the user's browser executes this javascript snippit in an iframe (to create a sandbox) and injects any new HTML into the actual search results. I think allowing javascript snippits would provide a good mix between ease of development, security, as well as robustness. A javascript widget would be able to do anything that a server side widget could accomplish. It would be able to fetch and process RSS feeds, make cross site XHR requests, and leverage additional "enviroment" variables we choose to make available to it. Using javascript would also (hopefully) eliminate the security woes. Additionally, we could enable persistent storage by providing a JSON interface into HBase (similar to KT maybe?) Anyway, these are just some thoughts. There are obviously problems, for example there would be no easy way to leverage APIs that don't have a RESTFul interface (ie Youtube) without additional infastructure. Ashish On Wed, Jul 23, 2008 at 9:41 AM, Mark (Markie) wrote: > > > On Wed, Jul 23, 2008 at 11:21 AM, Rainer Blome > wrote: > >> Mark (Markie) wrote: >> > On Tue, Jul 22, 2008 at 7:12 PM, Jimmy Wales > > > wrote: >> > >> > Mark (Markie) wrote: >> > > a very quick mockup of how widgets could look @ >> > > http://people.swlabs.org/~markie/widgetmockup.png >> > >> > > >> > > this is purely my input but i welcome more input on this >> > >> > >> http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search >> > < >> http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search> >> > >> http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search >> > < >> http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search> >> > >> > Google puts them right at the top of the page. >> > >> > >> > my thinking on using the space at the side was that by putting >> > (possibly multiple) widgets at the top of the page shifts down the >> > results, but this can be good and bad as the widget hopefully provides >> > them with the best answer. however to me it seemed to be a big waste >> > of space on the right so i thought it could be filled, but i dont feel >> > that strongly as to its position. >> Free space is often a good thing. It makes a page easier to use. >> Once there are more "resultlets" than would fit on one page, >> even in the "minimized" form shown in your mock-up, >> you'll long for the days when the results page looked relatively clean :). >> And note that you "cheated": the space you used for the "widgets" is >> often filled with about three ads :-). > > darn caught :-p > > >> >> >> Jer's idea of ranking "widgets" like ordinary results looks to me like >> the right answer to the question of which and how much space to give to >> "resultlets". >> >> On the other hand, when there are so many star-rated results >> (which currently always appear before non-rated results, correct?) >> that new results and "resultlets" do not show up at all on the first page, >> they will be hard to see. >> >> There may even be "resultlets" which generate more than one result, for >> example postal codes for multiple countries, making the competition even >> harder. >> >> In order to give new "resultlets" a chance >> (and maybe new ordinary results as well?), >> they might be "advertised" in a dedicated area (portlet) on the right. >> For users who don't want to see this, that area should be collapsible. > > hmm seems like a good idea to me: get them noticed then move them into the > "normal/main" :-) > > regards > > mark > > >> >> >> Rainer >> >> _______________________________________________ >> Search-UI mailing list >> Search-UI at wikia.com >> http://lists.wikia.com/mailman/listinfo/search-ui >> > > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080723/ddbafeeb/attachment.html From newsmarkie at googlemail.com Thu Jul 24 09:13:12 2008 From: newsmarkie at googlemail.com (Mark (Markie)) Date: Thu, 24 Jul 2008 10:13:12 +0100 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> Message-ID: hmm i thought JS was the only real choice of lang that was being considered for this, to match the previous code choices and the install/mashup/tweak - abilitiy of the code and would support it over Perl anyday of the week nothing much is gained by using perl, esp with the requirements for additional dev/oversight time for code submits and required infrastructure, and JS does most things and im sure we can manage to hack round problems :-p regards mark On Thu, Jul 24, 2008 at 12:42 AM, wrote: > Since the chat this morning I've been looking around at "widgets" and > thinking about what we discussed. > > I think that building the framework as a service where users submit > Perl/PHP/Python/ect that will be run on one of our servers is a dangerous > path to go down. > > The coice seems to be between security vs. application power. > > The big concern with allowing server side code is security. From the chat, > it sounds like the plan is to hand review any code that will be run on the > server and reject "dangerous" code. This seems like a daunting if not down > right impossible task as any reasonably complicated piece of code is going > to be extremely hard to certify as "safe". There are various examples of > sneaking nefarious code into a seemingly safe code block famously > http://en.wikipedia.org/wiki/Underhanded_C_Contest > > From a developer's standpoint - the big gain of being able to run server > side code is the ability to interact with the OS layer (files, sockets, > databases, ect) and libraries (GD, standard libraries, ect) . However, it > seems like there is no way we would be able to provide access to any OS > features while simultanously gaurenteeing security. It also seems like > providing persistent storage through this framework would require > significant infastructure. > > What I'd propose is to construct a framework that mirrors the Google > igoogle system. Following from this, developers would be able to write > javascript blocks that run on some set of triggers (either regex matches or > simple in_string()) Next, on the search results page, the user's browser > executes this javascript snippit in an iframe (to create a sandbox) and > injects any new HTML into the actual search results. > > I think allowing javascript snippits would provide a good mix between ease > of development, security, as well as robustness. A javascript widget would > be able to do anything that a server side widget could accomplish. It would > be able to fetch and process RSS feeds, make cross site XHR requests, and > leverage additional "enviroment" variables we choose to make available to > it. > > Using javascript would also (hopefully) eliminate the security woes. > > Additionally, we could enable persistent storage by providing a JSON > interface into HBase (similar to KT maybe?) > > Anyway, these are just some thoughts. There are obviously problems, for > example there would be no easy way to leverage APIs that don't have a > RESTFul interface (ie Youtube) without additional infastructure. > > Ashish > > > On Wed, Jul 23, 2008 at 9:41 AM, Mark (Markie) > wrote: > >> >> >> On Wed, Jul 23, 2008 at 11:21 AM, Rainer Blome >> wrote: >> >>> Mark (Markie) wrote: >>> > On Tue, Jul 22, 2008 at 7:12 PM, Jimmy Wales >> > > wrote: >>> > >>> > Mark (Markie) wrote: >>> > > a very quick mockup of how widgets could look @ >>> > > http://people.swlabs.org/~markie/widgetmockup.png >>> > >>> > > >>> > > this is purely my input but i welcome more input on this >>> > >>> > >>> http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search >>> > < >>> http://www.google.com/search?hl=en&safe=off&q=weather+94128&btnG=Search> >>> > >>> http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search >>> > < >>> http://www.google.com/search?hl=en&safe=off&q=SFO+to+JFK&btnG=Search> >>> > >>> > Google puts them right at the top of the page. >>> > >>> > >>> > my thinking on using the space at the side was that by putting >>> > (possibly multiple) widgets at the top of the page shifts down the >>> > results, but this can be good and bad as the widget hopefully provides >>> > them with the best answer. however to me it seemed to be a big waste >>> > of space on the right so i thought it could be filled, but i dont feel >>> > that strongly as to its position. >>> Free space is often a good thing. It makes a page easier to use. >>> Once there are more "resultlets" than would fit on one page, >>> even in the "minimized" form shown in your mock-up, >>> you'll long for the days when the results page looked relatively clean >>> :). >>> And note that you "cheated": the space you used for the "widgets" is >>> often filled with about three ads :-). >> >> darn caught :-p >> >> >>> >>> >>> Jer's idea of ranking "widgets" like ordinary results looks to me like >>> the right answer to the question of which and how much space to give to >>> "resultlets". >>> >>> On the other hand, when there are so many star-rated results >>> (which currently always appear before non-rated results, correct?) >>> that new results and "resultlets" do not show up at all on the first >>> page, >>> they will be hard to see. >>> >>> There may even be "resultlets" which generate more than one result, for >>> example postal codes for multiple countries, making the competition even >>> harder. >>> >>> In order to give new "resultlets" a chance >>> (and maybe new ordinary results as well?), >>> they might be "advertised" in a dedicated area (portlet) on the right. >>> For users who don't want to see this, that area should be collapsible. >> >> hmm seems like a good idea to me: get them noticed then move them into the >> "normal/main" :-) >> >> regards >> >> mark >> >> >>> >>> >>> Rainer >>> >>> _______________________________________________ >>> Search-UI mailing list >>> Search-UI at wikia.com >>> http://lists.wikia.com/mailman/listinfo/search-ui >>> >> >> >> _______________________________________________ >> Search-UI mailing list >> Search-UI at wikia.com >> http://lists.wikia.com/mailman/listinfo/search-ui >> >> > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080724/c0c514b1/attachment-0001.html From jwales at wikia.com Fri Jul 25 19:58:18 2008 From: jwales at wikia.com (Jimmy Wales) Date: Fri, 25 Jul 2008 15:58:18 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> Message-ID: <488A305A.6010405@wikia.com> This is very interesting. :) adatta02 at gmail.com wrote: > The big concern with allowing server side code is security. From the > chat, it sounds like the plan is to hand review any code that will be > run on the server and reject "dangerous" code. This seems like a > daunting if not down right impossible task as any reasonably complicated > piece of code is going to be extremely hard to certify as "safe". There > are various examples of sneaking nefarious code into a seemingly safe > code block famously http://en.wikipedia.org/wiki/Underhanded_C_Contest Well, this is a typical problem faced by open source projects all the time. Whom do we trust to "commit"? I think we just need to have clear terms and be careful. Excessive paranoia is not warranted. > From a developer's standpoint - the big gain of being able to run > server side code is the ability to interact with the OS layer (files, > sockets, databases, ect) and libraries (GD, standard libraries, ect) . > However, it seems like there is no way we would be able to provide > access to any OS features while simultanously gaurenteeing security. It > also seems like providing persistent storage through this framework > would require significant infastructure. I don't think that's such a big part of it. I would say that for both us and for developers, the major gain of running server side is performance and integration. I suppose there are 3 possibilities: 1. "pure" client side - javascript can compute 3*7 and return 21. No risk, all is well. EXAMPLE: calculator 2. "pure" server side - our servers compute something complex and relying on databases, etc. - one huge advantage of doing this is that it makes the code all open-source 3. third-party "RESTful" calls - no risk, but performance is an issue, as is proprietary-ness EXAMPLE: weather > What I'd propose is to construct a framework that mirrors the Google > igoogle system. Following from this, developers would be able to write > javascript blocks that run on some set of triggers (either regex matches > or simple in_string()) Next, on the search results page, the user's > browser executes this javascript snippit in an iframe (to create a > sandbox) and injects any new HTML into the actual search results. I have no opposition to this, except for the possible performance issue... we can require that before we send out the javascript, it has to be open source. > I think allowing javascript snippits would provide a good mix between > ease of development, security, as well as robustness. A javascript > widget would be able to do anything that a server side widget could > accomplish. It would be able to fetch and process RSS feeds, make cross > site XHR requests, and leverage additional "enviroment" variables we > choose to make available to it. I am nearly convinced. From adatta02 at gmail.com Wed Jul 30 02:09:27 2008 From: adatta02 at gmail.com (adatta02 at gmail.com) Date: Tue, 29 Jul 2008 22:09:27 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <488A305A.6010405@wikia.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> Message-ID: <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> We've been playing around with "widgets" and other "injections" for the last few days. So far, we've broken down search manipulations into "searchlets" and "widgets". As Aaron put it, "searchlets" consist of searches that support searching within some subset of the universal search space. We've been defining these using a ":" syntax. An example of this search is http://fp029.sjc.wikia-inc.com/search/search.html#twitter:earthquake which uses summize to search twitter. We also have "video", "users", and "torrent" built which presents results in a similar fashion (in the normal search view with KT accessible). Other "searchlets" we were throwing around are "now:" which would query the "nowsphere" (twitter, latest blog posts, ect) to try and get a sense of what people are saying about something right now. The vague aspects of "searchlets" seem to be presentation and how they interact with KT. A big question is should searchlets display in a different layout than regular search (ie should we be able to display videos in a table instead of a flat list? http://fp029.sjc.wikia-inc.com/search/search.html#video:wikia ) The KT question is basically what happens in the corner cases. For example, on a resultset for twitter:earthquake if a user "stars" or "spotlights" an item and then the item fall off the first page (twitter searches are sorted by date). Should this item automatically promote to the first page or should the KT actions be ignored? We've also been playing around with how to impliment and display "widgets" but haven't come to much of a consensus. So far, we've been "defining" widgets as additional discrete pieces of information (on top of the nutch/KT results). Jeff's implimented several widgets directly in our JS that display in-line with the search results. For example, http://fp029.sjc.wikia-inc.com/search/search.html#traffic%2010003 renders traffic information as a normal search result. We also experimented with rendering the widgets on top of the results - http://fp029.sjc.wikia-inc.com/search/search.html#Newark,%20NJ The issue then is, should widgets render in-line with the other results and therefore be subject to the same KT operations (star, delete, spotlight, ect) or should they render in a seperate area and be unaffected by KT. Also, what would the behavior be for keywords that return multiple widgets (ie weather 10003) Ashish On Fri, Jul 25, 2008 at 3:58 PM, Jimmy Wales wrote: > This is very interesting. :) > > adatta02 at gmail.com wrote: > > The big concern with allowing server side code is security. From the > > chat, it sounds like the plan is to hand review any code that will be > > run on the server and reject "dangerous" code. This seems like a > > daunting if not down right impossible task as any reasonably complicated > > piece of code is going to be extremely hard to certify as "safe". There > > are various examples of sneaking nefarious code into a seemingly safe > > code block famously http://en.wikipedia.org/wiki/Underhanded_C_Contest > > Well, this is a typical problem faced by open source projects all the > time. Whom do we trust to "commit"? I think we just need to have clear > terms and be careful. Excessive paranoia is not warranted. > > > From a developer's standpoint - the big gain of being able to run > > server side code is the ability to interact with the OS layer (files, > > sockets, databases, ect) and libraries (GD, standard libraries, ect) . > > However, it seems like there is no way we would be able to provide > > access to any OS features while simultanously gaurenteeing security. It > > also seems like providing persistent storage through this framework > > would require significant infastructure. > > I don't think that's such a big part of it. I would say that for both > us and for developers, the major gain of running server side is > performance and integration. > > I suppose there are 3 possibilities: > > 1. "pure" client side - javascript can compute 3*7 and return 21. No > risk, all is well. EXAMPLE: calculator > > 2. "pure" server side - our servers compute something complex and > relying on databases, etc. - one huge advantage of doing this is that it > makes the code all open-source > > 3. third-party "RESTful" calls - no risk, but performance is an issue, > as is proprietary-ness EXAMPLE: weather > > > What I'd propose is to construct a framework that mirrors the Google > > igoogle system. Following from this, developers would be able to write > > javascript blocks that run on some set of triggers (either regex matches > > or simple in_string()) Next, on the search results page, the user's > > browser executes this javascript snippit in an iframe (to create a > > sandbox) and injects any new HTML into the actual search results. > > I have no opposition to this, except for the possible performance > issue... we can require that before we send out the javascript, it has > to be open source. > > > I think allowing javascript snippits would provide a good mix between > > ease of development, security, as well as robustness. A javascript > > widget would be able to do anything that a server side widget could > > accomplish. It would be able to fetch and process RSS feeds, make cross > > site XHR requests, and leverage additional "enviroment" variables we > > choose to make available to it. > > I am nearly convinced. > > > > > _______________________________________________ > Search-UI mailing list > Search-UI at wikia.com > http://lists.wikia.com/mailman/listinfo/search-ui > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080729/da56dea5/attachment.html From aaron.wright at gmail.com Wed Jul 30 03:52:08 2008 From: aaron.wright at gmail.com (Aaron Wright) Date: Tue, 29 Jul 2008 23:52:08 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> Message-ID: On Tue, Jul 29, 2008 at 10:09 PM, wrote: > A big question is should searchlets display in a different layout than > regular search (ie should we be able to display videos in a table instead of > a flat list? http://fp029.sjc.wikia-inc.com/search/search.html#video:wikia ) I think it's worth experimenting with different ui "patterns" when slicing search results. For example, people digest images and video differently from text. I probably would prefer to see a side by side gallery of images rather than have each image listed one right below eachother. Videos might also be better digested as a gallery. Why not reflect this in the result pattern? I think there is potential for searchlets beyond just the "nowsphere", videos, images, and torrents. Let's say for arguments sake that we could categorize huge throngs of search data as being an "opinion" and another throng as being "encyclopedic". A user wanting to view only "opinion" based search results, would just have to type "opinion:<>" or if the user just wants to view "encyclopedic" based search results they would only have to type "encyclopedic:<>". Or "sports", "fashion", "botany" etc. > The issue then is, should widgets render in-line with the other results and > therefore be subject to the same KT operations (star, delete, spotlight, > ect) or should they render in a seperate area and be unaffected by KT. Also, > what would the behavior be for keywords that return multiple widgets (ie > weather 10003) I think widgets should be layered on top of search results, rendered inline, and subject to the same KT operations as other search results. Why? Because it will provide useful data. If people are deleting the widget, then there is a good argument the widget should be removed on these results. If people are continuing to annotate the data produced by the widget, then the widget probably needs to be tweaked to incorporate those changes. --Aaron From jeffrey.tierney at gmail.com Wed Jul 30 04:31:52 2008 From: jeffrey.tierney at gmail.com (Jeffrey J Tierney) Date: Wed, 30 Jul 2008 00:31:52 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: References: <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> Message-ID: <9131eace0807292131ye888d3fv69c7475c878cdb7c@mail.gmail.com> > I think widgets should be layered on top of search results, rendered > inline, and subject to the same KT operations as other search results. > Why? Because it will provide useful data. If people are deleting > the widget, then there is a good argument the widget should be removed > on these results. If people are continuing to annotate the data > produced by the widget, then the widget probably needs to be tweaked > to incorporate those changes. > > One problem that this presents, however, is with widgets that are providing time sensitive / news-type data. For example, if the weather widget is editable, and someone changes the data for "New York weather", then since the kt edits would override the service generated data (even after the service data changes), anyone using our widget to quickly check the weather for NYC is going to get incorrect information, presumably until someone who knows how to fix it realizes that it has been manually edited. In my opinion, if the widget framework is going to catch on, then we have to be careful that the information that is being presented from these services is as reliable as possible. I agree that there will be some widgets (perhaps even the majority) that would benefit from being subjected to the wisdom of the crowds, and that on those we should definitely let the community have the same abilities as with regular search results, but I think that we need to take into account that their are some (weather, traffic, movie showtimes, etc) where data reliability is more of a concern, and that these may need to be handled slightly differently. Perhaps this comes in a deciding on a clearer definition of searchlets and widgets. Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080730/454697e9/attachment-0001.html From aaron.wright at gmail.com Wed Jul 30 04:43:52 2008 From: aaron.wright at gmail.com (Aaron Wright) Date: Wed, 30 Jul 2008 00:43:52 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> Message-ID: I think Ashish provided an excellent definition for widget and searchlet. On Tue, Jul 29, 2008 at 10:09 PM, wrote: > "searchlets" consist of searches that support searching > within some subset of the universal search space. We've been defining these > using a ":" syntax. An example of this search is > http://fp029.sjc.wikia-inc.com/search/search.html#twitter:earthquake which > uses summize to search twitter. We also have "video", "users", and "torrent" > built which presents results in a similar fashion (in the normal search view > with KT accessible). > So far, we've been "defining" > widgets as additional discrete pieces of information (on top of the nutch/KT > results). From jeffrey.tierney at gmail.com Wed Jul 30 04:45:44 2008 From: jeffrey.tierney at gmail.com (Jeffrey J Tierney) Date: Wed, 30 Jul 2008 00:45:44 -0400 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <9131eace0807292131ye888d3fv69c7475c878cdb7c@mail.gmail.com> References: <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> <9131eace0807292131ye888d3fv69c7475c878cdb7c@mail.gmail.com> Message-ID: <9131eace0807292145j41d50b9cgb6af003bfe0ff5f5@mail.gmail.com> After re-reading Ashish's email, I think he did a great job of providing that clearer definition between searchlets and widgets, so disregard the last line of my previous email. Sorry Ashish :) The real point of my last email was just to state that widgets, as defined as "additional discrete pieces of information (on top of the nutch/KT results)", would benefit from some of the kt actions (delete, rating), but that some would be negatively affected by allowing the data to be editable, and that if there was a way to possibly limit which actions could be done on these, that might be a better solution. On Wed, Jul 30, 2008 at 12:31 AM, Jeffrey J Tierney < jeffrey.tierney at gmail.com> wrote: > > I think widgets should be layered on top of search results, rendered >> inline, and subject to the same KT operations as other search results. >> Why? Because it will provide useful data. If people are deleting >> the widget, then there is a good argument the widget should be removed >> on these results. If people are continuing to annotate the data >> produced by the widget, then the widget probably needs to be tweaked >> to incorporate those changes. >> >> > One problem that this presents, however, is with widgets that are providing > time sensitive / news-type data. For example, if the weather widget is > editable, and someone changes the data for "New York weather", then since > the kt edits would override the service generated data (even after the > service data changes), anyone using our widget to quickly check the weather > for NYC is going to get incorrect information, presumably until someone who > knows how to fix it realizes that it has been manually edited. In my > opinion, if the widget framework is going to catch on, then we have to be > careful that the information that is being presented from these services is > as reliable as possible. > > I agree that there will be some widgets (perhaps even the majority) that > would benefit from being subjected to the wisdom of the crowds, and that on > those we should definitely let the community have the same abilities as with > regular search results, but I think that we need to take into account that > their are some (weather, traffic, movie showtimes, etc) where data > reliability is more of a concern, and that these may need to be handled > slightly differently. Perhaps this comes in a deciding on a clearer > definition of searchlets and widgets. > > Jeff > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.wikia.com/pipermail/search-ui/attachments/20080730/e2cd9378/attachment.html From avatar at wikia.com Wed Jul 30 05:43:57 2008 From: avatar at wikia.com (Tim Bartel) Date: Wed, 30 Jul 2008 07:43:57 +0200 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> <488A305A.6010405@wikia.com> <24a809760807291909kab5f7acl79723095cb1c7c0e@mail.gmail.com> Message-ID: Hi, 2008/7/30 : > As Aaron put it, "searchlets" consist of searches that support searching > within some subset of the universal search space. We've been defining these > using a ":" syntax. An example of this search is > http://fp029.sjc.wikia-inc.com/search/search.html#twitter:earthquake which > uses summize to search twitter. This is a dev server, but just to mention it: German umlauts are broken again (wrong encoding). Bye, Tim. -- http://de.wikia.com From rainer.blome at gmx.de Wed Jul 30 19:33:47 2008 From: rainer.blome at gmx.de (Rainer Blome) Date: Wed, 30 Jul 2008 21:33:47 +0200 Subject: [search-ui] [Search-l] Widgets In-Reply-To: <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> References: <4884B486.5040405@wikia.com> <488622F9.3050307@wikia.com> <48870619.7050608@gmx.de> <24a809760807231642h316b9cd2vcacb450e626aaf2e@mail.gmail.com> Message-ID: <4890C21B.2030105@gmx.de> adatta02 at gmail.com wrote: > I think allowing javascript snippits would provide a good mix between > ease of development, security, as well as robustness. Agree. > A javascript widget would be able to do anything that a server side > widget could accomplish. Javascript and server-side are not necessarily exclusive, there are Javascript application servers out there. However, I think it's a good idea to, by default, let the clients aggregate and present the results. > It would be able to fetch and process RSS feeds, make cross > site XHR requests, Why would a Javascript "widget" be able to make cross site XMLHTTPRequests? From a security standpoint, CSS is dangerous to allow. > and leverage additional "enviroment" variables we > choose to make available to it. What do you mean by "environment variables"? What do you mean by "we choose to make available"? Rainer From jwales at wikia.com Tue Jul 22 19:14:32 2008 From: jwales at wikia.com (Jimmy Wales) Date: Tue, 22 Jul 2008 19:14:32 -0000 Subject: [search-ui] [Fwd: some widget framework thoughts] Message-ID: <48863178.6020505@wikia.com> Here is the "vaporware documentation" I mentioned... this is something I half wrote-up a while back to describe what I am thinking of. It is very basic, and I have actually said a bit more on this list. One of the key questions is performance... performance for a dictionary lookup will be fine, but what is harder for me to know (and I would love some thoughts or advice on this from anyone) is whether it is possible to do full scale regular expression matching on dozens or hundreds or thousands of expressions in realtime. That is, someone types in a query, and we test whether it matches against a pretty open ended set of things... that would be cool, but is it a performance nightmare? -------- Original Message -------- Subject: some widget framework thoughts Date: Mon, 09 Jun 2008 16:00:57 -0400 From: Jimmy Wales To: Wikia New York and Wikia Search staff I started writing up a help document for weak programmers like me to write apps for our nonexistence widget framework. The primary purpose of this project was to trigger some thinking about where I want us to head with this, as opposed to generating an actually useful document. :) Still, please read it over... it is short, and let me know what is impossible or what is possible and easy, that I did not mention. I want us to release early, release often, and make it super easy for people to insert "objects" via this framework. Well, read it and you will have an idea what I mean. -------------- next part -------------- A non-text attachment was scrubbed... Name: WikiaSearchWidgetFramework.pdf Type: application/pdf Size: 69617 bytes Desc: not available Url : http://lists.wikia.com/pipermail/search-ui/attachments/20080722/1ecc3e83/attachment-0001.pdf