Add a search utility

Add a search utility Implement a dark theme Add an FAQ page with some info
2020-12-28 04:01:07 +00:00 · 2020-12-28 04:01:07 +00:00 · 3a22c15dcd
parent 8a60675236
commit 3a22c15dcd
12 changed files with 318 additions and 48 deletions
--- a/assets/faq.html
+++ b/assets/faq.html
@ -0,0 +1,86 @@
+<!DOCTYPE html>
+<html lang="en">
+	<head>
+		<meta charset="utf-8">
+		<title>&#x1f351;</title>
+		<link href="/_css/milligram.css" rel="stylesheet">
+		<link href="/_css/style.css" rel="stylesheet">
+	</head>
+	<body class="container">
+		<main class="wrapper">
+			<h1>FAQ</h1>
+			An attempt to answer frequently asked questions, and document
+			the detail of site features. As with all software documentation,
+			this is how the developer <b>thinks</b> these things work,
+			to find out how things actually work, see the source code.
+			<h2>How do a register an account?</h2>
+			You can go <a href="/_claim">here</a> to register an account,
+			although you can also post pastes and comments anonymously.
+			<h2>Parsers? What are those?</h2>
+			Parsers are the way you can style the text you post. While
+			this software started as a pastebin clone, the need to style
+			text better than pastebin.com quickly became apparent. Two
+			parsers are currently available:
+			<h3>The plain parser</h3>
+			<p>The goal of the plain parser is to format plain text exactly as you would expect it. It does not try to to anything fancy and tries to be faithful to the plaintext representation.
+			<h3>Imageboard parser</h3>
+			<p>The imageboard parser delivers an approximation of Infinity markup, and will be familiar to those who have used imageboards before. The imageboard parser supports the following markup:
+			<p>Surround text with double single-quotes(') to make text <i>italic</i>
+			<p>Surround text with triple single-quotes to make text <b>bold</b>
+			<p>Surround text with underscores(_) to make it <u>underlined</u>
+			<p>Surround text with double asterisks(*) to make it <span class="spoiler">spoilered</span>
+			<p>Surround text with tildes(~) to make it <s>strike through</s>
+			<p>Begin a line with a greater-than followed by a space to make it
+			<p><span class="greentext">&gt;greentext</span>
+			<p>Begin a line with a less-than followed by a space to make it
+			<p><span class="pinktext">&lt;pinktext</span>
+			<p>Surround text with forum-style [spoiler] and [/spoiler] tags as a second way to <span class="spoiler2">spoiler</span>
+			<p>Surround text with forum-style [code] and [/code] tags to make it
+
+			<pre><code>
+Preformatted and monospace
+			</code></pre>
+
+			<p>If you have incomplete markup at the end, it shouldn't break anything, let me know if it does.
+			<h2>How does search work?</h2>
+			<p>The search utility searches for stories on the site.
+			At it's most simple, it searches stories based on tags,
+			but it can also filter stories based on: title, author,
+			date, and hits. In general, the syntax for search is
+			<code>+|-&lt;field&gt;&gt;operator&lt;&gt;value&lt;</code>
+			<p>The first <code>+</code> or <code>-</code> specifies
+			weather to include or exclude results based on this
+			search, the <code>&lt;field&gt;</code> specifies what
+			field to search for (or search based on tag if this is
+			missing), and <code>&lt;operator&gt;</code> specifies
+			how to search.
+			<p>For title, and author, the only allowed operator is
+			<code>=</code>. This operator will search for the text
+			appearing anywhere in the field, case insensitive. For
+			hits and time, the allowed operators are
+			<code>&gt;</code>,<code>&lt;</code>,<code>&gt;=</code>,
+			<code>&lt;=</code>,<code>=</code>, which searches for
+			greater than, less than, greater than or equal to, less
+			than or equal to, and strictly equal to, respectively.
+			Tag does not need a field or operator, and only allows
+			exact matches. As a quirk of this system, it is
+			impossible to search for the tags "author", "title",
+			"hits" or "date",
+
+			Examples:
+
+			<pre><code>+author=admin -meta</code></pre>
+			Will return all stories by the users "admin" and "badminton_enthusiast" that do not include the "meta" tag.
+			<pre><code>+hits&gt;20 -date&lt;=1609459201</code></pre>
+			Will return all stories with more than 20 hits that were posted before January 1, 2021 (unix timestamp 1609459201).<br/>
+			While the date field is a little hard to use for humans, it may be useful for robots.
+			<h2>How do I enable reader mode on my story?</h2>
+			Unfortunately, there is no web standard about how reader
+			modes for different browsers work, and no way to hint to
+			the browser that a page is readable. The site makes a
+			best effort to trigger browsers into thinking a post is
+			readable, but this is unreliable. In general, if reader mode
+			is broken you just need to make your post a little longer.
+		</main>
+	</body>
+</html>
--- a/assets/style.css
+++ b/assets/style.css
@ -20,7 +20,19 @@ p,.tag-list{margin-bottom:0px}
 	padding: 0 1em 0 1em;
 	margin: 0 1px 0 1px;
 }
+.search{
+	display:flex !important;
+	flex:0.5 0.5 auto !important;
+}
+.search>.button{
+	flex:10 10 auto;
+	translate: -100%;
+}
+.column-0{margin-right:5px;}

@media (prefers-color-scheme: dark){
-	@import "css/style_dark.css";
+	body, form>*{
+		background: #1c1428;
+		color: #d0d4d8 !important;
+	}
 }
--- a/conf/smr.conf
+++ b/conf/smr.conf
@ -36,6 +36,7 @@ domain * {
 	route  /_css/milligram.css asset_serve_milligram_css
 	route  /_css/milligram.min.css.map asset_serve_milligram_min_css_map
 	route  /_css/style_dark.css asset_serve_style_dark_css
+	route  /_faq asset_serve_faq_html
 	route  /favicon.ico asset_serve_favicon_ico
 	route  /_paste  post_story
 	route  /_edit   edit_story
@ -77,7 +78,7 @@ domain * {
 		validate tags v_any
 	}
 	params get /_search {
-		validate tag v_any
+		validate q v_any
 	}
 	params get ^/[^_].* {
 		validate comments v_bool
--- a/src/lua/endpoints/download_get.lua
+++ b/src/lua/endpoints/download_get.lua
@ -25,7 +25,7 @@ local function download_get(req)
 	local err = util.do_sql(stmnt_download)
 	if err == sql.DONE then
 		--No rows, story not found
-		http_responose(req,404,pages.nostory{path=story})
+		http_response(req,404,pages.nostory{path=story})
 		stmnt_download:reset()
 		return
 	end
--- a/src/lua/endpoints/index_get.lua
+++ b/src/lua/endpoints/index_get.lua
@ -52,21 +52,17 @@ local function get_author_home(req)
 	stmnt_author_bio:bind_names{author=subdomain}
 	local err = util.do_sql(stmnt_author_bio)
 	if err == sql.DONE then
-		print("No such author")
 		stmnt_author_bio:reset()
 		return pages.noauthor{
 			author = subdomain
 		}
 	end
-	print("err:",err)
 	assert(err == sql.ROW,"failed to get author:" .. subdomain .. " error:" .. tostring(err))
 	local data = stmnt_author_bio:get_values()
 	local bio = data[1]
 	stmnt_author_bio:reset()
-	print("Getting author's stories")
 	stmnt_author:bind_names{author=subdomain}
 	err = util.do_sql(stmnt_author)
-	print("err:",err)
 	local stories = {}
 	while err == sql.ROW do
 		local data = stmnt_author:get_values()
--- a/src/lua/endpoints/search_get.lua
+++ b/src/lua/endpoints/search_get.lua
@ -6,6 +6,7 @@ local util = require("util")
 local libtags = require("tags")
 local pages = require("pages")
 local config = require("config")
+local search_parser = require("parser_search")

 local stmnt_search
 local oldconfigure = configure
@ -18,42 +19,40 @@ local function search_get(req)
 	local host = http_request_get_host(req)
 	local path = http_request_get_path(req)
 	http_request_populate_qs(req)
-	local tag = http_argument_get_string(req,"tag")
-	if tag then
-		stmnt_search:bind_names{
-			tag = tag
-		}
-		local results = {}
-		local err
-		repeat
-			err = stmnt_search:step()
-			if err == sql.BUSY then
-				coroutine.yield()
-			elseif err == sql.ROW then
-				local id, title, anon, time, author = unpack(stmnt_search:get_values())
-				local idp = util.encode_id(id)
-				local tags = libtags.get(id)
-				table.insert(results,{
-					id = idp,
-					title = title,
-					anon = anon,
-					time = os.date("%B %d %Y",tonumber(time)),
-					author = author,
-					tags = tags
-				})
-			elseif err == sql.DONE then
-				stmnt_search:reset()
-			else
-				error("Failed to search, sql error:" .. tostring(err))
+	local searchq = assert(http_argument_get_string(req,"q"))
+	log(LOG_DEBUG,string.format("search: %q",searchq))
+	local sqltxt, data = search_parser(searchq)
+	local stmnt = assert(db.conn:prepare(sqltxt), db.conn:errmsg())
+	local i = 1
+	for field,values in pairs(data) do
+		if field ~= "tags" then
+			for _, value in pairs(values) do
+				stmnt:bind(i,value[3])
+				i = i + 1
 			end
-		until err == sql.DONE
-		local ret = pages.search{
-			domain = config.domain,
-			results = results,
-			tag = tag,
-		}
-		http_response(req,200,ret)
+		end
 	end
+	for _,value in pairs(data.tags) do
+		stmnt:bind(i,value[3])
+		i = i + 1
+	end
+	local results = {}
+	for row in stmnt:rows() do
+		table.insert(results,{
+			id = util.encode_id(row[1]),
+			title = row[2],
+			isanon = row[3] == 1,
+			author = row[4],
+			time = os.date("%B %d %Y",tonumber(row[5])),
+			tags = libtags.get(row[1])
+		})
+	end
+	local ret = pages.search{
+		domain = config.domain,
+		results = results,
+		q = searchq,
+	}
+	http_response(req,200,ret)
 end

 return search_get
--- a/src/lua/parser_search.lua
+++ b/src/lua/parser_search.lua
@ -1,10 +1,84 @@
+local lpeg = require('lpeg')
+local etlua = require('etlua')
+local args = {...}
+lpeg.locale(lpeg)
+local V,P,C,S,B,Cs = lpeg.V,lpeg.P,lpeg.C,lpeg.S,lpeg.B,lpeg.Cs
+--Identity function
+local ident = function(a) return a end
+--Lowercase and capitalize first letter
+local function capitalize(str)
+	return string.lower(str):gsub("^(.)",string.upper)
+end
+--Trim whitespace
+local function trim(str)
+	return str:match("^%s*(.-)%s*$")
+end
+--SQL like match anywhere
+local function like(str)
+	return "%" .. str .. "%"
+end
+--Tags are always trimed of whitepsace, lowercase with first letter capitalized
+local function tag_fmt(str)
+	return capitalize(trim(str))
+end
+--Title match
+local function title_fmt(str)
+	return like(trim(str))
+end
+--Author names are all lowercase alphanumeric, max 30 characters
+local function author_fmt(str)
+	return like(string.lower(trim(str)))
+end
+local fieldnames = {
+	title = {name="title",type="string",fmt=title_fmt},
+	author = {name="author",type="string",fmt=author_fmt},
+	date = {name="date",type="number",fmt=tonumber},
+	hits = {name="hits",type="number",fmt=tonumber},
+	tags = {name="tag",type="string",fmt=tag_fmt},
+}
+local field_default = "tag"
+local fields
+local grammar = P{
+	"chunk";
+	whitespace = S" \t\n"^0,
+	itm = C(P(1-S"+-")^0), --go until the next '+' or '-'
+	likefield = C(P"title" + P"author") * V"whitespace" * C(P"=") * V"whitespace" * V"itm",
+	rangeop = P"<=" + P">=" + P">" + P"<" + P"=",
+	rangefield = C(P"date" + P"hits") * V"whitespace" * C(V"rangeop") * V"whitespace" * C(V"itm"),
+	field = C(S"+-") * (V"likefield" + V"rangefield" + V"itm") / function(pn,field,expr,value)
+		if expr and value then
+			fields[field] = fields[field] or {}
+			table.insert(fields[field],{pn,expr,value})
+		else
+			fields.tags = fields.tags or {}
+			table.insert(fields.tags,{pn,"=",field})
+		end
+	end,
+	chunk = V"field"^0

+}
 --Grammar
 --Transpile a sting with + and - into an sql query that searches tags
+local fname = "pages/search_sql.etlua"
+local sqltmpl = assert(io.open(fname))
+local c = etlua.compile(sqltmpl:read("*a"),fname)
+sqltmpl:close()
 local function transpile(str)
-	for chunk in str:gmatch("([+-])([^+-])") do
-		print("found chunk:",chunk)
+	str = string.lower(str)
+	fields = {tags={}}
+	table.concat({grammar:match(str)}," ")
+	--Sanity perform formatting on data
+	for field,values in pairs(fields) do
+		for _,value in pairs(values) do
+			local pn, expr, val = unpack(value)
+			local nval = fieldnames[field].fmt(val)
+			value[3] = nval
+		end
 	end
+	local ressql = c{
+		result = fields
+	}
+	return ressql, fields
 end

 return transpile
--- a/src/lua/search_sql.etlua
+++ b/src/lua/search_sql.etlua
@ -0,0 +1,44 @@
+SELECT
+	posts.id,
+	posts.post_title,
+	authors.name,
+	posts.post_time
+FROM
+	posts,authors
+WHERE
+	authors.id = posts.authorid
+<% print("reqult:",result) -%>
+<% for field, values in pairs(result) do -%>
+	<% print("values:",values) -%>
+	<% for _,value in pairs(values) do -%>
+		<% local pn,expr,value = unpack(value) -%>
+		<% local n = (pn == "+" and "" or "NOT") -%>
+		<% if field == "title" then -%>
+			AND <%= n %> posts.post_title LIKE ?
+		<% elseif field == "author" then -%>
+			AND <%= n %> authors.name LIKE ?
+		<% elseif field == "date" then -%>
+			AND <%= n %> posts.post_time <%- expr %> ?
+		<% elseif field == "hits" then -%>
+			AND posts.views <%- expr -%> ?
+		<% end -%>
+	<% end -%>
+<% end -%>
+<% print("tags:",result.tags) %>
+<% for _,tag in pairs(result.tags) do -%>
+INTERSECT
+SELECT
+	posts.id,
+	posts.post_title,
+	authors.name,
+	posts.post_time
+FROM
+	posts,authors,tags
+WHERE
+	posts.authorid = authors.id
+	AND tags.postid = posts.id
+	<% local n,v = unpack(tag) -%>
+	<% n = (n == "-" and "NOT" or "") -%>
+	AND <%= n %> tags.tag = <%= v %>
+<% end -%>
+;
--- a/src/lua/util.lua
+++ b/src/lua/util.lua
@ -45,7 +45,7 @@ function util.sqlbind(stmnt,call,position,data)
 	assert(call == "bind" or call == "bind_blob","Bad bind call, call was:" .. call)
 	local f = stmnt[call](stmnt,position,data)
 	if f ~= sql.OK then
-		error(string.format("Failed to %s at %d with %q: %s", call, position, data, db:errmsg()),2)
+		error(string.format("Failed to %s at %d with %q: %s", call, position, data, db.conn:errmsg()),2)
 	end
 end

--- a/src/pages/index.etlua
+++ b/src/pages/index.etlua
@ -15,9 +15,15 @@
 			</h1>

 			<div class="container">
-				<a href="/_paste" class="button">New paste</a>
-				<a href="/_login" class="button">Log in</a>
-				<a href="/_claim" class="button">Register</a>
+				<div class="row">
+					<a href="/_paste" class="button column column-0">New paste</a>
+					<a href="/_login" class="button column column-0">Log in</a>
+					<a href="/_claim" class="button column column-0">Register</a>
+					<form action="https://<%= domain %>/_search" method="get" class="search column row">
+						<input class="column" type="text" name="q" placeholder="+greentext -dotr +hits>20"/>
+						<input class="column column-0 button button-clear" type="submit" value="&#x1F50E;"/>
+					</form>
+				</div>
 				<p>
 				Welcome to slash.monster, stories of fiction and fantasy<br/>
 				Not safe for work<br/>
@ -43,7 +49,7 @@
 						</td><td>
 							<ul class="row tag-list">
 							<% for i = 1,math.min(#v.tags, 5) do %>
-								<li><a class="tag button button-outline" href="https://<%= domain %>/_search?tag=<%= v.tags[i] %>"><%= v.tags[i] %></a></li>
+								<li><a class="tag button button-outline" href="https://<%= domain %>/_search?q=+<%= v.tags[i] %>"><%= v.tags[i] %></a></li>
 							<% end %>
 							<% if #v.tags > 5 then %>
 								<li>+<%= #v.tags - 5 %></li>
--- a/src/pages/search.etlua
+++ b/src/pages/search.etlua
@ -11,6 +11,12 @@
 			<h1 class="title">
 				<a href="https://<%= domain %>"><%= domain %></a>/
 			</h1>
+			<div class="row">
+				<form action="https://<%= domain %>/_search" method="get" class="search column row">
+					<input class="column" type="text" name="q" placeholder="+greentext -dotr +hits>20" value="<%= q %>"/>
+					<input class="column column-0 button button-clear" type="submit" value="&#x1F50E;"/>
+				</form>
+			</div>
 			<div class="content">
 				<% if #results == 0 then %>
 				No stories matched your search.
@ -32,6 +38,9 @@
 							<% for i = 1,math.min(#v.tags, 5) do %>
 								<li><a class="tag button button-outline" href="https://<%= domain %>/_search?tag=<%= v.tags[i] %>"><%= v.tags[i] %></a></li>
 							<% end %>
+							<% if #v.tags > 5 then %>
+								<li>+<%= #v.tags - 5%></li>
+							<% end %>
 							</ul>
 						</td><td>
 							<%= v.time %>
--- a/src/pages/search_sql.etlua
+++ b/src/pages/search_sql.etlua
@ -0,0 +1,43 @@
+SELECT
+	posts.id,
+	posts.post_title,
+	posts.isanon,
+	authors.name,
+	posts.post_time
+FROM
+	posts,authors
+WHERE
+	authors.id = posts.authorid
+<% for field, values in pairs(result) do -%>
+	<% for _,value in pairs(values) do -%>
+		<% local pn,expr,value = unpack(value) -%>
+		<% local n = (pn == "+" and "" or "NOT") -%>
+		<% if field == "title" then -%>
+			AND <%= n %> posts.post_title LIKE ?
+		<% elseif field == "author" then -%>
+			AND <%= n %> authors.name LIKE ?
+		<% elseif field == "date" then -%>
+			AND <%= n %> posts.post_time <%- expr %> ?
+		<% elseif field == "hits" then -%>
+			AND posts.views <%- expr -%> ?
+		<% end -%>
+	<% end -%>
+<% end -%>
+<% for _,tag in pairs(result.tags) do -%>
+INTERSECT
+SELECT
+	posts.id,
+	posts.post_title,
+	posts.isanon,
+	authors.name,
+	posts.post_time
+FROM
+	posts,authors,tags
+WHERE
+	posts.authorid = authors.id
+	AND tags.postid = posts.id
+	<% local n,v,t = unpack(tag) -%>
+	<% n = (n == "-" and "NOT" or "") -%>
+	AND <%= n %> tags.tag = ?
+<% end -%>
+;