team picohttps://blog.pico.sh2024-03-06T02:09:47Zofficial blog for pico.sh servicespicoRFC A link aggregator service2024-03-07T16:56:19Zhttps://blog.pico.sh/rfc-link-aggregator<p>We want to create a link aggregator service that can only be accessible via SSH.
Think hacker news but authentication and authorization happens via SSH.</p>
<p>The trick is making it usable.</p>
<p>Here's how we imagine it could work:</p>
<ul>
<li>User initiates an SSH tunnel to links.sh (imaginary name)</li>
<li>links.sh launches a dedicated web server for that user</li>
<li>This allows the user to access links.sh from localhost</li>
<li>We automatically authenticate and authorize the user accessing links.sh</li>
<li>User can read, comment, and post articles in their browser</li>
</ul>
<p>We would also create an SSH app to allow the user to easily submit links without
having to setup an SSH tunnel</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl">ssh links.sh post https://bower.sh/front-end-complexity
</span></span></code></pre><p>How does this differentiate from existing link aggregators? Well, this will not
be widely accessible because users need to understand how to use SSH. This will
add a barrier for mobile users to access this website. In fact, we will not
build a mobile friendly version of the website at all.</p>
<blockquote>
<p>Our hypothesis is that mobile access to a link aggregator degrades discussion.</p>
</blockquote>
<p>This will be an exclusive service for hackers: a clubhouse for us to relish in
our own intrigue. Steeped in our exclusive discussion. What makes us excited
about this idea is having a passwordless user experience, all relying on the
familiar public-key cryptography. Even further this has really interesting
implications: We don't need cookies or <code>Authorization</code> headers for
authentication. No passwords, no cookies, no http headers, only a lone SSH
command.</p>
<hr/>
<p>Join our <a href="https://pico.sh/irc" rel="nofollow">irc <span class="hashtag">#pico</span>.sh on libera</a> or email us at
<a href="mailto:~erock/pico.sh@lists.sr.ht" rel="nofollow">~erock/pico.sh@lists.sr.ht</a>.</p>
<p>Be sure to subscribe to our <a href="/rss" rel="nofollow">rss feed</a> to get the latest updates at team
pico.</p>
RFC pico pro2024-03-07T16:56:19Zhttps://blog.pico.sh/rfc-pico-pro-2023-11-06<h1 id="mission-statement"><a class="anchor" href="#mission-statement" rel="nofollow">#</a> Mission statement</h1>
<p>We want to build tools and services that are useful for software development. We
want to empower individual contributors to rapidly prototype and boost
productivity.</p>
<h1 id="design-goals"><a class="anchor" href="#design-goals" rel="nofollow">#</a> Design goals</h1>
<ul>
<li>Primary directive is to be useful to ourselves</li>
<li>Developer tools and services</li>
<li>Focus on the individual developer</li>
<li>Enable developers to rapidly prototype</li>
<li>Ability to host small web services</li>
<li>Authentication with SSH</li>
<li>Move fast and break things</li>
</ul>
<h1 id="what-it-is-not"><a class="anchor" href="#what-it-is-not" rel="nofollow">#</a> What it is not</h1>
<ul>
<li>Not a PaaS</li>
<li>Not designed for teams or organizations</li>
<li>Not going to provide 99.99% uptime</li>
<li>It's not going to scale with demand</li>
</ul>
<h1 id="services"><a class="anchor" href="#services" rel="nofollow">#</a> Services</h1>
<ul>
<li>static site hosting <a href="https://pgs.sh" rel="nofollow">pgs.sh</a></li>
<li>tunnels as a service <a href="https://tuns.sh" rel="nofollow">tuns.sh</a></li>
<li><a href="https://github.com/89luca89/distrobox" rel="nofollow">distrobox</a> as a service</li>
<li>docker compose as a service
<a href="https://github.com/antoniomika/pcompose" rel="nofollow">pcompose</a></li>
</ul>
<h2 id="static-site-hosting"><a class="anchor" href="#static-site-hosting" rel="nofollow">#</a> static site hosting</h2>
<p>An easy to use static site hosting platform. Publish your sites as easy as
copying files to our SSH app.</p>
<h3 id="todo"><a class="anchor" href="#todo" rel="nofollow">#</a> TODO</h3>
<ul>
<li>Figure out storage, bandwidth limits</li>
</ul>
<h2 id="distrobox-as-a-service"><a class="anchor" href="#distrobox-as-a-service" rel="nofollow">#</a> distrobox as a service</h2>
<p>Create a development container in the cloud. Use familiar linux distros. Remote
into your dev container from any client.</p>
<h3 id="todo-1"><a class="anchor" href="#todo-1" rel="nofollow">#</a> TODO</h3>
<ul>
<li>One big benefit of distrobox is that it lets you mount your home directory
<ul>
<li>Do we want to support persistent disk for distrobox containers?</li>
<li>That way a user can spin up as many containers as they want and they have
the same persisted home directory</li>
</ul>
</li>
<li>Many developers need access to Docker, which is a technical challenge</li>
<li>Could we link this service with our docker compose one?</li>
<li>For example, I could have a distrobox with arch but I need a postgres server
and I'd like to run it inside a docker container
<ul>
<li>I know there's a way to proxy docker commands to a remote docker daemon, I
wonder if we could leverage that here?</li>
</ul>
</li>
<li>How many containers do we want to support?</li>
<li>Do we want to provide an ephemeral version of this? (e.g. <code>docker run --rm</code>)</li>
<li>Do we want to automatically load remote vscode?</li>
</ul>
<h2 id="docker-compose-as-a-service"><a class="anchor" href="#docker-compose-as-a-service" rel="nofollow">#</a> docker compose as a service</h2>
<p>If all you need is a place to host your simple services, use docker compose.
This service is not designed to scale, rather, a quick place to get services up
and running with minimal effort. This service is also great for rapidly
prototyping and getting a product or service out-the-door quickly.</p>
<h3 id="todo-2"><a class="anchor" href="#todo-2" rel="nofollow">#</a> TODO</h3>
<ul>
<li>How do we handle network isolation?</li>
<li>How do we charge?</li>
<li>How do we technically allow pcompose to be multi-tenant?</li>
<li>Should we provide officially supported images for some apps/services/tools?
<ul>
<li>git repos</li>
<li>image registry</li>
<li>CI/CD</li>
</ul>
</li>
<li>How many distinct docker compose files should we support?
<ul>
<li>Current thinking is just one but that might be limiting</li>
</ul>
</li>
</ul>
<h2 id="tunnels-as-a-service"><a class="anchor" href="#tunnels-as-a-service" rel="nofollow">#</a> tunnels as a service</h2>
<p>Need to access <code>localhost</code> from <code>https</code>? Not only that, but we also use tunnels
to allow you to connect to all your other containers.</p>
<h3 id="todo-3"><a class="anchor" href="#todo-3" rel="nofollow">#</a> TODO</h3>
<ul>
<li>Can we leverage tuns for network isolation?</li>
<li>Can we leverage tuns to connect to all user containers?</li>
<li>Can we leverage tuns for web tunnels?</li>
</ul>
<h1 id="pricing"><a class="anchor" href="#pricing" rel="nofollow">#</a> Pricing</h1>
<p>We would like to keep pricing as simple as possible to reduce overhead. The
current idea is we only offer a yearly subscription service. Ideally we would be
able to charge somewhere around $20/yr, but that might change depending on how
much compute we offer users. I think we could implement a tier pricing model but
that is kind of a pain. It would be better if there was just one single plan
that works for most users.</p>
<h3 id="todo-4"><a class="anchor" href="#todo-4" rel="nofollow">#</a> TODO</h3>
<ul>
<li>How much do we want to charge users?</li>
<li>How much compute do we want to offer users?</li>
<li>Do we want to allow users to horizontally scale their compute?</li>
<li>Do we want to allow users to vertically scale their compute?</li>
</ul>
<hr/>
<p>Join our <a href="https://pico.sh/irc" rel="nofollow">irc <span class="hashtag">#pico</span>.sh on libera</a> or email us at
<a href="mailto:~erock/pico.sh@lists.sr.ht" rel="nofollow">~erock/pico.sh@lists.sr.ht</a>.</p>
<p>Be sure to subscribe to our <a href="/rss" rel="nofollow">rss feed</a> to get the latest updates at team
pico.</p>
What does charging for services look like for us?RFC imgs.sh2024-03-07T16:56:19Zhttps://blog.pico.sh/rfc-imgs<p>The pico team has been thinking about a new premium image hosting service. We
haven't written a single line of code yet but have spent time thinking about it.
This document serves as our proposal not only for how the service ought to
function, but also details about the technical implementation.</p>
<h1 id="imgs-the-service"><a class="anchor" href="#imgs-the-service" rel="nofollow">#</a> imgs the service</h1>
<p>It's an image hosting service. Users will be able to upload their images along
with metadata about the image (e.g. title, caption, date, tags). The intention
is to store the images permanently until service is canceled.</p>
<p>This will be a paid service. We are considering a "pico pro" plan or just
charging for this service as a stand alone. More details on that later, but it's
enough to know that this will not be a service offered for free.</p>
<p>Based on <a href="https://hey.prose.sh/imgs-market-research" rel="nofollow">previous research</a>, and in
order to stay competitive with other image hosting services, we would need
similar features:</p>
<ul>
<li>No trial
<ul>
<li>If we do want to offer a trial, we delete the images after 10 minutes</li>
</ul>
</li>
<li>Permanent storage</li>
<li>Public only
<ul>
<li>No private images allowed</li>
</ul>
</li>
<li>Hotlinking enabled</li>
<li>Ad free</li>
<li>No JS
<ul>
<li>But we don't want to sacrifice on UX so we might want to introduce a little
bit of javascript, primarily with gallery views</li>
</ul>
</li>
<li>Nominal storage
<ul>
<li>We don't want to encourage whales here</li>
<li>We should limit the storage, somewhere in the range of 25-100GB</li>
</ul>
</li>
</ul>
<h2 id="the-twist"><a class="anchor" href="#the-twist" rel="nofollow">#</a> the twist</h2>
<p>SSH app to upload images and metadata. You should also be able to easily
download the images using ssh. All content management happens inside the
terminal and with a key pair. Our target audience are people that are
comfortable with the terminal.</p>
<p>The default route to the image would be optimized for the device requesting to
view the image. We would read the <code>User-Agent</code> to try to understand the device
and then change the quality and resolution based on that information.</p>
<p>We would also provide ways to specify:</p>
<ul>
<li>width</li>
<li>height</li>
<li>preserve aspect ratio</li>
<li>quality</li>
<li>original image</li>
</ul>
<p>We would also seamlessly integrate with any of the services we create that could
benefit from sharing images.</p>
<p>This service would be all about sharing media with other people, that's why it's
public only. We don't want to be a data warehouse for all personal media.</p>
<h2 id="photo-albums"><a class="anchor" href="#photo-albums" rel="nofollow">#</a> photo albums</h2>
<p>We also want to support photo albums on <code>imgs.sh</code>. To implement this feature, we
are leveraging tagging. So the user will be able to add tags to their image
which we will then aggregate into photo albums.</p>
<h1 id="why"><a class="anchor" href="#why" rel="nofollow">#</a> why?</h1>
<p>We think this service would be genuinely useful to terminal enthusiasts who want
to quickly take a pic and share it in chat or use it in a blog.</p>
<p>We also received a few questions asking about charging for our services in order
to help sustain it long term. No one wants to join a platform that then
disappears after a year. This really is a confluence of us wanting to host
images for our personal blogs, imagining others would find it useful, and also a
way to help support service costs and active development.</p>
<h1 id="moderation"><a class="anchor" href="#moderation" rel="nofollow">#</a> moderation</h1>
<p>This is not a "post whatever image you want without repercussions" image hosting
service. We will accept DMCA take down notices and we will ban users for posting
illegal images. I think we should also reject pornographic images.</p>
<p>Moderation is going to be the biggest time sink with this service so we need a
system in place to make it painless. In the beginning, I think we should have an
RSS feed of all images posted to <code>imgs.sh</code> and review them. If there is an image
posted that we deem inappropriate, we will remove the image and potentially ban
the user who published it.</p>
<p>We should also provide a reporting endpoint so users can report images for us to
review.</p>
<h1 id="closed-beta"><a class="anchor" href="#closed-beta" rel="nofollow">#</a> closed beta</h1>
<p>In the beginning the service will be online but closed to registration. To
enroll in the beta program, users <strong>must</strong> join our IRC channel and request an
invite. They will then be able to use the service for free while we tweak the
imgs for mass adoption. We will make no guarantees about uptime, service
reliability, or even the possibility that their images will be deleted.</p>
<h1 id="tos-and-privacy-policy"><a class="anchor" href="#tos-and-privacy-policy" rel="nofollow">#</a> tos and privacy policy</h1>
<p>We need to make sure we have these docs locked down since this will be a premium
service.</p>
<h1 id="technical-details"><a class="anchor" href="#technical-details" rel="nofollow">#</a> technical details</h1>
<p>I think we should build this to potentially support multi-region. But we would
implement this service similarly to our other services. I think we will be able
to leverage our CMS to handle most of the heavy lifting. Uploading an image
would use <code>scp</code> and we would store the image inside the <code>posts</code> table.</p>
<p>Then we would build out a web api for retrieving the images.</p>
<h2 id="third-party-services-interacting-with-imgs"><a class="anchor" href="#third-party-services-interacting-with-imgs" rel="nofollow">#</a> third-party services interacting with imgs</h2>
<p>Since we have a monorepo setup, we could pretty easily just reach into the code
for <code>imgs</code> inside <code>prose</code> and perform the necessary operations within <code>prose</code>.</p>
<p>We could also figure out a clean way to send the images to <code>imgs</code> using agent
forwarding or x509 certs. The "third-party service" (e.g. prose) would request a
certificate that they could use to send uploads to <code>imgs</code> on behalf of a user.</p>
<p>However, for the MVP, I think we should just use the code directly in our
services.</p>
<p>I do think it's important that services we don't control should still have a
path to using <code>imgs</code> that can upload images on behalf of a user.</p>
<h2 id="where-do-we-host-the-files"><a class="anchor" href="#where-do-we-host-the-files" rel="nofollow">#</a> where do we host the files?</h2>
<p>This is tricky. We could store the files to S3 or some other object storage, but
the costs are pretty high. We could store the files directly on our VM FS, but
we'd need to make sure we have enough space and it can scale. I'm going to defer
to Antonio for this section.</p>
<h2 id="integration-with-pico-services"><a class="anchor" href="#integration-with-pico-services" rel="nofollow">#</a> integration with pico services</h2>
<p>The entire point of this service is to enhance our pico services with image
hosting capabilities, so it's critical we figure out the ergonomics of
integration this service with pico.</p>
<p>Ideally, the user would be able to upload images on <code>prose</code> and we would reach
out to the <code>imgs</code> service to store them. Once the image has been uploaded to
<code>imgs</code> any reference to the image would be swapped at runtime inside <code>prose</code>.</p>
<p>Let me demonstrate an example workflow inside a <code>prose</code> blog:</p>
<p>User's blog folder at <code>~/blog</code>:</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl">blog/
</span></span><span class="line"><span class="ln">2</span><span class="cl"> trip-to-paris.jpg <span class="c1"># image to upload to imgs</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl"> trip-to-paris.md <span class="c1"># metadata for image</span>
</span></span><span class="line"><span class="ln">4</span><span class="cl"> tour-to-paris.md <span class="c1"># blog post that contains reference to image</span>
</span></span></code></pre><p>Inside <code>tour-to-paris.md</code> we would have something like:</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl">---
</span></span><span class="line"><span class="ln">2</span><span class="cl">title: My trip to paris!
</span></span><span class="line"><span class="ln">3</span><span class="cl">---
</span></span><span class="line"><span class="ln">4</span><span class="cl">
</span></span><span class="line"><span class="ln">5</span><span class="cl">My trip was great! Here is a pic from my trip
</span></span><span class="line"><span class="ln">6</span><span class="cl">
</span></span><span class="line"><span class="ln">7</span><span class="cl">![](/trip-to-paris.jpg)
</span></span><span class="line"><span class="ln">8</span><span class="cl">
</span></span><span class="line"><span class="ln">9</span><span class="cl">It's a tourist trap but we couldn't resist checking it out.
</span></span></code></pre><p>Inside <code>trip-to-paris.md</code> we would provide metadata for the image:</p>
<pre class="chroma"><code><span class="line"><span class="ln"> 1</span><span class="cl">---
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">title: Close up to the eiffel tower
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">date: 2022-08-04
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">tags: [paris]
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">caption: Eiffel tower, Tower in Paris, late morning.
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">---
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="gu">## Day 1
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="gu"></span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">We arrived in Paris around 5 AM. When we eventually found the place to pick up
</span></span><span class="line"><span class="ln">11</span><span class="cl">our rented car, and got our first warning to "park facing the wind so you don't
</span></span><span class="line"><span class="ln">12</span><span class="cl">lose your doors," we started towards the Eiffel tower. I looked out into the
</span></span><span class="line"><span class="ln">13</span><span class="cl">darkness, mostly only able to see the slight embankments on both sides of the
</span></span><span class="line"><span class="ln">14</span><span class="cl">road, and was reminded of Hawaii, where I was this time last year — large,
</span></span><span class="line"><span class="ln">15</span><span class="cl">rolling hills all barren of trees.
</span></span></code></pre><p>It might seem weird that we have a description of the image in a separate
markdown file, but this metadata will also be posted to <code>imgs.sh</code> and not just
<code>prose</code>. A user doesn't have to add metadata to their image but it provides
potentially important information (e.g. tags, date) that could be useful on its
own inside the <code>imgs</code> service. We would pull that data into the blog post if it
exists.</p>
<p>Example: <a href="https://snap.as/matt/iceland/zUUoCon" rel="nofollow">https://snap.as/matt/iceland/zUUoCon</a></p>
<p>Once the content is written, the user would upload all files to <code>prose</code>:</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl">scp ~/blog/*.md ~/blog/*.jpg erock@prose.sh:
</span></span></code></pre><p>When we upload all the files, since we now have two markdown files that need to
go to different locations, we will need special logic:</p>
<ul>
<li>If there's an image, upload to <code>imgs</code></li>
<li>If there's a markdown file that matches the name of an image, upload to <code>imgs</code></li>
<li>Otherwise upload markdown files to <code>prose</code></li>
</ul>
<p>Now when a blog post is requested, we do a few things:</p>
<ul>
<li>Find the markdown post</li>
<li>Scan for relative image urls</li>
<li>Replace the URL with <code>imgs.sh</code> url</li>
<li>Also query for the metadata</li>
<li>Embellish the markdown with metadata</li>
<li>Convert markdown to HTML</li>
</ul>
<p>Before:</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl">---
</span></span><span class="line"><span class="ln">2</span><span class="cl">title: My trip to paris!
</span></span><span class="line"><span class="ln">3</span><span class="cl">---
</span></span><span class="line"><span class="ln">4</span><span class="cl">
</span></span><span class="line"><span class="ln">5</span><span class="cl">My trip was great! Here is a pic from my trip
</span></span><span class="line"><span class="ln">6</span><span class="cl">
</span></span><span class="line"><span class="ln">7</span><span class="cl">![](/trip-to-paris.jpg)
</span></span><span class="line"><span class="ln">8</span><span class="cl">
</span></span><span class="line"><span class="ln">9</span><span class="cl">It's a tourist trap but we couldn't resist checking it out.
</span></span></code></pre><p>After:</p>
<pre class="chroma"><code><span class="line"><span class="ln"> 1</span><span class="cl">---
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">title: My trip to paris!
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">---
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">My trip was great! Here is a pic from my trip
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">[<span class="nt">![Close up to the eiffel tower</span>](<span class="na">https://erock.imgs.sh/trip-to-paris.jpg</span>)](https://erock.imgs.sh/trip-to-paris)
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="ge">_Eiffel tower, Tower in Paris, late morning._</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="gu">## Day 1
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="gu"></span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">We arrived in Paris around 5 AM. When we eventually found the place to pick up
</span></span><span class="line"><span class="ln">13</span><span class="cl">our rented car, and got our first warning to "park facing the wind so you don't
</span></span><span class="line"><span class="ln">14</span><span class="cl">lose your doors," we started towards the Eiffel tower. I looked out into the
</span></span><span class="line"><span class="ln">15</span><span class="cl">darkness, mostly only able to see the slight embankments on both sides of the
</span></span><span class="line"><span class="ln">16</span><span class="cl">road, and was reminded of Hawaii, where I was this time last year — large,
</span></span><span class="line"><span class="ln">17</span><span class="cl">rolling hills all barren of trees.
</span></span><span class="line"><span class="ln">18</span><span class="cl">
</span></span><span class="line"><span class="ln">19</span><span class="cl">It's a tourist trap but we couldn't resist checking it out.
</span></span></code></pre><p>Adding the markdown from the metadata file into the blog post might be overkill
and be awkward so we could get rid of it. My guess is whatever is inside the
metadata file should be rendered in the blog post but that's something we can
discuss.</p>
<hr/>
<p>Join our <a href="https://pico.sh/irc" rel="nofollow">irc <span class="hashtag">#pico</span>.sh on libera</a> or email us at
<a href="mailto:~erock/pico.sh@lists.sr.ht" rel="nofollow">~erock/pico.sh@lists.sr.ht</a>.</p>
<p>Be sure to subscribe to our <a href="/rss" rel="nofollow">rss feed</a> to get the latest updates at team
pico.</p>
proposal for imgs serviceRFC RSS Service2024-03-07T16:56:19Zhttps://blog.pico.sh/rfc-rss<p>RSS/Atom is a great companion in the smol web. It's relatively standard, easy to
write, easy to consume, and provide users with choice on how to view their
feeds.</p>
<p>I think an RSS service using an SSH app could be useful.</p>
<h1 id="market-research"><a class="anchor" href="#market-research" rel="nofollow">#</a> market research</h1>
<p>Here are some other RSS readers in the market: <a href="https://hey.lists.sh/rss-readers" rel="nofollow">https://hey.lists.sh/rss-readers</a></p>
<h1 id="features"><a class="anchor" href="#features" rel="nofollow">#</a> features</h1>
<ul>
<li>Keypair authentication</li>
<li>Ability to upload feeds</li>
<li>Ability to upload <a href="https://en.wikipedia.org/wiki/OPML" rel="nofollow">opml</a> file</li>
<li>We would manage fetching feeds and keeping them up-to-date</li>
<li>We could send an email digest (if they provide their email)</li>
<li>Provide a web view for the feeds</li>
</ul>
<h1 id="what-can-we-offer-over-the-other-readers"><a class="anchor" href="#what-can-we-offer-over-the-other-readers" rel="nofollow">#</a> what can we offer over the other readers?</h1>
<p>We would try to provide a great reading experience from the terminal. No need to
install an RSS reader like newsboat. No need to sync a config file across
multiple apps. Just go to your rss read homepage and start reading. Furthermore,
many of the readers do not provide an rss-to-email feature and most rss-to-email
services do not provide readers so there's an interesting opportunity here to
capture both audiences.</p>
<p>The other nice thing about an RSS reader app is that it ties into our other
services that leverage RSS as well. It's hard to let users know of new features
when they aren't notified about them.</p>
<p>By providing a service that emails users of our services, it would hopefully
improve our communication with our users.</p>
<p>Because the web version doesn't require authentication, anyone could navigate to
any user's feed collection and read its content. This would also provide mobile
support for users since they can just navigate to our website. The only issue is
we might have to deal with content security policy and ensuring we could render
the html content consistently. It definitely opens us open to a bunch of edge
cases. Creating a proxy service might be necessary in that case.</p>
<h1 id="how-it-works"><a class="anchor" href="#how-it-works" rel="nofollow">#</a> how it works</h1>
<p>A user would <code>scp</code>:</p>
<ul>
<li>an <code>opml</code> file</li>
<li>a file containing a lists of rss feeds</li>
<li>or <code>echo "<feed>" | ssh reads.sh</code></li>
</ul>
<p>It doesn't matter how many opml or feed files the user uploads, we would dedupe
them when figuring out how to fetch their feeds. Because an RSS feed can contain
a bunch of metadata about a feed, we should capture as much of that as possible
inside the <code>posts</code> table. The downside is we use <code>posts</code> for a lot of our
services (e.g. lists, prose, and pastes) so we want to be careful not to
overload this table. Having said that, I think an rss feed fits into the post
paradigm. We just need to add a <code>data jsonb</code> column to <code>posts</code>.</p>
<pre class="chroma"><code><span class="line"><span class="ln">1</span><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">posts</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="n">jsonb</span><span class="p">;</span><span class="w">
</span></span></span></code></pre><h2 id="fetching"><a class="anchor" href="#fetching" rel="nofollow">#</a> fetching</h2>
<p>We want to be smart about how we fetch feeds because it could be resource
intensive if the service gets big enough.</p>
<p>What would trigger us fetching feeds?:</p>
<ul>
<li>Maybe we just use a cron?</li>
<li>Prior to sending out daily email digest</li>
<li>When the user requests to view the feed on our web site</li>
</ul>
<p>Fetching feeds can be a little tricky since some feeds do not provide the html
inside their atom entry. Instead they provide a link for users to click on to
navigate to their site. This kind of defeats the purpose of using RSS so we
could just render the link and force users to open their browser. Or we fetch
the link provided in the atom entry and store the html in our database. This
would probably provide a better user experience but it opens us open to a slew
of edge cases and weird behavior.</p>
<h2 id="email-digest"><a class="anchor" href="#email-digest" rel="nofollow">#</a> email digest</h2>
<p>I also think that if we do send out a daily digest, we add a button in the email
that they need to click within 30 days or else we disable sending them an email.
They click the button in the email -> we delay shutdown for 30 days.</p>
<h2 id="tracking-feed-entries"><a class="anchor" href="#tracking-feed-entries" rel="nofollow">#</a> tracking feed entries</h2>
<p>We would probably create a separate table for the feed results in order to
optimize storing an retrieval.</p>
<pre class="chroma"><code><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">feed_entry</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="n">uuid</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">NILL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">uuid_generate_v4</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w"> </span><span class="n">post_id</span><span class="w"> </span><span class="n">uuid</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="nb">boolean</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="k">false</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w"> </span><span class="n">author</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">250</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w"> </span><span class="n">category</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">250</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w"> </span><span class="n">published</span><span class="w"> </span><span class="k">timestamp</span><span class="w"> </span><span class="k">without</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="k">zone</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">NOW</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w"> </span><span class="n">rights</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">2000</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">2000</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w"> </span><span class="n">content</span><span class="w"> </span><span class="nb">text</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w"> </span><span class="n">contributor</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">250</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w"> </span><span class="n">atom_id</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">250</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">2000</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="w"> </span><span class="n">summary</span><span class="w"> </span><span class="nb">text</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="w"> </span><span class="n">title</span><span class="w"> </span><span class="nb">character</span><span class="w"> </span><span class="nb">varying</span><span class="p">(</span><span class="mi">250</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="w"> </span><span class="n">created_at</span><span class="w"> </span><span class="k">timestamp</span><span class="w"> </span><span class="k">without</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="k">zone</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">NOW</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="w"> </span><span class="n">updated_at</span><span class="w"> </span><span class="k">timestamp</span><span class="w"> </span><span class="k">without</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="k">zone</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">NOW</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">entry_unique_atom_id</span><span class="w"> </span><span class="k">UNIQUE</span><span class="w"> </span><span class="p">(</span><span class="n">atom_id</span><span class="p">,</span><span class="w"> </span><span class="n">post_id</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">feed_entry_pkey</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">fk_entry_posts</span><span class="w">
</span></span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">KEY</span><span class="p">(</span><span class="n">post_id</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="w"> </span><span class="k">REFERENCES</span><span class="w"> </span><span class="n">posts</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">DELETE</span><span class="w"> </span><span class="k">CASCADE</span><span class="w">
</span></span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">UPDATE</span><span class="w"> </span><span class="k">CASCADE</span><span class="w">
</span></span></span><span class="line"><span class="ln">25</span><span class="cl"><span class="w"></span><span class="p">);</span><span class="w">
</span></span></span></code></pre><h2 id="queue-system"><a class="anchor" href="#queue-system" rel="nofollow">#</a> queue system</h2>
<p>We will probably also want a queuing system. I figured we could just build one
that fits our purposes inside our database.</p>
<pre class="chroma"><code><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">TYPE</span><span class="w"> </span><span class="n">JOB_STATUS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">ENUM</span><span class="w"> </span><span class="p">(</span><span class="s1">'in_progress'</span><span class="p">,</span><span class="w"> </span><span class="s1">'success'</span><span class="p">,</span><span class="w"> </span><span class="s1">'fail'</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w"></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TYPE</span><span class="w"> </span><span class="n">JOB_TYPE</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">ENUM</span><span class="w"> </span><span class="p">(</span><span class="s1">'fetch_feed'</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w"></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="k">IF</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="n">app_queue</span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="n">uuid</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="n">NILL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">uuid_generate_v4</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w"> </span><span class="n">post_id</span><span class="w"> </span><span class="n">uuid</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="n">JOB_STATUS</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="n">JOB_TYPE</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w"> </span><span class="k">input</span><span class="w"> </span><span class="n">jsonb</span><span class="w"> </span><span class="o">#</span><span class="w"> </span><span class="n">params</span><span class="w"> </span><span class="n">needed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">execute</span><span class="w"> </span><span class="n">job</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w"> </span><span class="k">output</span><span class="w"> </span><span class="n">jsonb</span><span class="w"> </span><span class="o">#</span><span class="w"> </span><span class="k">result</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">job</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w"> </span><span class="n">created_at</span><span class="w"> </span><span class="k">timestamp</span><span class="w"> </span><span class="k">without</span><span class="w"> </span><span class="n">time</span><span class="w"> </span><span class="k">zone</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">NOW</span><span class="p">(),</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">queue_pkey</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="p">(</span><span class="n">id</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">fk_queue_posts</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="w"> </span><span class="k">FOREIGN</span><span class="w"> </span><span class="k">KEY</span><span class="p">(</span><span class="n">post_id</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="w"> </span><span class="k">REFERENCES</span><span class="w"> </span><span class="n">posts</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">DELETE</span><span class="w"> </span><span class="k">CASCADE</span><span class="w">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">UPDATE</span><span class="w"> </span><span class="k">CASCADE</span><span class="w">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="w"></span><span class="p">);</span><span class="w">
</span></span></span></code></pre><h1 id="metadata"><a class="anchor" href="#metadata" rel="nofollow">#</a> metadata</h1>
<p>I haven't figured out a great way for users to add metadata to their feeds. For
example, if they want to add tags to a feed so they could view a collection of
feeds in one list. We could do it within the CMS but I feel like it would be
better if there were a file format that could do that for us. the <code>opml</code> format
seems like a good candidate.</p>
<p>I like the idea of storing the results in the database, but I could also see an
argument for using something more ephemeral like redis.</p>
<h1 id="risks"><a class="anchor" href="#risks" rel="nofollow">#</a> risks</h1>
<ul>
<li>RSS readers have been done before</li>
<li>Syncing feeds can be costly in terms of compute resources</li>
<li>Following atom entry links to the webpage puts us in the scraping category
which opens us up to stability issues (e.g. some sites deny scrapers)</li>
<li>Web view might run into content security policy issues</li>
</ul>
<hr/>
<p>Join our <a href="https://pico.sh/irc" rel="nofollow">irc <span class="hashtag">#pico</span>.sh on libera</a> or email us at
<a href="mailto:~erock/pico.sh@lists.sr.ht" rel="nofollow">~erock/pico.sh@lists.sr.ht</a>.</p>
<p>Be sure to subscribe to our <a href="/rss" rel="nofollow">rss feed</a> to get the latest updates at team
pico.</p>
A proposal for an RSS service