By Kenton Varda - 14 Jan 2024
TL;DR: Primary maintainership of the Sandstorm project has changed, and a along with that, the project has moved to sandstorm.org.
I’ll start with the point: I, Kenton Varda, am no longer the maintainer of the Sandstorm project. In fact, I haven’t really been for quite some time. Instead, a group of avid Sandstorm users, led by Jacob “ocdtrekkie” Weisz, have been doing a lot more work than me. Many have been involved for quite a few years now.
Yet until recently I have continued to hold the role of gatekeeper: I was the one who could push releases, merge changes, approve blog posts, etc., but was failing to do so. That doesn’t make sense.
So, I’m relinquishing that control. Sandstorm now belongs to the Sandstorm Community under Open Source Collective.
However, Jake and I agree that this change should be very explicit. People coming to the Sandstorm web site today are often confused about Sandstorm’s status, even thinking it is backed by a company – we need to make things clear. And, it doesn’t necessarily make sense for for people who who opted into auto-updates back when Sandstorm was a company to keep getting them from the community. Not only because those users may not be comfortable with it, but also because the community itself may not be. Sandstorm’s users included (and may still include – there’s no way for us to tell) companies, newspapers, educational institutions, research laboratories, and even government agencies. An auto-update that, say, breaks SAML login could do a lot of damage overnight. For the community to make progress, the stakes need to be a little lower.
To that end, the new project is going to live under a new domain, sandstorm.org. Hopefully, the .org TLD, along with some design changes, makes it clear that this is a community project, not a company. sandstorm.io will stay as it is, but with the banner you see above flagging it as historical. If and when sandstorm.org decides to push auto-updates, users will have to manually opt into them.
For my part, I continue to be a Sandstorm user, and as a user I do hope the project is able to make progress under new maintainership. Time will tell.
For those curious how we got here, here’s the whole story from the beginning…
Long ago – circa 2014 to 2016 – Sandstorm was a startup, of which I was co-founder. We ran a successful crowdfunding campaign in 2014 and raised VC money in early 2015. We hired 5 people, making a team of 7. We had aimed for a series A round in 2016. We saw a lot of developer excitement for our product on Hacker News and the like, and we expected to follow in the footsteps of other startups who were able to raise a pre-revenue series A based on such excitement. Unfortunately, that did not happen. For whatever reason – and I could list several possible reasons, but I’ll never really know which it was – investors weren’t interested. This set off a panic where we lowered our sights to raising a “bridge round” while trying to produce actual revenue from enterprise sales. But this made things worse: we did not have any idea how enterprise sales worked. We made two sales ever, for a measly 4 figures. Investors ran away.
By the end of 2016, it was obvious the company was going to crash, so we started looking for an acquisition. Too late, of course – our potential acquirers knew it was a fire sale. There was a common saying in Silicon Valley startup circles at the time that in a pure acquihire, your company is worth $1M per engineer. We learned quickly what should have been obvious: this is a fanciful myth. You cannot block another company from hiring your employees, so why on Earth would they pay you – or rather, your investors – for the right? Just to be nice? Several companies were very interested in our engineers, but the best they offered was to acquire the company for $0 and help us wind it down, while making job offers to some or all of the employees.
One company (whom I won’t name, but you’ve heard of) actually told us: “We will add up all the salaries and stock grants we offer your employees, and that’s what you can say was the exit price. If you’d like, you can take a chunk of the equity and give it to your investors rather than the employees.” I found this extremely disturbing. Another company (you’ve heard of them too) interviewed the whole team and then informed us they were passing on the founders but hiring two employees.
Another company we talked to was Cloudflare. They too offered to acquire the company for $0, but they made good offers to the whole team without any funny business. Additionally, they had a project in mind for me. They said: “We know we want to let people run code on our edge network, but we haven’t figured out exactly how. We could give you ownership of this.” I was intrigued. And after paying myself $60k/year for several years, boy did that regular salary look nice.
But I still very much wanted Sandstorm to succeed. So I told Cloudflare, I’ll come work for you, but I want to keep Sandstorm the company, and be able to work on it as an open source project. They agreed.
So in early 2017 we announced that Sandstorm was moving to a community model and, shortly after, that most of the team joined Cloudflare.
For some time, I continued to work actively on Sandstorm, and a motley crew of community members helped out. But at the same time, by day, I was having the most fun of my career, having been given a greenfield project to hack on with no distractions. That project became Cloudflare Workers. Within six months we released a beta, and (coincidentally) exactly a year after I joined Cloudflare, we officially launched the product. The product has continued to grow on a startup-like trajectory ever since.
So, I was (and am) now essentially the founder of a startup-within-Cloudflare which was actually succeeding. It became hard to put energy into Sandstorm, the project that felt like a failure, even if I still believed in it. Sandstorm became increasingly a chore: Once a month I would review and merge pull requests, update dependencies, and push a new build. If I spent time on it beyond that, it was to shut down the paid hosting service (which never had enough users to make money) or switch sandcats.io to Let’s Encrypt when our previous contract with Globalsign expired.
Over time, even just basic maintenance became difficult. Sandstorm uses MongoDB for metadata storage, but it is stuck on version 2.6, which is ancient. I couldn’t update it because updating Mongo requires manual intervention, but Sandstorm is something that thousands of users have installed and expect to auto-update. Most of them aren’t even aware they are running Mongo. Maybe I could have written code that would automatically handle the migration? That sounded like more work than I was interested in. Eventually, though, the Mongo drivers used by Meteor were updated to a version that didn’t support 2.6. So now I couldn’t update Meteor, Sandstorm’s primary dependency. And we can’t update Node, because old Meteor doesn’t work on newer Node. (That, by the way, is largely because V8 made a change which permanently broke node-fibers, which Meteor deeply depended on. Oh the tangled web we weave…)
During all this time, the Sandstorm web site remained mostly unchanged, save for the occasional blog post. I think this has been misleading. The site was designed when Sandstorm was a company, by our full-time professional designer. We made some changes after the company shut down, but it still looks like a company’s web site. People seem to stumble across the page and believe it is backed by a full-time team rather than a few volunteers. From time to time it even shows up on the front page of Hacker News.
I have felt bad about this, but couldn’t quite figure out what to do. Redesign the web site to make it look less professional? I’m not quite sure even how to do that, but I’m pretty sure I don’t have time.
In late 2022, the one person who had been contributing code changes to Sandstorm with some regularity, Ian “zenhack” Denhardt, decided to embark on a total rewrite called Tempest, so contributions to the main project dried up. In early 2023, I gave up pushing monthly releases, since there seemed to be no point: no code changes had been made and no dependencies could be updated.
And then in mid-2023, tragically, Ian passed away in an accident. I’m no good at eulogies, but suffice to say it was a big loss for Sandstorm, Cap’n Proto, and other communities Ian was in.
Later in 2023, Jake wrote a blog post, originally to be posted on sandstorm.io, requiring my approval. The post was ambitious, and even though I wasn’t being asked to do anything except approve it, I felt that doing so implied a commitment on my part that I wasn’t prepared to make. Instead, we made the decision that I should take myself out of the loop. We came up with the plan to transfer leadership of the project. Sandstorm-the-company had long owned the domain sandstorm.org but not used it; I decided it made sense for this to become the new home of the community project and transferred it to them.
For reasons I can’t go into here, I was unable to dissolve Sandstorm the company until some time in 2022. (It turns out I really should have accepted the $0 offer from Cloudflare, to make it their problem, but alas.) But once I could finally do it, our investors and I agreed on a dissolution plan that allowed Sandstorm’s remaining IP to be transferred to a non-profit entity representing the community. Of course, the code is open source under the Apache license, so anyone can freely use that. But, the agreement also covers trademarks, domain names, etc. Open Source Collective is a qualifying non-profit.
And so, here we are. Sandstorm now belongs to Sandstorm.org.
By Ian Denhardt - 17 Jun 2021
Hey Everyone!
Members of the Sandstorm community are doing some fundraising!
Specifically,
The organizers are familiar faces, but we aren’t and haven’t ever been formally affiliated with Sandstorm, Inc; we’re just long-time community members looking to chart a path for the Sandstorm project going forward. We all miss the days when there were several people working on Sandstorm full-time, and want to find ways to support continued development of new features and additional app packaging efforts.
If you want to help us out here’s how you can:
By Ian Denhardt - 08 Aug 2020
This past week I updated the Sandstorm package for Tiny Tiny RSS. In addition to updating to the latest upstream code, I’ve also made some improvements to the package’s integration with Sandstorm. In particular:
The latter of these is critical to enable Sandstorm’s goal of full network isolation for applications. It has always been the intent that apps running on a Sandstorm server should not be able to phone home or otherwise access the network without the user’s consent, but early on the team implemented some temporary hacks that apps could make use of until Sandstorm’s Powerbox API was actually available. Now it is, so it’s time to migrate existing code away from these legacy APIs. Once we’ve done that, we can remove the holes, and finally deliver on Sandstorm’s security promise of network isolation for server code.
In order to make this happen, I wrote a daemon that runs inside of the grain, intercepting HTTP traffic and making powerbox requests for access to the relevant hosts on-demand. The daemon is re-usable and could be used as a quick way to port other existing apps that have legitimate need to access a variety of different hosts at runtime, which are not known in advance.
Unfortunately, this transition does come with some growing pains, and there are a few things users should be aware of when upgrading.
First, since existing TTRSS grains will already have subscriptions to feeds that they haven’t requested access to, the first time you start the new version of TTRSS you’ll see a flood of requests for all of your existing feeds. This is annoying, but fortunately after this initial setup you’ll only need to grant access when adding new feeds.
Second, a more serious limitation of the current implementation is that it makes it impossible to add new feeds through mobile & desktop clients – at least without also having a browser window open somewhere. This is because, without the Sandstorm UI open, Sandstorm has no way of actually showing a user the powerbox dialog. We’d like to find ways to improve on this.
Finally, while the daemon is exciting for app packagers, in that it can make basic porting of apps not designed for Sandstorm much quicker, the integration is imperfect, resulting in a sub-par UX: as it stands, users will see an extra prompt (or two) when first subscribing to a feed.
Maintaining security while avoiding annoying extra prompts like this is one of the design goals of Sandstorm’s powerbox, but to fully take advantage of it will require more invasive changes to TTRSS; rather than prompting the user for a feed URL and then trying to fetch it (causing the daemon to request access), it should just ask Sandstorm for a feed directly. If you’re an enterprising PHP hacker and want to help make this happen, get in touch on the sandstorm-dev mailing list or the #sandstorm IRC channel on libera.chat.
By Kenton Varda - 13 Jun 2020
Sandstorm now has built-in support for fetching certificates from Let’s Encrypt. This applies both to Sandcats and to custom domains.
Sandcats.io is Sandstorm’s free dynamic DNS and TLS (aka SSL) certificate service. Since 2015, Sandcats has been making it easier to set up self-hosted Sandstorm servers, by letting anyone claim a subdomain of sandcats.io
on which to host their server, automatically configuring DNS and HTTPS.
Sandstorm requires a wildcard host so that it can create sandboxes by assigning random hostnames. So, the certificates issued by Sandcats.io have always been wildcard certs. Back in 2015, that was a challenge: at the time, Let’s Encrypt did not yet offer wildcard certs, so the only way to get one was to pay for it. Unfortunately, many CAs considered wildcards to be an “enterprise feature” and charged excessive amounts of money for them, often hundreds or even thousands of dollars.
So how were we able to offer wildcard certificates for free? Well, we negotiated a deal with GlobalSign. Recognizing that subdomains of Sandcats.io were unlikely to be enterprise customers, and recognizing a growth opportunity if Sandstorm took off, GlobalSign offered us a deal that worked. Sandstorm was a funded startup at the time, and we were happy to pay a few bucks per certificate-year and call it “customer acquisition cost”. We paid for thousands of certificate-years upfront, anticipating growth.
But, the growth we hoped for didn’t happen, and we only ever had about 500 self-hosted servers using Sandcats. In 2017, Sandstorm failed as a startup and reverted to being just an open source project. Having never seen the growth we were hoping for, we hadn’t even come close to using up the first block of certificates we had paid for from GlobalSign. Since we had already paid, we left the system running, happily issuing certificates.
Fast forward to 2020. Finally, our contract is running out. In fact, it did run out, in early March – embarrassingly, I had miscounted how much time we had left. However, GlobalSign was nice enough to give us a small extension, giving us time to migrate our users without disruption. And it turns out that, these days, Let’s Encrypt supports wildcards. So, moving to them is the obvious choice.
Starting a few days ago, all Sandstorm servers using Sandcats.io are in the process of switching to Let’s Encrypt for future certificates. The process is designed to happen slowly so that we can address any problems that arise, but so far everything has been smooth. All servers should be transitioned by the end of the month.
If you’d like your server to start using Let’s Encrypt immediately, visit the TLS certificates admin page at /admin/certificates
on your server, click the button to create an ACME account (you will be prompted to agree to Let’s Encrypt’s Terms of Service), and then click “Fetch Certificate Now”.
Note that at present, we have not yet updated the install flow, so newly-installed Sandstorm servers will still use a GlobalSign certificate initially and then will switch to Let’s Encrypt later. We’ll be updating the installer soon, and then we will decommission the GlobalSign flow.
To implement Let’s Encrypt support for Sandcats.io, it made sense for your self-hosted Sandstorm server to directly talk to Let’s Encrypt via the ACME protocol. This differs from the old GlobalSign flow, in which Sandstorm would talk to a central Sandcats.io server, which then talked to the GlobalSign API on your behalf. This was necessary for GlobalSign since Sandstorm the company was paying for the certificates, so our credentials were needed when talking to the API. But for Let’s Encrypt, the Sandcats.io central server only needs to set some DNS records to pass an ACME DNS-01 challenge; everything else happens on your own server.
Given this implementation, it was straightforward to support domains other than sandcats.io. I chose to use the ACME.js library to implement the ACME protocol, and it turns out this library already had a suite of plugins to support a variety of popular DNS providers. If your domain uses any of the DNS providers supported by ACME.js, then Sandstorm can now automatically obtain TLS certificates for your domain from Let’s Encrypt.
However, at present, initial setup is difficult, because of the chicken-and-egg problem: How do you access the admin UI to configure certificates, without a certificate? And than brings me to…
Currently, there are two major problems with setting up Sandstorm TLS on your own domain:
If you’d like to help implement either of these, click on the issue links above and comment!
By Jacob Weisz - 22 Feb 2020
Hello! I’m Jacob Weisz, a member of the Sandstorm community, a long-time contributor, and the new maintainer of the vagrant-spk tool. I’m thrilled to announce the 1.0 release of vagrant-spk.
vagrant-spk is the premier tool for packaging Sandstorm apps. Unlike the spk tool built into Sandstorm, vagrant-spk creates a virtual environment within which to build your app. This provides a reasonable measure of reproducibility and maintainability, along with default templates (or “stacks”) of common configurations apps likely need to run.
upgradevm
feature has been added to make it easy to move to the latest Vagrantfile and global-setup file supported by vagrant-spk. Along with this we’ve improved detection for broken configurations.upgradevm
to switch your package to this.Historically, vagrant-spk was frequently released alongside Sandstorm releases and was regularly dependent on those Sandstorm releases for functionality. At the time, the convention was to release vagrant-spk with versioning complimentary to Sandstorm releases.
However, times have changed. vagrant-spk improvements are rarely strongly correlated with Sandstorm releases, and of course, both Sandstorm and vagrant-spk releases are less frequent in nature.
vagrant-spk is also now a mature tool, having been used to package a large portion of Sandstorm apps. The natural progression at this time is to move from 0.236 to 1.0. This release is fairly small because the priority for 1.0 is to deliver a polished, stable release that we can then iterate upon.
We have a list of improvements and features we are beginning to think about for vagrant-spk’s future. As part of renewed community efforts to revive the Sandstorm project, we are planning a regular cadence of releases for vagrant-spk. We have already begun designating goals for vagrant-spk 1.1, which we’d like to deliver later this year.