This site is historical, from when Sandstorm was a startup. Sandstorm the open source project has moved to sandstorm.org. More info »

Sandstorm Blog

Sandstorm is a self-hostable web productivity suite. Host your own

Recommended Posts

Further Reading

Decentralization is about diversity

By Kenton Varda - 17 Aug 2016

Lately, there has been a flurry of activity around decentralizing the web. Summits have been held. New projects – and companies – have been started. Having worked on this issue for several years now, we’re excited to see it becoming increasingly mainstream.

But what, exactly, are we trying to solve? Most people think it has something to do with privacy, and maybe also security. Some argue that data ownership and mobility are the most important things. Sometimes this leads decentralization projects to focus on data storage while neglecting compute. Some projects even propose that so long as storage is decentralized, it doesn’t matter where the software actually runs.

Privacy, security, ownership, and mobility are all important, but I feel there is a much more important goal that is often poorly understood:

The most important reason to decentralize is software—and developer—diversity.

By “diversity”, I mean this: Who can develop software that other people can plausibly use? Is it primarily mega-corps like Apple and Google? VC-backed startups? Or can one random person, working in their spare time, build just the right app and reach millions of people? Can a community of unfunded volunteers build an open source app that wins because it is the best? Can an employee of one company, having built a useful internal app to solve a problem they had, give (or sell) that app to another company in the same position, without a lot of hassle? Can a teenage fan of a particular video game build an app that assists players of the game, and then share that app with the rest of the game’s community?

All of the above happens regularly with mobile and desktop apps, but it is far more difficult on the web.

We live in a time when our tools are so good that a single competent application developer working weekends can create almost any web application you can imagine. However, rarely can that single developer also run a secure, scalable service and a business around it. As a result, when software is delivered as a service, the only software that is available is that which was deemed a priori to be sufficiently lucrative to interest a mega-corp like Google or a VC-backed startup. Therefore, the only software services we get come courtesy of these gatekeepers. Experimental, indie, or amateur projects are rare. Novel services serving a small niche community are rare. Services designed by people who aren’t well-represented in the tech industry are rare. Community-driven open source projects basically aren’t viable.

The only way to solve these problems is by decentralizing the software (not just the storage). Software must be provided as a package – not as a service – with each user running their own private copy. It doesn’t really matter if the user chooses to deploy to “the cloud” or to their own machine, as long as they can run any package they want. It is, however, important that the means to deploy software be accessible even to non-technical users, so that everyone can participate and developers can reach a wide audience. Deploying an app on your server must therefore be as easy as installing an app on your phone, and must be “secure by default”.

This is the focus of Sandstorm.

Share many as one, with the Collections app

By David Renshaw - 09 Aug 2016

Starting work on a project in Sandstorm often means creating many grains, each pertaining to a different aspect of your work. In a typical project, you might have a spreadsheet, a chat room, a kanban board, and several source code repositories. Sandstorm makes it easy to share these grains with your collaborators, but until recently, you would need to share each grain to each collaborator separately — a task that could quickly become tedious. What if you want to share the entire project as a single unit?

Now that we have released the Collections app, we have a satisfying answer to that question; to share many grains at once, you add them to a collection.

A collection is a list of grains. Any collaborator with whom you share a collection gets access to all of the grains in it. On the flip side, when you remove a grain from the collection, your collaborators lose access to that grain. Moreover, when you revoke a collaborator from the collection, that (now former-) collaborator loses access to all of the collection’s grains.

And since a collection is itself a grain, sharing one works just like sharing anything else on Sandstorm, through the “Share access” button.

Powerbox makes it possible

The fact that we have implemented collections as an app may come as a surprise, since the notion of a collection might seem fundamental enough to deserve being baked-in as a core feature of Sandstorm. However, Sandstorm bakes in an even more fundamental notion: the idea that grains can refer to and coordinate with one another. The embodiment of this idea is the powerbox, an interface — mediated and auditable by the user — through which grains can exchange capabilities.

When you click the “Add grain” button in a collection, the collection in fact initiates a powerbox request. Sandstorm then asks you, the user, to choose a grain with which to fulfill the request.

Once you select a grain, the collection receives a reference to that grain. It can then use that reference to retrieve metadata such as the grain’s icon and the the name of the app that created the grain. Crucially, it can also offer the reference to your collaborators, so that your single “Add grain” action can result in all of your collaborators receiving access to the grain. The behind-the-scenes details of how this all works can be found in our technical documentation.

Other apps can use the powerbox in the same way. You could write an alternative implementation of collections, either by starting from scratch or by forking ours. Apps whose primary purpose has nothing to do with collections can also benefit from the ability to request and offer grain references. For example, a chat room could be enhanced by the ability to embed Sandstorm-aware links to other grains. Indeed, the latest release of Rocket.Chat does just that.

Built with Rust

Although a fast startup time and a small memory footprint are important performance goals for any Sandstorm app, they are especially important for the Collections app, as it provides such a central piece of functionality. Collections need to be lightweight so that their integration with the rest of Sandstorm can feel seamless. Our primary strategy for achieving such performance has been to develop the Collections app using the Rust programming language, interfacing directly with Sandstorm’s Cap’n Proto interfaces . Rust has worked well so far, and along the way we have produced some libraries and examples to help others also get started using it for Sandstorm app development.

Whether or not you are a developer, now is an exciting time to get involved with Sandstorm. The powerbox is still in its early stages, and the Collections app is a hint at the kinds of things it will enable. So try it out and let us know what you think!

July changelog - what's new in Sandstorm

By Asheesh Laroia - 03 Aug 2016

In July, we introduced a new Sandstorm feature: Apps can notify users when something interesting happens within a grain. For example, Etherpad uses this to tell you when someone leaves a comment on a document. This ActivityEvent API was designed by Kenton Varda and its visual elements were designed by Néna Nguyễn. You can read more in its Cap’n Proto definition; any app within Sandstorm can use it. The easiest way to try it out is to make a new Etherpad document, then open it in an incognito window, and use the incognito window to leave yourself a comment.

This change, and many more, went live during July. To keep your Sandstorm server updated, you can sit back and relax. In the default configuration, Sandstorm’s automatic updates will install updates within 24 hours of a new release.

Here’s the full July changelog!

v0.173 (2016-07-23)

v0.172 (2016-07-15) [bugfixes]

v0.171 (2016-07-09)

v0.170 (2016-07-02) [bugfixes]

June changelog - what's new in Sandstorm

By Asheesh Laroia - 13 Jul 2016

June’s big change is that we’ve reworked what first-time users see when using Sandstorm. There are speech bubbles guiding them to install an app, create a grain, and to take a look at the “Share access” interface, all powered by intro.js. I’m hopeful this helps people become productive and comfortable with Sandstorm faster than before.

It applies to all users, including demo users, so test it out by opening demo.sandstorm.io in an incognito window and click Try a quick demo.

Some readers will appreciate that this month, we also chased down a subtle memory leak caused by a lack of flow control in the HTTP implementation spread across proxy.js and sandstorm-http-bridge. You’ll find the details in pull request #2056.

Read the full changelog here, or visit your own Sandstorm server’s “About Sandstorm” section for a changelog at any time.

v0.169 (2016-06-26) [bugfixes]

v0.168 (2016-06-24) [bugfixes]

v0.167 (2016-06-18) [bugfixes]

v0.166 (2016-06-11) [bugfixes]

v0.165 (2016-06-04) [bugfixes]

Upcoming Event: Web app packaging in Sandstorm: It's not LAMP

By Nena Nguyen - 29 Jun 2016

On July 19th, Sandstorm core dev Asheesh Larioa will be giving a talk about web packaging at our South Bay Sandstorm meetup.

Talk details

This talk covers how web app packaging works for Sandstorm.io. Asheesh will compare and contrast Sandstorm packaging against the typical install process on a Linux/Apache/MySQL/PHP system. This talk was featured at Debconf16, the yearly Debian conference.

He will outline how Sandstorm’s packaging tools do a few strange things to allow unprivileged users to install apps with one click:

• Every app package is a tiny Debian derivative, often as small as 20MB.

• Apps have no Internet connectivity to the outside world.

• Sandstorm uses a FUSE filesystem to identify which files are needed to run the app.

• An app bundles all its needed services, as well as files, resulting in one MySQL service per app.

• Users click and run one instance of an app like Etherpad per document, which is all handled transparently via a web app, a strategy that has neutralized 95% of 0-day web app vulnerabilities, based on our analysis.

• Developers on Mac OS and Windows can create packages for Sandstorm, even though Sandstorm is Linux-only, due to an emphasis on Linux VMs in our development tools.

Somehow we manage to make this scale reasonably well. Additionally, it is popular with upstream authors: of the >58 web apps packaged for Sandstorm, about 1/3 are maintained by their upstreams.

This talk focuses on how the Sandstorm packaging tools work, with community insights as well as technical ones, with the hopes of showing Debian how to more effectively package web apps for end users.

Come for the opportunity to meet others in the Sandstorm community, and work on your project with guidance from our core devs. This event will be held at RethinkDB in Mountain View on July 19. RSVP here.