Sandstorm.io Blog

We're changing the way identities work

Mon, 08 May 2017 00:00:00 -0700

Over the next few weeks, I’ll be making a major change to the way Sandstorm handles user accounts and identities. My goal is to make things far less confusing.

Most Sandstorm users probably have no idea that these features exist, and so won’t notice the change. However, if you’ve linked multiple “identities” (multiple e-mail, Google, or GitHub accounts) to your Sandstorm account, you may want to read this.

History: The current system and why it was built this way

Way back in late 2015, we introduced the concept of “multiple identities” in Sandstorm. Ever since, a user account on a Sandstorm server has been able to have multiple identities attached to it, each with a different profile (name, picture, etc.). Once a user has multiple identities, they can choose which identity to act as when using apps. When users share with each other, they often do so “by identity”, meaning you choose the identity (not the account) with whom you want to share. It’s even possible for multiple accounts to share a common identity, in order to have multiple people “acting as” the same persona.

When we designed this system, we were attempting to solve several different problems at once:

We had long offered users the ability to log in with Google, GitHub, or e-mail. Sometimes, though, people wanted to attach multiple login mechanisms to the same account, e.g. both their Google and their GitHub account. This redundancy can protect people against one of these services being down, and would help avoid confusion when people can’t remember which login mechanism they used last time. We also anticipated that in the future, Sandstorm apps might want to talk to Google and GitHub APIs, and this access would be authenticated through the user’s connected accounts.
We wanted to ensure that the user ID seen by an app was a global identifier – meaning it would be the same even if the app was moved to a different Sandstorm server. This required that the ID be derived from the user’s credentials, e.g. a hash of their Google or GitHub user ID.
When users share with each other, we wanted them to be able to do so using well-known public identifiers, for additional security. Typically, Sandstorm users share with each other by creating and sending “secret links”; anyone who receives the link can get access. This is convenient, but sometimes you want some additional assurance that a link can’t be leaked. In that case, you might want to specify a specific GitHub username or Google account e-mail address with whom to share, and have Sandstorm guarantee that the receiver must authenticate as that identity to open the link.
We wanted people to be able to manage multiple “personas” with which they interact with other users within Sandstorm. We imagined that users may in fact want to prevent other users from discovering that these personas belonged to the same physical person. We imagined that this would in particular be useful to groups who fear harassment and abuse, and therefore choose to interact under a pseudonym.
We wanted multiple users to be able to collectively manage a shared persona, like a “brand account” on Twitter, while keeping independent Sandstorm accounts.

With all of these goals mashed together, we came up with a design: We would allow you to connect multiple credentials (e.g. Google, GitHub, e-mail, even multiple of each) to your account. Each connected credential became an “identity”, with its own profile (name, picture, and other details shown to users you collaborate with). Each identity could also be marked as being for authentication (in which case you could use it to log into your account) or not (in which case you couldn’t use it to log in, but you could use it as a persona and a sharing target). Non-login identities could also be shared between multiple accounts. Whenever you interacted with a grain, you would choose which identity you wanted to act as, which determined how collaborators saw you.

Why it didn’t work

In retrospect, it was a mistake to try to solve all of these problems at the same time, as they are all fundamentally different. By trying to link them together, we largely failed to solve any of them.

The idea that your Google and GitHub accounts each represented a different “identity” confused many users. Most people in fact only have one persona, and their accounts on various services represent that same persona. But if you attached both to your Sandstorm account, then Sandstorm would begin asking you to choose which identity you wished to act as – a question that made no sense to most people. Moreover, other people, when sharing with you, would begin to see multiple “copies” of you showing up in their contacts list – one for each credential. Often it was hard for people to tell what the difference was or which they should choose. Because of this confusion, the Sandstorm team has tended to steer people away from linking multiple identities in the first place. Obviously, this defeats the goals for which multi-identity was created.

Meanwhile, for people who actually wanted to manage multiple personas – pseudonyms, brand identities, etc. – we never managed to provide a good experience. It was very easy to accidentally use the wrong identity and reveal yourself. At the same time, the need to support this use case often exacerbated the confusion for users who only had one logical persona (as described in the previous paragraph). Without this design constraint, we could make a better UX for most people.

Finally, in practice this design didn’t really solve the ability to move grains between Sandstorm servers, because different Sandstorm servers may very well use totally different login mechanisms. Often, people want to transition grains from Oasis – where they used e-mail login – to a new self-hosted server – where they used LDAP or SAML login. Since the login mechanisms differed, user IDs didn’t match up anyway. To really solve this problem, we need something more dynamic.

What we’re doing instead

We are dropping support for multiple “personas”. Going forward, each Sandstorm account will have only one profile – one name, one profile picture, one entry in the sharing auto-complete list, etc. If you have multiple identities today, the profile from exactly one of those identities will become your account profile, and the other profiles will disappear. If you’re worried about which one Sandstorm will take, make sure to edit all your identities so that their profiles are the same.

What we call “identities” today will be renamed to “credentials”. A credential will no longer have its own profile, and you will no longer need to choose which credential you are acting as when accessing grains. Authentication-related features of credentials (for login, and for secure sharing) will remain mostly intact.

Users who rely on the ability to manage multiple personas today will need to transition to using multiple accounts instead. I suspect that there are vanishingly few such users, as most users never understood Sandstorm’s identity system in the first place. That said, if you are affected, I apologize. I would love it if you would get in contact with us to let us know about your specific needs, so that we can try to design a better experience for you. (For what it’s worth, I personally recommend using the multi-profile feature provided by various browsers to separate your personas into totally separate browser contexts with different window themes – this makes it much easier to prevent accidental leakage.)

Finally, we will be disassociating the user ID as seen by apps from the underlying user credentials. In the future, each grain will actually see a totally unique ID for each user. When a grain is transferred between servers, the owner will have the opportunity to remap users in arbitrary ways – though by default we’ll correlate based on verified e-mail address. This scheme has the additional benefit that because a particular user’s ID will be different in every grain they access, it will no longer be possible for apps to illicitly identify and correlate users by their IDs. Instead, they will have to use explicit APIs for this purpose, which can be carefully controlled for privacy.

Timeline

As of this post, these changes have not yet been implemented. We wanted to let people know of the changes in advance, in case anyone is relying on the current behavior. The next Sandstorm update will include code which detects users who might be affected and notifies them to read this blog post. The full change will probably take several weeks or maybe months to implement, as it is a huge change. We’ll update this post when it’s done.

The Sandstorm Team is joining Cloudflare

Mon, 13 Mar 2017 00:00:00 -0700

Recently, we announced that Sandstorm is returning to its community roots, transitioning away from being a for-profit startup and towards being a community-effort, open source project. The Sandstorm team is committed to keep building Sandstorm in collaboration with our amazing community, and we will keep operating Sandstorm Oasis as a service. But, Sandstorm will no longer be our full-time jobs.

So, what will be?

Starting today, most of the Sandstorm team – including Jade and myself – now work for Cloudflare.

Sandstorm has actually been a user of Cloudflare since the beginning. Our web site and static parts of Oasis are served through them. I’ve long been a fan of Cloudflare’s focus on security. A few years back, I enjoyed their Heartbleed Challenge which proved conclusively that, yes, private keys can be leaked by Heartbleed. Although Cloudflare recently suffered from a widely-reported security incident, their response was impressively fast and transparent.

But a bigger reason I’m excited about joining Cloudflare is that they are big users – perhaps the biggest users – of Cap’n Proto, the serialization and RPC framework developed by Sandstorm. Cloudflare actually developed the Lua bindings for Cap’n Proto and has spoken publicly about using Cap’n Proto in their logging pipeline.

Sandstorm, for its part, remains an independent entity, and I won’t be working on it during my day job at Cloudflare. However, I will be working on Cap’n Proto. This should be good news for Cap’n Proto fans who have been disappointed by the lack of releases lately. Although Sandstorm uses Cap’n Proto extensively, in our scramble to build Sandstorm we tended to commit changes only to git master while neglecting to build (and test) formal releases. With a larger corporate backer, we can now afford to be more professional, doing more frequent releases, and putting effort into supporting more languages and platforms.

I will also be working on building some fascinating new infrastructure at Cloudflare. As a member of the edge platform team, I’ll be coming up with new ways to utilize Cloudflare’s machines located in over a hundred data centers around the world. Jade, meanwhile, will be working on building out Cloudflare’s community and developer advocacy program.

The Sandstorm community can expect to see me merging pull requests and pushing releases on weekends, with new releases at least every three weeks. There has been a flurry of activity lately around mobilizing community contributions. Join the discussion on the sandstorm-dev mailing list or on IRC (#sandstorm on Freenode)!

Sandstorm gets a security review

Thu, 02 Mar 2017 00:00:00 -0800

Sandstorm recently underwent a security review by DevCore Inc., commissioned by Department of Cyber Security of Taiwan. The company has in the past found RCEs in big-name services like Uber and Facebook.

Although the full report is not public (and is not in English), Taiwan’s Digital Minister, Audrey Tang, tells us that overall DevCore feels Sandstorm has “excellent security design.”

As would be expected of any good security review, DevCore did find several issues. Last night we released build 0.203 to fix them. All servers should automatically update within 24 hours of the release (by midnight tonight, PST), but if you’d like to speed up the process (or if you have disabled automatic updates), you can SSH into your server now and type:

sudo sandstorm update

Vulnerabilities Discovered (and fixed)

1. Insufficient e-mail address validation

When logging in with an e-mail address, a user could enter an address like “foo@evil.com,bar@example.com”. In this case, login e-mails would be sent to both addresses, while Sandstorm would treat the user as if they were a member of the latter domain, “example.com”. Hence, an attacker could specify their real address as the first address and a fake address at a domain of their choice as the second, in order to appear to be a member of the domain. Servers that use Sandstorm’s organizational features (until recently only available as part of the now-defunct product “Sandstorm for Work”) can be configured to automatically promote members of a domain to full users, allowing them to install apps, create grains, and discover the identities of other members of the organization. Such users would not receive access to any other grains on the server, but an attacker could install their own apps that consume the server’s resources, including CPU, RAM, and disk. An attacker could also combine this vulnerability with one of the others below, all of which require full user status as a starting point.

The root cause of this bug is Sandstorm’s use of a library called Nodemailer to send e-mail. It turns out that Nodemailer “helpfully” interprets comma-separated lists of addresses as multiple addresses. This is true even if the string is already part of an array. For instance, say you pass the following array in the to field of a mail sent via Nodemailer:

[ "foo@example.com",
  "bar@example.com,baz@example.com",
  { name: "Garply", address: "qux@example.com,corge@example.com" } ]

Surprisingly, this three-element array will send e-mail to five addresses – foo@, bar@, baz@, qux@, and corge@.

We did not anticipate this feature and assumed that Nodemailer would fail to send e-mail to an invalid address.

Severity: Moderate if Sandstorm is configured with an organization defined by e-mail domain. If you do not define an organization, or if you define it by LDAP, SAML, or G Suite domain, then we do not believe this bug can be meaningfully exploited.

Fix: Sandstorm now performs its own validation of addresses to ensure, among other things, that no separator characters are accepted and only one @-sign is allowed. Fixed in commit 37bd9a7.

2. Insufficient path validation when building ZIP backups

The owner of a Sandstorm grain can download a backup of the grain’s content in ZIP format. Sandstorm shells out to the zip utility to build this archive, but does so inside a sandbox designed to prevent exploitation of bugs in the utility.

Unfortunately, a combination of factors resulted in the possibility of a minor data leak:

The list of files to include in the archive is passed to zip on standard input, delimited by newlines. However, it is possible for a filename to contain a newline character. Sandstorm actually checked for this, but accidentally failed to perform the check in one code path triggered by the existence of an empty directory.
The zip sandbox carefully hid sensitive directories like /var, /proc, and even /etc. However, after the zip sandbox was written, two new directories, /etc.host and /run.host, which are aliases for the host system’s /etc and /run, were added. Unfortunately, the zip sandbox was not updated to hide these directories, hence they were visible to the zip program.

As a result of this, a user could create a grain containing an empty directory named /var/blah\n/etc.host/passwd, save a backup, and find the backup contained a copy of the host’s /etc/passwd file (which, despite its name, does not contain passwords, but does contain a list of local user accounts).

Severity: Low. An attacker who has full user status (permission to create grains) can read files located in the server’s /etc and /run directories that are set world-readable (or specifically readable to the Sandstorm service account). No other host system directories are exposed. Usually, any files under /etc that contain sensitive secrets would be hidden from all users except root, hence would not be readable by Sandstorm nor by the attacker.

Fix: We have fixed the validation of filenames so that newlines are never permitted, and we have enhanced the zip sandbox to use a whitelist of top-level directories rather than a blacklist. Fixed in commits 4ea8df7 and 6e8572e.

3. Server-side request forgery

Apps running on Sandstorm can make outgoing HTTP requests through a few mechanisms. Additionally, users can instruct Sandstorm to download an app package file from an arbitrary URL. Sandstorm did not attempt to prevent these requests from reaching localhost or private network destinations. Because of this, a full user of the Sandstorm server (but not a visitor) could probe the internal network on which the Sandstorm server was hosted, as well as services exposed to localhost on the server itself.

This is a problem if unprotected services are exposed to the Sandstorm server and you have invited users to your server who should not be trusted to access those services. Sandstorm itself does not expose any unprotected services to localhost, so some other service must be running locally to cause a problem. Only HTTP requests are possible.

Severity: High if unprotected HTTP services are visible to the Sandstorm server and you have invited users to your server who should not be trusted to access those services. Otherwise, low. Sandstorm itself does not expose any unprotected services to localhost, so some other service must be running locally to cause a problem. Only HTTP requests are possible.

Fix: Sandstorm now allows the server admin to configure an IP address blacklist. User-initiated requests will not be allowed from these addresses. By default, standard private network address ranges are blacklisted. Annoyingly, this “fix” is likely to break some servers who use private app indexes, but avoiding such breakage would mean leaving others vulnerable. Admins will need to update their blacklists accordingly. Fixed in commit 164997f.

4. Failure to mitigate Linux kernel CVE-2017-6074

(This was caught by Sandstorm core developer David Renshaw, not by DevCore, but is fixed in the same release.)

A recent Linux kernel bug could allow a malicious app to cause a kernel panic or perhaps break out of its sandbox. The bug is in Linux, but Sandstorm aims to protect against such bugs via seccomp. Historically, Sandstorm’s sandbox has been very successful at blocking Linux vulnerabilities before they are discovered. However, this one slipped through. On Sandstorm, the published PoC exploit for this bug only causes a kernel panic (not a sandbox breakout), but a developer with deep understanding of the bug and the Linux kernel might be able to create a successful attack in other ways.

Severity: High if you permit users you don’t trust to install apps on your server, or commonly install apps from untrustworthy sources, and you do not keep your kernel up-to-date.

Fix: Sandstorm’s seccomp filter has been updated to prohibit the creation of DCCP sockets. Fixed in commit 34749f9.

Conclusion

We are excited to have received such a thorough security review. Some of the problems discovered were very obscure and we’re impressed that DevCore was able to find them. We thank Audrey Tang and the Department of Cyber Security of Taiwan for commissioning this review.

Impact of Cloudflare security advisory on Sandstorm Oasis

Tue, 28 Feb 2017 00:00:00 -0800

A few days ago, Cloudflare disclosed a security bug in their services which may have leaked secrets from sites which use Cloudflare. Sandstorm uses Cloudflare as a cache in front of our web site and Sandstorm Oasis, and so some of our users have asked if this problem affects them.

TL;DR: If you uploaded or downloaded a grain backup (zip file) between September 22, 2016 and February 18, 2017 (more so between Feb 13 to Feb 18), there is a very remote chance that some portion of the content of your backup could have been leaked to a third party, due to a bug in Cloudflare. We suspect, but cannot say for sure, that no such leakage occurred. We believe that no data other than grain backups could have leaked.

Details

We believe that Sandstorm services are mostly unaffected (with the exception of grain backups, described below). We intentionally divide our services such that public, static content is served from different hosts from dynamic/private content. We only enable Cloudflare in front of the public, static content (to provide caching and faster delivery). Moreover, the public, static hosts avoid storing sensitive credentials in cookies.

Specifically, we have Cloudflare enabled on the following hosts:

sandstorm.io, which serves our public web site. There are no secrets here.
docs.sandstorm.io, which serves our public documentation. There are no secrets here.
dl.sandstorm.io, from which Sandstorm install packages and updates are served. There are no secrets here.
oasis.sandstorm.io, which serves the static HTML, Javascript, CSS, and images for Sandstorm Oasis. Dynamic content in the Sandstorm UI is delivered – and user actions are communicated – via a WebSocket on ddp.oasis.sandstorm.io, which does not use Cloudflare. The user’s authentication token is stored in localStorage rather than a cookie and is only ever transmitted over the WebSocket. Sandstorm apps are served from randomly-generated subdomains of oasis.sandstorm.io, which are not served through Cloudflare. However, grain backup uploads and downloads occur on the main host and could be affected – see below.
app-index.sandstorm.io, which serves public Sandstorm app packages and metadata for use by the app market and for implementing automatic updates. There are no secrets here.
apps.sandstorm.io, which serves the static HTML, Javascript, CSS, and images for our app market. As with Oasis, dynamic content and interaction (mainly, reading/posting reviews) happens via a WebSocket on apps-ddp.sandstorm.io and credentials are stored in localStorage.
Various internal testing hosts and hosts that only redirect to other hosts.

We chose this design because we rely on Cloudflare only for caching purposes. Since dynamic data cannot be cached anyway, it made sense to form direct connections for that data rather than pass through a proxy, both to reduce our TCB and to improve performance. On the other hand, this means we cannot fully take advantage of Cloudflare’s security features such as its Web Application Firewall and DDoS protection, but we felt comfortable with this trade-off.

That said, we’ve identified one notable exception to our design: When you download a backup of a grain via the Sandstorm Oasis UI, or upload a backup to be restored, the zip file is transmitted through the main Oasis host, proxied through Cloudflare. This seems to be an oversight; it would make sense for these requests to go through the DDP host (I’ve prepared a pull request to fix this). These transfers are authorized via short-lived tokens, so there’s no risk of Oasis credentials being leaked. However, the contents of a .zip file could possibly have been leaked by Cloudflare.

Cloudflare reports that the bug existed between 2016-09-22 and 2017-02-18, with the greatest potential impact being in the last four days of that period. If you uploaded or downloaded a grain backup in that time, it could have been leaked. However, Cloudflare reports that the incidence of such leaks was extremely small, and each instance would have leaked data from an essentially random website served by Cloudflare, of which there are many (Cloudflare handles some 10% of all internet traffic). Given the probabilities, it is likely that no Sandstorm grain backup data was actually leaked. There is also so far no evidence that anyone was actively exploiting this bug.

With all that said, if you are worried about the possibility that your data was leaked, the proper course of action depends on the app and grain involved. Sandstorm is explicitly designed to keep credentials out of the hands of apps, so usually the only sensitive information in a grain backup is information that you gave to it yourself. For example, if you had an Etherpad document where you kept a list of passwords (and you downloaded a backup of that document recently), you may want to change those passwords. Otherwise, there is nothing to do.

Editorial

Bugs like this probably exist in a lot of software. The Cloudflare bug is notable because:

It was discovered. (Most bugs aren’t.)
Cloudflare handles some 10% of all internet traffic, meaning that basically everyone on the internet is possibly affected.

It’s important to keep in mind that security is not binary. Security is about risk management. The Cloudflare bug did not change anyone’s security level from “secure” to “insecure” – instead, it increased the likelihood of any particular person becoming the victim of a real attack from, say, 5.26% to 5.28% (numbers made up to illustrate point). Had Cloudflare been slow to respond once the vulnerability was found, this risk may have shot up. Fortunately, though, they were anything but.

Now, consider the use of password-based authentication. Nearly every service on the web uses it. However, it is demonstrably horribly insecure: Not only do people regularly use bad passwords or reuse the same password across sites, but passwords stored in a database (even if hashed) make it much easier for attackers to escalate a read-only database snapshot into full access privileges (possibly even to sites other than the one attacked). By requiring the user to receive an authentication code by e-mail each time they log in from a new location, a service can hugely reduce their users’ risk of compromise – probably by an order of magnitude! By this measure, bare password-based authentication should be considered a vulnerability, and a much more serious one than Cloudflare’s bug.

And yet, people accept passwords because they are more convenient than other mechanisms. (Well, many people do. Sandstorm does not implement passwords.) So, we admit that security is not absolute, but rather a trade-off which needs to be judged against other factors. We need to weigh the risks of our infrastructure being insecure against the value that infrastructure brings – and Cloudflare certainly brings a lot of value. That said, with good design, it’s often possible to reduce and contain risks without sacrificing too much utility. This is one of the core goals of Sandstorm – to introduce an infrastructure model and security practices which mitigate risks.

Sandstorm is returning to its community roots

Mon, 06 Feb 2017 00:00:00 -0800

Most people know Sandstorm as an open source, community-driven project aiming to enable self-hosting of cloud services and to make it possible for open source web apps to compete with today’s cloud services.

Many people also know that Sandstorm is a for-profit startup, with a business model centered on charging for enterprise-oriented features, such as LDAP and SAML single-sign-on integration, organizational access control policies, and the like. This product was called “Sandstorm for Work”; it was still open source, but official builds hid the features behind a paywall. Additionally, we planned eventually to release a scalable version of Sandstorm for big enterprise users, based on the same tech that powers Sandstorm Oasis, our managed hosting service.

As an open source project, Sandstorm has been successful: We have a thriving community of contributors, many developers building and packaging apps, and thousands of self-hosted servers running in the wild. This will continue.

However, our business has not succeeded. To date, almost no one has purchased Sandstorm for Work, despite hundreds of trials and lots of interest expressed. Only a tiny fraction of Sandstorm Oasis users choose to pay for the service – enough to cover costs, but not much more.

We attribute this failure to two main problems:

As a product, Sandstorm is still alpha-quality. As you may know, Sandstorm is pioneering a paradigm shift in the way servers operate, which we call fine-grained containerization. The benefits of this technology are huge, but a lot of engineering work is needed to make it work smoothly. Although we’ve built a working product that many people – including ourselves – use for real work every day, many limitations still exist that make Sandstorm feel rough and incomplete compared to alternatives. We know how to fix these problems, but we need more time to do so.
We underestimated, in classic fashion, the challenge of enterprise sales. As a company of engineers and geeks, we knew that enterprise sales would be outside of our comfort zone, but we didn’t realize just how much work it would be to learn, and how much time we needed to be successful at it. By the end, we were only scratching the surface of how to generate leads and get in the door at big companies; we never managed to do a “real” sales call. In retrospect, we should have hired a business development expert much earlier on. We also should have raised more money and planned for a longer runway.

We plan to publish a more complete postmortem in a subsequent blog post.

Unfortunately, Sandstorm the business has now run out of money, and we have been unable to raise more.

The Project Lives On

Although it will no longer be our full-time job, Sandstorm will continue as an open source project. We still strongly believe in Sandstorm’s long-term vision and cannot abandon it. I personally will continue to lead Sandstorm’s technical development: reviewing and merging pull requests, pushing releases, and developing new features. We will continue to operate Sandstorm Oasis – your data there is safe. Meanwhile, we will make it easier for our extended community to be involved in core development and decision-making. Jade will be in contact with individual community members to appoint community leaders and grant them the authority to handle a variety of community organizing functions, from App Market approval to organizing meetups.

Ironically, the pace of development may not even be hurt much. Over the past year, the Sandstorm team has spent a great deal of our time on enabling our business, e.g. building a payment mechanism, processing customers, marketing, and the like. I personally have spent far too much time on fundraising, sales, and other deal-making rather than on coding. With this shift in direction, we can now focus strictly on building out the core platform, getting more done with less time.

Immediate Action Items

As a result of this change, the following has happened today:

We have pushed a release which removes the Sandstorm for Work paywall. Sandstorm for Work features (like LDAP/SAML integration) are now available to all self-hosted Sandstorm servers at no cost. The product name “Sandstorm for Work” has been retired; there is now only Sandstorm. (Your server should already have received the update. If not, SSH in and type sudo sandstorm update. Don’t have a server yet? Install Sandstorm now.)
We have open sourced Blackrock, an alternate back-end for Sandstorm which is able to scale out across a cluster of machines. We have long used this technology to power Sandstorm Oasis, but had hesitated to release it for fear that it would harm our business model. We no longer have a business model to protect, so the code can now be set free.
We have added the Sandstorm Technology Roadmap to the Sandstorm repo, where you can learn about everything Sandstorm has built and plans to build. The roadmap is full of projects and features marked “TODO”, so you can see what needs to be built.
We’ve written a new contributing guide with a list of projects you can work on. See if there’s anything that interests you!
No changes are expected to Sandstorm Oasis nor Sandcats.io. Both services will continue to operate going forward.

How you can help

Want to see Sandstorm succeed? Then, contribute!

If you’d like to help build Sandstorm, sign up on the sandstorm-dev mailing list and join us on IRC at #sandstorm on Freenode. There is lots to do, from hardcore systems hacking to community organizing to tasks that require nothing but enthusiasm. For ideas on how you can help, check out the community page and the contributing guide.
If you have more money than time and would like to support the project financially, the best way to do it is to sign up for a paid account on Oasis.

Sandstorm Solutions: pay for feature prioritization

Thu, 01 Dec 2016 00:00:00 -0800

Is there a feature or bug holding you back from deploying Sandstorm at your company or for your team? Do you need the Sandstorm core devs to prioritize a feature? Sandstorm Solutions can fix that.

There’s always way more on our roadmap than time and people to do it. We receive requests from potential customers around the world who would love to use Sandstorm, but need a particular planned but unimplemented feature, or hit a bug that affects them in particular. We now offer Sandstorm feature prioritization to our customers.

We’re happy to adjust our priorities to support these folks, so we figure we should state that clearly and make sure it’s something everyone can take advantage of, not just the people who inquire.

So here’s the deal with Sandstorm Solutions: if you want a Sandstorm feature prioritized, or a particular bug fixed, we can do it. The way this works is:

You email us at solutions@sandstorm.io with an explanation of what you need.
We’ll make sure it fits into our roadmap, and if so, we’ll give you an preliminary estimate of how long we think it’ll take to implement, and how much it’ll cost.
We own the copyright on our work.
If everything sounds good, we’ll draw up a contract and get to work!

Get an estimate »

Need something different? We’re happy to talk about how we can help you succeed - drop us a line at solutions@sandstorm.io.

Sandstorm Oasis emerging from beta

Thu, 17 Nov 2016 00:00:00 -0800

Sandstorm is about making it easy to run a personal server. But we also offer Sandstorm Oasis, a service which runs your Sandstorm server for you.

Contradiction?

Actually, no: Even if you run your own server, Oasis benefits you. Oasis is important because it makes it possible for anyone to use Sandstorm’s library of open source apps, even if they really don’t want to run their own server. A larger audience means that more and better apps will become available. Indeed, after we launched Oasis last year, the rate of new apps becoming available on Sandstorm spiked.

That benefits self-hosters, because those same apps can be used on your private server, too.

In fact, we at Sandstorm don’t necessarily think “the cloud” is a bad idea. What we believe is that you should have the freedom to choose what makes sense for you. But that choice is moot if the particular app you need to use is only available in the cloud – we need the same apps to be available everywhere.

Oasis has now been running reliably for over a year. The Sandstorm team uses Oasis every day to get our own work done. I am composing this blog post in Etherpad, while organizing my task list in Wekan, chatting with teammates in Rocket.Chat, and syncing files with Davros.

Here are just some of the things we’ve changed since Oasis was launched:

Implemented automatic updates for apps.
Added cryptographic verification of app author identity.
Added unified notifications and activity indicators, so you can find out when apps need attention without opening them to find out.
Added collections, allowing you to share a group of grains as one unit.
Added the ability to share grains through other apps, e.g. sharing a document to a Rocket.Chat channel, without generating a magic link.
Added a “trash” mechanism to make it harder to accidentally delete an important grain.
Added the ability to delete your own account.
Added support for apps to export WebDAV APIs.
Made it easier for Sandstorm apps to connect with mobile client apps.
Made Sandstorm’s UI mobile-friendly.
Made it so that our most-used apps are pre-installed for new users.
Made sharing easier and more secure by adding contact auto-complete and “request access” UIs.
Added the ability to attach multiple credentials to one account, so that one login provider being unresponsive won’t prevent you from accessing Sandstorm.
Made the left sidebar collapseable for more screen space.
With the help of our developer community, doubled the number of apps available on Sandstorm.
Improved onboarding for new users with tutorial hints.
Performance, scalability, and stability improvements.
Much more.

We’ve so far kept Oasis labeled “beta”, mostly because, as engineers, we always feel like there’s so much more to do. But, that will always be true – no good software project is ever “done”. With Oasis being used for so much real work, the time has come to remove the “beta” label.

Oasis will officially emerge from beta on November 27. We wanted to give advance notice of this change because it affects our paying users: we will no longer be waiving your subscription fee as we have during the beta period. For backers of our Indiegogo campaign who opted for free hosting as a perk, the timer on your service will start now (hey, you got an extra free year!). For the rest, your next monthly invoice will be charged from your credit card. Your subscription payments help support further development of Sandstorm and packaging more apps. Thank you for your support!

Demo Oasis now »
(no sign-in required)

Sandstorm now supports RHEL 7, CentOS 7, Arch, and more

Thu, 10 Nov 2016 00:00:00 -0800

As of a couple weeks ago – October 23, 2016 – Sandstorm can now be installed on systems with:

Linux Kernel 3.10+ (previously required 3.13+)
User namespaces disabled (previously required unprivileged user namespaces)

This means that Sandstorm can now be installed on Red Hat Enterprise Linux (RHEL) 7, as well as its cousin CentOS 7, both of which use kernel version 3.10.

It also means that Sandstorm can now be installed on Arch Linux, which has historically shipped kernels compiled with CONFIG_USER_NS=n.

So if you previously couldn’t install Sandstorm because you were using one of these distros, now you can!

Install Sandstorm now »

What changed?

For the technically curious…

Sandstorm uses the Linux kernel’s “namespaces” feature as part of setting up the secure sandboxes in which apps run. Normally, creating namespaces requires root privileges, because these features could be used to escalate privileges. However, using “user namespaces”, a process that does not have root privileges can create a special kind of namespace in which other namespaces are (ostensibly) safe to use. Hence, it allows unprivileged processes to create sandboxes.

For security reasons, most of Sandstorm does not run with root privileges. Because of this, it has long relied on user namespaces to allow it to set up sandboxes. At the time the Sandstorm project started, it looked like user namespaces would soon be broadly available across Linux distros, so this seemed like a reasonable strategy.

Unfortunately, this has not been entirely true in practice. The enterprise-oriented RHEL and CentOS distros have long release cycles. Today, they still use kernel version 3.10, which is nearly three years old. Because user namespaces still had many problems in this kernel version, they were disabled by default and are today available only with a boot flag. Meanwhile, some faster-moving distros like Arch have chosen to keep user namespaces disabled even with newer kernel versions due to security concerns: the user namespaces feature has been the source of many local privilege escalation exploits in Linux. Although these vulnerabilities can’t be exploited by Sandstorm apps, such frequent vulnerabilities are problematic for servers which rely on user account separation for security outside of Sandstorm.

Even as it became apparent that Sandstorm’s use of user namespaces was preventing it from being used on some distros, we were hesitant to try other approaches. It seemed like the only way to solve the problem would be to employ a setuid-root binary to set up sandboxes when user namespaces were not available. setuid-root binaries are inherently risky – if not written exactly correctly, it could open its own privilege escalation vulnerability. Also, it would require a major refactoring of Sandstorm internals to move the supervisor into its own binary.

But a couple weeks ago, I realized suddenly that a different idea would work. The Sandstorm server normally starts up as root, but then runs several child processes under a regular user account. Most of Sandstorm’s business logic is in a node.js web server. That process talks via Cap’n Proto RPC to a “back-end” daemon written in C++, which in turn launches app sandboxes. This back-end daemon is hand-coded in C++, with the core logic all living in a single file.

Because of this design, it turned out to be relatively easy to pass superuser privileges down through the back-end, while still keeping them away from the web server. Specifically, the back-end can execute with its effective UID set to a normal user account, but its real UID being root. Then, when it comes time to start a sandbox, it can promote itself back to root to do the work.

This turned out to take only a couple hours to implement. In retrospect, the design seems obvious, and I wish I’d thought of it sooner!

There is a minor downside: If a vulnerability allows an attacker to cause the back-end to execute arbitrary code, that code could claim the superuser privileges, whereas before it would be limited to the Sandstorm server UID. This risk is probably small because the back-end is a relatively simple program that only speaks directly to other trusted programs (although it speaks indirectly to potentially-malicious actors). Nevertheless, if user namespaces are available, then Sandstorm will avoid handing root privileges to the back-end at all, continuing to operate as it did historically.

What do I need to do?

Existing Sandstorm users need not take any action. Your servers will continue to operate exactly as they always have.

But if you’ve been held back from installing Sandstorm before because it wouldn’t work on your distro, you should try again now!

Install Sandstorm Standard »

Try Sandstorm for Work (supports corporate SSO via LDAP/SAML/AD and organization management features) »

Linux kernel CVE-2016-5195 "Dirty COW" mitigated by Sandstorm

Tue, 25 Oct 2016 00:00:00 -0700

Last week, a Linux kernel bug, CVE-2016-5195, was described as “the most serious Linux local privilege escalation ever”. The bug – which potentially allowed any code running on a Linux machine to escalate its privileges to root – was already being actively exploited in the wild before it was fixed, and had existed in the kernel for many years.

Since Sandstorm allows any user of a server to upload their own apps, you might wonder if this bug could allow a Sandstorm user to compromise the server.

We’re happy to report that the answer appears to be “no”. As is often the case with Linux kernel bugs, our sandbox blocked the exploit.

Of course, we still recommend updating your kernel in case the bug can be exploited in ways that have not been discovered yet.

Technical Details

The bug in question was a race condition in the handling of memory pages mapped copy-on-write. A process can ask that a read-only file be mapped into its memory space in such a way that it is allowed to modify the mapped memory. When the process writes to the memory, the kernel makes a private copy of the affected page, so that the process only modifies its copy, not the original. Meanwhile, a process can request later on that the modifications it made be discarded, returning the page to its original state. In certain circumstances, by both writing to a page and requesting this discard at the same time (in separate threads), the process could end up writing to the original pages that are shared with other processes on the system, instead of its own private copy. Hence, the process could modify any file on the system. By modifying, say, the sudo utility, it could give itself a backdoor which allows it to gain root privileges trivially.

However, not just any old write worked here. In order to trigger the race condition, the process had to write in a way that calls the kernel’s get_user_pages() function with the force parameter set to 1. The force parameter says: “If this page is mapped copy-on-write, then let me write to it (making a private copy) even if the page’s protection mode is read-only.” As it turns out, it is possible for a memory mapping to be both read-only and copy-on-write, and in fact this is the mode that is usually used when mapping in a program’s main binary and shared libraries. Normally, no copy is ever performed, because the writes that would trigger them are not allowed. However, there is a special case where this combination of flags matters: If you are running a program in a debugger, and you ask the debugger to insert a breakpoint, it does so by overwriting the instruction at the given address with a break instruction. That is, it modifies the mapped executable. The force flag actually exists for exactly this purpose: so that the debugger can inject breakpoints into the program being executed by the process being debugged (without affecting any other processes that happen to be running the same program).

Because the force flag is only useful in very specific circumstances, only certain code paths can trigger the vulnerability. Kernel security engineer and Sandstorm contributor Andrew Lutomirski tells us the only entry points appear to be:

The ptrace() system call’s PTRACE_POKEDATA operation, which is explicitly meant to be used by debuggers, often for the purpose of setting breakpoints.
Writes to /proc/<pid>/mem. It’s unclear why this code uses force – possibly it was a mistake.
Various drivers, which are also probably using the flag by mistake.

As it turns out, none of these code paths can be exploited by Sandstorm apps:

Sandstorm uses seccomp to block the app from invoking ptrace().
Sandstorm does not mount /proc inside app sandboxes.
Sandstorm does not expose driver interfaces from inside app sandboxes. For example, /dev contains only null, zero, and urandom.

So, as far as we can tell, Sandstorm has never been vulnerable to this bug.

Defense in depth

Even if Sandstorm were vulnerable, the exploit would have far reduced impact inside Sandstorm than in a typical Linux environment, because:

Within a Sandstorm app’s sandbox, the visible filesystem consists of the contents of its own package. It cannot see the host system’s files nor files belonging to other apps, hence it would not be able to memory-map them in order to modify them using this bug.
App packages cannot contain setuid binaries and, even if they could, apps would not be able to execute them, because Sandstorm sets the NO_NEW_PRIVS prctl() flag inside the sandbox.

When running on Sandstorm, a user’s data in an app like Etherpad is containerized separately from another user’s data. In fact, we go one step further and containerize each document separately. In the case that Sandstorm had not mitigated the bug outright, it appears the impact of the bug would be that an app could break Sandstorm’s per-document isolation and read/write documents from any number of users, so long as those users all use the same version of the same app on the same server. The app still would not have been able to interfere with other apps. This is the status quo in a typical Linux environment: in most non-Sandstorm environments, an app keeps all users’ data in a single database without per-user isolation. Overall, this is much less significant than a privilege escalation to root. Thankfully, our seccomp mitigation prevented this.

Sandstorm’s Security Record

This is not the first Linux security bug mitigated by Sandstorm. In fact, we’ve kept a long list. Moreover, in addition to mitigating Linux kernel problem, Sandstorm mitigates most vulnerabilities in the apps that run on top of it. Check out the whole list of mitigated vulnerabilities that we’ve compiled: Sandstorm Security Non-Events

Want to try out Sandstorm as a user? Try the online demo »

Sharing documents with a Rocket.Chat room in Sandstorm

Thu, 13 Oct 2016 00:00:00 -0700

I’m sharing a pro-tip today because I like making sure that everyone gets the most productivity they can out of Sandstorm.

Let’s say I want to share a grain (e.g., a document, spreadsheet, git repository, or a Collection) with a group of colleagues who are already in the same Rocket.Chat chatroom. To do so, I first click the + icon in Rocket.Chat.

This opens a Powerbox request with a type-ahead search box. Before I’ve typed anything, I can see the grains that have most recently been opened by me.

For instance, if I’m looking for feedback for a blog post I drafted in Etherpad, I can type “Etherpad” and it will list all Etherpads that I have access to on this server.

Here are all the Etherpads I have access to. But today, I’m actually searching for something else: a Collection titled “Sales / Revenue docs”. I can also search for grains by title, so I search for “revenue”:

Once I’ve selected the grain, I can choose whether I’d like to give everyone in this chatroom the permission to edit or only view this Collection before I connect the grain with this Rocket.Chat room.

When I share a grain in a chatroom, it automatically renders a snippet which includes the icon for the app that opens the grain. It looks like this:

Now, everyone who is in this chatroom has access to this Collection (as well as every grain inside it). No need to go share it with them separately.

To try it out for yourself, go install Rocket.Chat now!

Do you use Sandstorm to collaborate at work? Sandstorm for Work (60-day free trial) comes with priority support, organization management features, and integration with enterprise infrastructure.

By the way, if you found this useful and would like to see more bite-sized pro-tip style blog posts in the future, please reshare this and let me know (I’m @qiqing on Twitter)!