Sandstorm Blog

The Sandstorm Team is joining Cloudflare

By Kenton Varda - 13 Mar 2017

Recently, we announced that Sandstorm is returning to its community roots, transitioning away from being a for-profit startup and towards being a community-effort, open source project. The Sandstorm team is committed to keep building Sandstorm in collaboration with our amazing community, and we will keep operating Sandstorm Oasis as a service. But, Sandstorm will no longer be our full-time jobs.

So, what will be?

Starting today, most of the Sandstorm team – including Jade and myself – now work for Cloudflare.

Sandstorm has actually been a user of Cloudflare since the beginning. Our web site and static parts of Oasis are served through them. I’ve long been a fan of Cloudflare’s focus on security. A few years back, I enjoyed their Heartbleed Challenge which proved conclusively that, yes, private keys can be leaked by Heartbleed. Although Cloudflare recently suffered from a widely-reported security incident, their response was impressively fast and transparent.

But a bigger reason I’m excited about joining Cloudflare is that they are big users – perhaps the biggest users – of Cap’n Proto, the serialization and RPC framework developed by Sandstorm. Cloudflare actually developed the Lua bindings for Cap’n Proto and has spoken publicly about using Cap’n Proto in their logging pipeline.

Sandstorm, for its part, remains an independent entity, and I won’t be working on it during my day job at Cloudflare. However, I will be working on Cap’n Proto. This should be good news for Cap’n Proto fans who have been disappointed by the lack of releases lately. Although Sandstorm uses Cap’n Proto extensively, in our scramble to build Sandstorm we tended to commit changes only to git master while neglecting to build (and test) formal releases. With a larger corporate backer, we can now afford to be more professional, doing more frequent releases, and putting effort into supporting more languages and platforms.

I will also be working on building some fascinating new infrastructure at Cloudflare. As a member of the edge platform team, I’ll be coming up with new ways to utilize Cloudflare’s machines located in over a hundred data centers around the world. Jade, meanwhile, will be working on building out Cloudflare’s community and developer advocacy program.

The Sandstorm community can expect to see me merging pull requests and pushing releases on weekends, with new releases at least every three weeks. There has been a flurry of activity lately around mobilizing community contributions. Join the discussion on the sandstorm-dev mailing list or on IRC (#sandstorm on Freenode)!

Sandstorm gets a security review

By Kenton Varda - 02 Mar 2017

Sandstorm recently underwent a security review by DevCore Inc., commissioned by Department of Cyber Security of Taiwan. The company has in the past found RCEs in big-name services like Uber and Facebook.

Although the full report is not public (and is not in English), Taiwan’s Digital Minister, Audrey Tang, tells us that overall DevCore feels Sandstorm has “excellent security design.”

As would be expected of any good security review, DevCore did find several issues. Last night we released build 0.203 to fix them. All servers should automatically update within 24 hours of the release (by midnight tonight, PST), but if you’d like to speed up the process (or if you have disabled automatic updates), you can SSH into your server now and type:

sudo sandstorm update

Vulnerabilities Discovered (and fixed)

1. Insufficient e-mail address validation

When logging in with an e-mail address, a user could enter an address like “foo@evil.com,bar@example.com”. In this case, login e-mails would be sent to both addresses, while Sandstorm would treat the user as if they were a member of the latter domain, “example.com”. Hence, an attacker could specify their real address as the first address and a fake address at a domain of their choice as the second, in order to appear to be a member of the domain. Servers that use Sandstorm’s organizational features (until recently only available as part of the now-defunct product “Sandstorm for Work”) can be configured to automatically promote members of a domain to full users, allowing them to install apps, create grains, and discover the identities of other members of the organization. Such users would not receive access to any other grains on the server, but an attacker could install their own apps that consume the server’s resources, including CPU, RAM, and disk. An attacker could also combine this vulnerability with one of the others below, all of which require full user status as a starting point.

The root cause of this bug is Sandstorm’s use of a library called Nodemailer to send e-mail. It turns out that Nodemailer “helpfully” interprets comma-separated lists of addresses as multiple addresses. This is true even if the string is already part of an array. For instance, say you pass the following array in the to field of a mail sent via Nodemailer:

[ "foo@example.com",
  "bar@example.com,baz@example.com",
  { name: "Garply", address: "qux@example.com,corge@example.com" } ]

Surprisingly, this three-element array will send e-mail to five addresses – foo@, bar@, baz@, qux@, and corge@.

We did not anticipate this feature and assumed that Nodemailer would fail to send e-mail to an invalid address.

Severity: Moderate if Sandstorm is configured with an organization defined by e-mail domain. If you do not define an organization, or if you define it by LDAP, SAML, or G Suite domain, then we do not believe this bug can be meaningfully exploited.

Fix: Sandstorm now performs its own validation of addresses to ensure, among other things, that no separator characters are accepted and only one @-sign is allowed. Fixed in commit 37bd9a7.

2. Insufficient path validation when building ZIP backups

The owner of a Sandstorm grain can download a backup of the grain’s content in ZIP format. Sandstorm shells out to the zip utility to build this archive, but does so inside a sandbox designed to prevent exploitation of bugs in the utility.

Unfortunately, a combination of factors resulted in the possibility of a minor data leak:

As a result of this, a user could create a grain containing an empty directory named /var/blah\n/etc.host/passwd, save a backup, and find the backup contained a copy of the host’s /etc/passwd file (which, despite its name, does not contain passwords, but does contain a list of local user accounts).

Severity: Low. An attacker who has full user status (permission to create grains) can read files located in the server’s /etc and /run directories that are set world-readable (or specifically readable to the Sandstorm service account). No other host system directories are exposed. Usually, any files under /etc that contain sensitive secrets would be hidden from all users except root, hence would not be readable by Sandstorm nor by the attacker.

Fix: We have fixed the validation of filenames so that newlines are never permitted, and we have enhanced the zip sandbox to use a whitelist of top-level directories rather than a blacklist. Fixed in commits 4ea8df7 and 6e8572e.

3. Server-side request forgery

Apps running on Sandstorm can make outgoing HTTP requests through a few mechanisms. Additionally, users can instruct Sandstorm to download an app package file from an arbitrary URL. Sandstorm did not attempt to prevent these requests from reaching localhost or private network destinations. Because of this, a full user of the Sandstorm server (but not a visitor) could probe the internal network on which the Sandstorm server was hosted, as well as services exposed to localhost on the server itself.

This is a problem if unprotected services are exposed to the Sandstorm server and you have invited users to your server who should not be trusted to access those services. Sandstorm itself does not expose any unprotected services to localhost, so some other service must be running locally to cause a problem. Only HTTP requests are possible.

Severity: High if unprotected HTTP services are visible to the Sandstorm server and you have invited users to your server who should not be trusted to access those services. Otherwise, low. Sandstorm itself does not expose any unprotected services to localhost, so some other service must be running locally to cause a problem. Only HTTP requests are possible.

Fix: Sandstorm now allows the server admin to configure an IP address blacklist. User-initiated requests will not be allowed from these addresses. By default, standard private network address ranges are blacklisted. Annoyingly, this “fix” is likely to break some servers who use private app indexes, but avoiding such breakage would mean leaving others vulnerable. Admins will need to update their blacklists accordingly. Fixed in commit 164997f.

4. Failure to mitigate Linux kernel CVE-2017-6074

(This was caught by Sandstorm core developer David Renshaw, not by DevCore, but is fixed in the same release.)

A recent Linux kernel bug could allow a malicious app to cause a kernel panic or perhaps break out of its sandbox. The bug is in Linux, but Sandstorm aims to protect against such bugs via seccomp. Historically, Sandstorm’s sandbox has been very successful at blocking Linux vulnerabilities before they are discovered. However, this one slipped through. On Sandstorm, the published PoC exploit for this bug only causes a kernel panic (not a sandbox breakout), but a developer with deep understanding of the bug and the Linux kernel might be able to create a successful attack in other ways.

Severity: High if you permit users you don’t trust to install apps on your server, or commonly install apps from untrustworthy sources, and you do not keep your kernel up-to-date.

Fix: Sandstorm’s seccomp filter has been updated to prohibit the creation of DCCP sockets. Fixed in commit 34749f9.

Conclusion

We are excited to have received such a thorough security review. Some of the problems discovered were very obscure and we’re impressed that DevCore was able to find them. We thank Audrey Tang and the Department of Cyber Security of Taiwan for commissioning this review.

Impact of Cloudflare security advisory on Sandstorm Oasis

By Kenton Varda - 28 Feb 2017

A few days ago, Cloudflare disclosed a security bug in their services which may have leaked secrets from sites which use Cloudflare. Sandstorm uses Cloudflare as a cache in front of our web site and Sandstorm Oasis, and so some of our users have asked if this problem affects them.

TL;DR: If you uploaded or downloaded a grain backup (zip file) between September 22, 2016 and February 18, 2017 (more so between Feb 13 to Feb 18), there is a very remote chance that some portion of the content of your backup could have been leaked to a third party, due to a bug in Cloudflare. We suspect, but cannot say for sure, that no such leakage occurred. We believe that no data other than grain backups could have leaked.

Details

We believe that Sandstorm services are mostly unaffected (with the exception of grain backups, described below). We intentionally divide our services such that public, static content is served from different hosts from dynamic/private content. We only enable Cloudflare in front of the public, static content (to provide caching and faster delivery). Moreover, the public, static hosts avoid storing sensitive credentials in cookies.

Specifically, we have Cloudflare enabled on the following hosts:

We chose this design because we rely on Cloudflare only for caching purposes. Since dynamic data cannot be cached anyway, it made sense to form direct connections for that data rather than pass through a proxy, both to reduce our TCB and to improve performance. On the other hand, this means we cannot fully take advantage of Cloudflare’s security features such as its Web Application Firewall and DDoS protection, but we felt comfortable with this trade-off.

That said, we’ve identified one notable exception to our design: When you download a backup of a grain via the Sandstorm Oasis UI, or upload a backup to be restored, the zip file is transmitted through the main Oasis host, proxied through Cloudflare. This seems to be an oversight; it would make sense for these requests to go through the DDP host (I’ve prepared a pull request to fix this). These transfers are authorized via short-lived tokens, so there’s no risk of Oasis credentials being leaked. However, the contents of a .zip file could possibly have been leaked by Cloudflare.

Cloudflare reports that the bug existed between 2016-09-22 and 2017-02-18, with the greatest potential impact being in the last four days of that period. If you uploaded or downloaded a grain backup in that time, it could have been leaked. However, Cloudflare reports that the incidence of such leaks was extremely small, and each instance would have leaked data from an essentially random website served by Cloudflare, of which there are many (Cloudflare handles some 10% of all internet traffic). Given the probabilities, it is likely that no Sandstorm grain backup data was actually leaked. There is also so far no evidence that anyone was actively exploiting this bug.

With all that said, if you are worried about the possibility that your data was leaked, the proper course of action depends on the app and grain involved. Sandstorm is explicitly designed to keep credentials out of the hands of apps, so usually the only sensitive information in a grain backup is information that you gave to it yourself. For example, if you had an Etherpad document where you kept a list of passwords (and you downloaded a backup of that document recently), you may want to change those passwords. Otherwise, there is nothing to do.

Editorial

Bugs like this probably exist in a lot of software. The Cloudflare bug is notable because:

  1. It was discovered. (Most bugs aren’t.)
  2. Cloudflare handles some 10% of all internet traffic, meaning that basically everyone on the internet is possibly affected.

It’s important to keep in mind that security is not binary. Security is about risk management. The Cloudflare bug did not change anyone’s security level from “secure” to “insecure” – instead, it increased the likelihood of any particular person becoming the victim of a real attack from, say, 5.26% to 5.28% (numbers made up to illustrate point). Had Cloudflare been slow to respond once the vulnerability was found, this risk may have shot up. Fortunately, though, they were anything but.

Now, consider the use of password-based authentication. Nearly every service on the web uses it. However, it is demonstrably horribly insecure: Not only do people regularly use bad passwords or reuse the same password across sites, but passwords stored in a database (even if hashed) make it much easier for attackers to escalate a read-only database snapshot into full access privileges (possibly even to sites other than the one attacked). By requiring the user to receive an authentication code by e-mail each time they log in from a new location, a service can hugely reduce their users’ risk of compromise – probably by an order of magnitude! By this measure, bare password-based authentication should be considered a vulnerability, and a much more serious one than Cloudflare’s bug.

And yet, people accept passwords because they are more convenient than other mechanisms. (Well, many people do. Sandstorm does not implement passwords.) So, we admit that security is not absolute, but rather a trade-off which needs to be judged against other factors. We need to weigh the risks of our infrastructure being insecure against the value that infrastructure brings – and Cloudflare certainly brings a lot of value. That said, with good design, it’s often possible to reduce and contain risks without sacrificing too much utility. This is one of the core goals of Sandstorm – to introduce an infrastructure model and security practices which mitigate risks.

Sandstorm is returning to its community roots

By Kenton Varda - 06 Feb 2017

Most people know Sandstorm as an open source, community-driven project aiming to enable self-hosting of cloud services and to make it possible for open source web apps to compete with today’s cloud services.

Many people also know that Sandstorm is a for-profit startup, with a business model centered on charging for enterprise-oriented features, such as LDAP and SAML single-sign-on integration, organizational access control policies, and the like. This product was called “Sandstorm for Work”; it was still open source, but official builds hid the features behind a paywall. Additionally, we planned eventually to release a scalable version of Sandstorm for big enterprise users, based on the same tech that powers Sandstorm Oasis, our managed hosting service.

As an open source project, Sandstorm has been successful: We have a thriving community of contributors, many developers building and packaging apps, and thousands of self-hosted servers running in the wild. This will continue.

However, our business has not succeeded. To date, almost no one has purchased Sandstorm for Work, despite hundreds of trials and lots of interest expressed. Only a tiny fraction of Sandstorm Oasis users choose to pay for the service – enough to cover costs, but not much more.

We attribute this failure to two main problems:

We plan to publish a more complete postmortem in a subsequent blog post.

Unfortunately, Sandstorm the business has now run out of money, and we have been unable to raise more.

The Project Lives On

Although it will no longer be our full-time job, Sandstorm will continue as an open source project. We still strongly believe in Sandstorm’s long-term vision and cannot abandon it. I personally will continue to lead Sandstorm’s technical development: reviewing and merging pull requests, pushing releases, and developing new features. We will continue to operate Sandstorm Oasis – your data there is safe. Meanwhile, we will make it easier for our extended community to be involved in core development and decision-making. Jade will be in contact with individual community members to appoint community leaders and grant them the authority to handle a variety of community organizing functions, from App Market approval to organizing meetups.

Ironically, the pace of development may not even be hurt much. Over the past year, the Sandstorm team has spent a great deal of our time on enabling our business, e.g. building a payment mechanism, processing customers, marketing, and the like. I personally have spent far too much time on fundraising, sales, and other deal-making rather than on coding. With this shift in direction, we can now focus strictly on building out the core platform, getting more done with less time.

Immediate Action Items

As a result of this change, the following has happened today:

How you can help

Want to see Sandstorm succeed? Then, contribute!

Sandstorm Solutions: pay for feature prioritization

By Drew Fisher - 01 Dec 2016

Is there a feature or bug holding you back from deploying Sandstorm at your company or for your team? Do you need the Sandstorm core devs to prioritize a feature? Sandstorm Solutions can fix that.

There’s always way more on our roadmap than time and people to do it. We receive requests from potential customers around the world who would love to use Sandstorm, but need a particular planned but unimplemented feature, or hit a bug that affects them in particular. We now offer Sandstorm feature prioritization to our customers.

We’re happy to adjust our priorities to support these folks, so we figure we should state that clearly and make sure it’s something everyone can take advantage of, not just the people who inquire.

So here’s the deal with Sandstorm Solutions: if you want a Sandstorm feature prioritized, or a particular bug fixed, we can do it. The way this works is:

Get an estimate »

Need something different? We’re happy to talk about how we can help you succeed - drop us a line at solutions@sandstorm.io.