Discourse 1.9 released!

Jeff Atwood January 8, 2018

We released Discourse 1.9 on January 3rd, building on Discourse 1.8 from last June.

Show Who’s Replying

Discourse is all about replies being disconnected in time and space, but we agree that showing when people are actually replying to a topic is genuinely useful in a variety of contexts. So we’ve added live indicators at the bottom of a topic, and in the composer itself, to let you know when other people are actively replying.

You may also notice we’ve restyled the composer to bring it a lighter, airier, more modern look.

Improved Emoji Selector

We’ve completely revamped the emoji selector to support search by typing and skin tone selection — and of course we’ve incorporated the latest Unicode Emoji, too.

We’ve also added support for the Messenger emoji set (in addition to Apple, Twitter, Google, Microsoft, and EmojiOne), which is selectable in your admin site settings.

Improved Rich Pasting

Discourse natively supports basic bbcode, HTML, and of course Markdown. When pasting content into the composer, we’ll now attempt to convert what is in your clipboard to native Markdown on your behalf!

This turned out to be a bit more complex than we expected, but Excel (and clones) table pasting definitely shipped in 1.9, and further HTML / RTF paste conversions are behind the enable rich text paste site setting — update to 2.0 beta and flip that flag to see it working live!

Better Automatic Link Formatting

If you paste a link on a line by itself, Discourse will summarize the link on your behalf — we call this oneboxing. We’ve improved our default onebox to support more metadata from the link, such as the site icon, the date, and the site title.

Discourse 1.9 improved onebox metadata

And now, even if you paste a link that’s not on a line by itself, we’ll summarize that link for you by pulling the title out of the link, like so:

We call this “inline oneboxing”, and it’s on by default for internal links. If you’d like to turn on this feature for all links, tick enable inline onebox on all domains in your site settings.

Admin Search Stats

Discourse has robust native search and advanced search, as well as excellent web spider search indexability through Google and other popular search engines. But nothing beats looking at what your community is actually searching for and making editorial tweaks to ensure that the most searched for information is easily available. To that end, we now show stats in the admin dashboard on what people are actually searching for on your Discourse:

Large Image Handling

Discourse tries to onebox and mirror all remote linked images by default, so that your topics look good, and don’t decay over time as images disappear from the web. But we also don’t want to use up all your file storage — or blow up your mobile plan when someone links a 30 megabyte GIF file! We now provide better visual feedback when giant remote linked images are posted:

Discourse image too big to display

Watched Words

One question we heard a lot from site owners was how to take action based on the presence of certain words (or word patterns) in posts.

Discourse 1.9 watched words

As of Discourse 1.9, you can set it up so that when any particular word or words appear in posts, you have the option of automatically flagging the post, censoring those specific words, requiring manual staff approval for the post, or completely blocking the post altogether.

Improved Personal Messages

If you’d prefer not to receive any personal messages, you can now disable those in your user preferences. (Staff PMs will still go through, of course.)

Discourse 1.9 disable PMs

Speaking of staff, it’s now possible for staff to specify an email address in PMs, if you want to bring someone into a PM that isn’t yet a member of your Discourse.

Discourse 1.9 email address in PM

Improved Groups

All security in Discourse is handled through groups, and we continue to refine groups in each release to make them easier to understand and organize. If you are a group owner, it’s now possible to send an invite that includes membership in your group:

Discourse 1.9 invite to group

If you are a member of the group and would like to send a message to the entire group, we’ve added a helpful message button to the group page.

And More!

These are just highlights of 1.9 — there are literally hundreds of other tiny improvements, refinements, and bugfixes in the full release notes.

Easy One Click Upgrade

We launched a public exploit bounty program at Hacker One as a part of our security policy one year ago. We continuing to follow up on any concerns and issues raised, and there are important security fixes in 1.9, so we urge everyone to upgrade to it as soon as possible.

If you are on our hosting, you’re already upgraded. Otherwise, upgrading is as easy as clicking the Update button in our built in one click updater linked right from your dashboard:

In some upgrade scenarios, you may need to SSH in to update your server. It’s just 3 commands:

cd /var/discourse
git pull
./launcher rebuild app

If you don’t have a Discourse to upgrade, why not? Install it yourself in under 30 minutes, or get a free 14 day hosting trial!

Thank You

As always, let us first thank our customers for their direct financial support, without which there would be no Discourse project at all.

Any open source project is only as good as its code contributions, and we’re lucky enough to have 104 contributors in this release — 17 more than last time! Thanks for the pull request contributions in this release from:


We had a remarkable number of translators in 2017 who contributed their time and effort into translating Discourse into dozens of languages. It’s because of you that so many people in the world can benefit from great free, open source discussion software!

Finally, thanks to the greater Discourse community for posting support / bug request / feedback topics on meta.discourse. All your suggestions make Discourse better for yourself, and everyone else, too.

As far as we’ve come with Discourse, there are so many important things left to do. Keep an eye on the releases category to see what’s coming … and we’ll see you in Discourse 2.0!


Discourse Gives Back 2017

Jeff Atwood December 21, 2017

We’ve had a fantastic year at Discourse, growing our team to 20 21 people. (There’s also something exciting and major happening early in 2018 that we’ve worked hard on for the last six months that we can’t quite announce yet. But trust me when I say we are very excited!)

open source word cloud

Starting from our initial prototype work in late 2012, Discourse has always been a 100% open source project that builds upon the decades of hard work of many other open source projects to survive. We try to contribute as many upstream fixes as we can, particularly in the case of bugs we run into along the way — but we can and should do more. I’m happy to announce that this year, we’ve grown large enough to earmark funds to contribute directly to the projects we rely on most:

Ember.js $20,000 — Our most critical technology bet when we started in 2013 was on the idea that full blown applications will be built in the web browser, using JavaScript. Ember.js was one of the earliest frameworks for building ambitious JavaScript applications, and continues to be one of the best out there. We’re very encouraged by the recent focus on performance by the Ember team, and we’ve tagged this substantial donation for that continued goal.

Jenkins $5,000 — While we may be beginning to outgrow Jenkins as a build system, for the last few years it has been absolutely essential to everything about our software development process and continuous integration, where we build live on every single GitHub checkin, and deploy rolling beta changes to our customers by tier.

Debian $5,000 — We run Ubuntu Server on all our colocated servers, and Ubuntu is based on the Debian Linux distribution. Without great free, open source operating systems our hosting would be considerably more expensive.

Postgres $5,000 — Discourse’s primary data storage is via the open source Postgres database, which has outstanding performance, proven itself extremely reliable, and generally been a pleasure to work with. We’re looking forward to our global Postgres 10 upgrade in 2018 as well for even more free performance and features!

RubyGems $5,000 — Without the public, shared RubyGems hosting repositories it would be impossible to build any Ruby application, including Discourse.

Linux Conference Australia $4,000 — As the largest linux and open source conference in the Asia-Pacific region, this is near and dear to the heart of our very own Matt Palmer, who hails from Tasmania and basically single handedly got Discourse to where it is today on the hosting side. We’re thrilled to support the conference and the growth of open source software in this area of the world.

Rails Girls $2,500 — We strongly support diversity in the field of software development and have mentored Rails Girls teams for the last four years. We’re also working with Outreachy this year as well, and we pledge to continue to contribute strongly in this area as we grow and have more resources to bring to bear.

Let’s Encrypt $2,500 — Widespread and pervasive encryption helps the web become more resilient and resistant to tampering. We’ve supported this goal with a free enterprise hosted instance for Let’s Encrypt support, as well as a yearly direct monetary donation in at least the amount we would have paid commercially for these now free-for-everyone SSL certificates! Here’s to a more secure web, for everyone.

EFF $2,500 — Discourse is a project that is fundamentally about communities truly owning their own spaces online, rather than temporarily renting space from a giant corporation. Some might call that freedom. The Electronic Frontier Foundation fights for everyone’s digital freedom and we’re proud to support them for 2017.

The Internet Archive $2,500 — Many of the design decisions in Discourse revolve around retaining digital history, such as automatically downloading topic images so they aren’t lost via web rot over time. There is no single organization working to provide a collective memory for the web like The Internet Archive, and we believe future generations will thank them.

OSD $1,000 — Open source software sometimes gets a bad reputation for user interface. The Open Source Design project aims to change that. We support this goal with a free hosted Discourse instance, and now a direct financial contribution as well.

While it’s not a direct donation, per se, we have also paid out $7,872 in Hacker One bounties for valid security issues in 2017 that white hat hackers have reported to us — a special thanks for your efforts in keeping Discourse safe and secure (and free!) for everyone.

Additionally, we became a gold Unicode sponsor by adopting (what else?) the discussion emoji for $5,000:

Discourse, official gold sponsort of the unicode consortium

As 2017 winds to a close, we’re looking forward to 2018 and making these direct financial contributions a regular thing. As we work together, let’s continue to make open source software a part of the public good for everyone.


An Instrumentation Story (How I Learned to Love the Elastic Stack)

Michael Brown November 24, 2017

in which we discover a better way to meet our customer’s needs while showing us where we can improve our code

When you’re operating a hosting service it’s essential to know what’s going on. Visibility into your environment is required, both on a macro scale (performance graphs) and on a micro scale (request logs). Analyzing those request logs can provide you with details hidden by the big picture.

We have a few customers that are outliers in terms of how they use Discourse; their data size might scale out in one particular direction (such as millions of users or a large number of categories) in a way that we don’t expect. We’ve worked with them to ensure they don’t accidentally DOS themselves by sending too many expensive requests via the API, which has caused us to temporarily revoke their API key on their site in response to overloading their resources in a particular way. It’s intrusive and we don’t like doing that – we consider that to be less responsive than we ought to be.

During the discussion with one particular customer about their API usage, we were asked:

Are you able to provide more details so we might be better equipped to understand what might have happened (e.g., time window, quantity of requests, how many requests per API key, requested endpoints, etc.)?

That’s an ENTIRELY SENSIBLE question to ask of us, and with Kibana we’re easily equipped to provide an answer! Let’s work through how to get there together.

On top of the usual information about the request (hostname, remote IP) the Discourse backend adds information about the Ember route used and the CPU processing time taken into response headers:

X-Discourse-Route: list/latest
X-Runtime: 0.370071

which are then captured by our front-end haproxy. The resulting data is fed to logstash along which massages it and injects a document into Elasticsearch. It’s fantastic having this information in Elastisearch, but when you have millions of log entries, what’s the next step? How do you analyze? What do you look for?

To start, I’ve created a new visualization and chosen to work with an entire day’s data (actually: a subset of data across one hosting centre).

My starting screen is not that interesting but it’s still summarizing 279 million documents, comprising syslog entries, nginx logs, application logs and haproxy logs. Let’s pare that down a bit with a filter:


so that we only look at the documents relating to requests that actually hit our application. At the same time let’s add some more metrics that do some statistical analysis on our X-Runtime header value (reminder: this is in milliseconds) so we can gain some insight:

Naturally, we don’t care about EVERYTHING blended together so at this point it makes sense to aggregate statistics along a dimension (for you SQL users out there, this is a GROUP BY) to tease out the variability across types of requests:

And here’s where we start to see something interesting about those requests:

As expected, the most common request to Discourse is to show a topic. Fantastic! But let’s focus and dig into our original mission to find out information about a particular customer. I’ve restricted the record search to only their site and added a new metric: the total runtime taken for that route.

Restricting the search to only those requests with an API key, we arrive at the answer to the question asked by our customer:

That outlier route users/index route is always expensive. Something is probably be improved there. How expensive is it for other customers?

Just processing the requests for this single API route for Customer G is consuming 6 hours of CPU time per day. What is going on, and why is this so expensive?

A quick dig reveals the users/index route is an admin-restricted route that handles a search for users, filtering on username and email. This requires two full table scans to complete. Our asking customer has a lot of users in the system which stresses this particular query.

The fact that this route was being called often (and was so expensive) led us to investigate why it was being used. Quckly we discovered a gap in our API as customers had a need that wasn’t being met by our design. Their workflow looks roughly like this:

  • sync a user details
    • look up a user by email address:
    • check returned users to identify one with an exact email match
    • perform an operation on that user (e.g. add/remove from groups, update user information)

We addressed this by modifying our API in two ways:

  1. Use the column index to shortcut the logic: if there is an exact match in the email column, return only that user (commit)
  2. Add a new API call that requests only an exact email match and isn’t interested in anything else (commit)

These changes (as it turned out, mostly the second one) significantly reduced the time spent on these queries.

That’s a MASSIVE savings of 92.3% for us when handling these API calls from Customer G since they switched to using email= as a filter – more than 5.5 cpu-hours per day of savings just for a single site.

Read more about the Elasticsearch stack at

Many thanks to Jamie Riedesel, reading her article on Getting started with Logstash helped me understand the injest pipeline.