blog

archives

Moving from Facebook Groups to Discourse

Erlend Sogge Heggen March 11, 2017

The following is a guest post by Martin Eriksson @meriksson

On January 4th 2017, the news aggregation site cor.ax completed its migration from private Facebook Groups to a private Discourse community.

Why the move?

We used to have about 20 Facebook groups for people involved in a network of alternative media projects. Some of the groups were dedicated to editorial discussions, some were general discussion groups for our paying members, some were interest groups about financial issues, gaming, cultural topics etc.

We started using Facebook since it was very easy but we were never comfortable with it. As our network grew and the groups become more numerous and active we become more and more worried about key parts of our infrastructure being under the control of a third-party. If we lost access to our groups it would have caused massive problems for us, including potentially decreasing revenues.

In particular, we were worried since Facebook has increasingly been shutting down not only public pages but also private groups – at least, so we have heard. Since our project has to do with non-mainstream politics and activism, we did not want to take any risks with possible future purges of groups perceived to promote subversive ideas.

Another recurring problem for us was being unable to bring new participants into the project due to them not being Facebook users. One typical case was someone being interested in becoming one of our volunteer editors, but being rejected due to not being able to follow the crucial editorial discussions in the Facebook group for editors. Another, and increasingly costly, problem was people being reluctant to sign up as paying members since our community forum was based on a platform they did not want to use.

After making the move, we have indeed seen an influx of new editors as well as paying members. And not least, we now have people making good contributions to discussions who previously supported the project financially but was on the outside of the social parts of it. Dozens of people actually created Facebook accounts, or reopened previously closed accounts, for the sole reason of accessing our groups. This was something I was personally very ashamed of. Happily, many of these and others as well are now shutting down their accounts or drastically decreasing their activity level on Facebook.

How we made it happen

When we began the migration project I started to look around for ready-made scripts for exporting from Facebook and importing to Discourse. I found one but it was rather rudimentary or experimental. So I realized that we would have to develop a solution ourselves. Luckily I happen to have a background as a Ruby programmer so in part it was a fun and stimulating project for me. And having the existing basic script as a starting point made it really easy to get started.

Some of the major obstacles were working with undocumented aspects of the Facebook API – it turns out that there are many! For example, it was often impossible to access particular objects without any explanation given. In part I had to work by trial and error to figure out how things worked. There are also inconsistencies, most of which are noted in the importer description.

Besides purely technical problems we had to think carefully about how our 20 existing groups could be mapped to the Discourse model. Should we use one Discourse instance or multiple ones? We considered having one Discourse instance per language (we have two languages in daily use among members) and also having one for the editors who run the core projects and one for our community of paying members (readers, podcast listeners etc).

In the end we decided to go for one single Discourse for all groups and instead created an intricate system of categories. One of the reasons for this was to enable searching through all the material instead of forcing users to search on multiple sites. In hindsight this was certainly a good decision, but it created some problems due to the Discourse model being – according to my personal impressions – more oriented toward relatively open forums rather than rigorously segmented ones.

Each Facebook group was imported to a single category to begin with, but since Discourse gives us so much more flexibility than Facebook we could then immediately begin restructuring things. A couple of weeks later we had about 40 categories and more than 10 groups to control access to them. One of the primary groups gives access to about half the categories. This is the group for our paying members who get access to a large number of discussion groups – previously this was very complicated to maintain as a set of Facebook groups, since new members needed to be added individually to each group and we did not know what groups they were interested in. Using categories and groups in Discourse makes it much easier.

We are not done reorganizing all our imported topics in the new categories; a handful of moderators are working on it but we still have more than 2,000 topics in a non-specific category. As more topics get organized, the forum becomes better and better. However, since Discourse is not really intended to be used like this (i.e. doing intricate manipulations of historical content) we also have a number of quirks still be being worked out. We’re actively engaged with the meta.discourse.org community, discussing our issues in some new, some existing topics.

The perks of a Discourse Community

  • People do not need to have Facebook accounts to be active in the community.

  • Search works really well in Discourse (except for being limited to 50 results per query). The search feature in Facebook groups is badly crippled. It seems Facebook wants to steer users away from discovering old threads and instead get them to start new ones. This breaks up interesting discussions in ways that are very bad for exchanging ideas, including as many users as possible, having long-running discussions etc. Needless to say, if you have multiple Facebook groups, there is no way of making one search query for all of them. Moving to Discourse has given new life to thousands of threads which have high-quality content but was previously in effect hidden by Facebook.

  • The Discourse model for threads, replies, quoting etc is a far superior way of structuring conversations. This really comes down to Discourse having a very good and thought-through model whereas Facebook optimizes for entirely different things than serious discussions. It took some time for people to get used to quoting but by now discussions are much easier to follow.

  • Access management through Discourse groups and categories is far more convenient than managing membership in Facebook groups. Also, SSO integration removes the need for adding and removing people for groups. Once members pay the membership fee and get accounts on our site, they can access the forum without someone needing to add them to groups, figure out which of our groups they are likely to want access to etc.

  • The category system in Discourse makes it a lot easier to create a customized structure, it can be changed easily later on if we change our thinking about it etc. In practice, the forum feels much more like our own which creates an atmosphere if familiarity and trust. Along the same lines, all the detailed customization options makes it feel much more like a true home for our project, rather than some space we borrowed from someone.

  • Mail notifications are way better, from customizable levels of following/watching to summary emails. Related to this, tagging works very well and we can for example ping an entire group (normally generating mail notifications to each person). On Facebook we used tagging a lot but it was very cumbersome and time-consuming and there are several limits (only 50 users can be tagged per post/comment, there is an additional limits on how many tags a particular user can create per day – of course, Discourse also has such limits, but we can customize them ourselves).

  • Activity statistics are great for managing a community, both for admins to keep track and for users to see how their engagement compares to other users. Facebook has nothing similar to this. There is a tool called Grytics which can do similar things but it is cumbersome to use and in any case inferior to the built-in Discourse tools. After importing all our historical Facebook data into Discourse it was amazing to be able to see aggregate statistics on e.g. total number of likes received and other per-user breakdowns.


Many thanks to Martin Eriksson @meriksson for not only sharing his migration experience, but also the code used to accomplish it, which is shared freely on GitHub.

0 comments

Discourse Team Grows by 50%

Jeff Atwood December 2, 2015

The Discourse team has grown slowly:

Look at any successful open source project, and you’ll see that it takes a while. And we’re OK with that. We’ve said from the beginning we are on a 10 year mission. In a few months we’ll be 3 years into that mission. (We’ve also had a lot of part-time help from Michael Brown and Kris Aubuchon in the sysadmin and design departments, respectively.)

That’s six people. But today I’m happy to announce that we’re growing the Discourse team by a whopping fifty percent:

Matt Palmer (Australia)

I’ve known Matt as womble on Server Fault forever, at least in avatar form:

womble-se

Did I mention Matt was recently – and completely coincidentally – elected a moderator on Server Fault? Apparently he is a sysadmin of some repute. Not bad for a fictional, pointy-nosed furry creature. That is to say, I didn’t know Matt, but I knew Matt as a creature of the Stack Q&A engine, much like myself.

Turns out he lives in Australia, close-ish to Sam Saffron, and the timing allowed us to meet him in person for our yearly Discourse World Meetup. We even had a delightful boat tour around the Sydney harbor, pictured here.

matt-palmer-boat

As previously documented we had desperate need in the sysadmin department, so having someone incredibly talented in this position is crucial. We’re very fortunate to add Matt to our team.

Erlend Sogge Heggen (Norway)

You may recognize Erlend from his previous work producing the cool little feature demo videos we use in each Discourse release announcement blog post (1.2, 1.3, 1.4). He started building this completely on his own, just out of a love for the project. And those customer interviews, such as OpenROV and Choice of Games? All totally his idea, he approached us with the concept, and he came up with the questions, too. On top of all that, Erlend has been heavily involved in Discourse from the early days of meta.discourse with tons of useful feedback and insight. It was a pleasure to sponsor his work on these great ideas he came up with for us.

erlend-pic

Now it’s very much my pleasure to welcome Erlend to a full time role at Discourse as our first official community advocate. Naturally the whole team participates on meta, but also I believe strongly that it’s the whole team’s job to participate as an active user on Discourse communities. And we do, every one of us. But as the number of hosted Discourse sites has grown, it’s been tough to keep tabs on all of them and jump in and help as needed. Erlend is our man on the street, looking out for everyone, being helpful and letting us know where the whole team can be most helpful.

Guo Xiang Tan (Singapore)

Sam met Alan through his work on RubyBench, which we sponsor through a free hosted Discourse community. (We have a policy of hosting Discourse sites that are part of the open source stack we rely on, as a way of giving back; other examples are Ember.js discussion, CommonMark discussion, and Let’s Encrypt discussion.)

tgx-pic

You may also recognize Guo Xiang as tgxworld from meta, or from his fantastic and frequent GitHub pull request contributions to the Discourse project. Alan became one of our favorite project contributors in recent memory, and reminded us very much of Régis, who we are obviously quite fond of.

Alan technically does not start until January 1st, but we’ve convinced him to work part time through December to ramp up quickly for January. We have a lot of internal hosting improvements we want to make, and since poor Matt has a year long backlog of work we put in front of him, he can use all the help he can get.

Welcome Matt, Erlend, Alan

And then there were nine.

Thanks for joining us on our mission – I think Clay Shirky said it best:

In the past, we could do little things for love, but big things, big things required money. Now, we can do big things for love.

Here’s to the next 7 years of completely free, open source civilized discussion software on the web – and hitting the ground running in 2016. We can’t do it without you.

4 comments

Welcome Arpit Jalan to the Discourse team

Jeff Atwood May 24, 2015

It’s been almost two years since someone joined the Discourse team, but we’re speeding things up this year as our hosting service expands.

Today we’d like to officially welcome Arpit Jalan from Jodhpur, India to the Discourse team.

Here’s what you need to know about Arpit: we did not pick Arpit. He picked us.

We discovered Arpit through our meta site where he just … started building things in early 2014, like the first easy Discourse install guide. We were impressed, and when he wrote us in May 2014 asking for any kind of opportunity to work together, I can’t say I didn’t see it coming.

What I didn’t see coming is Arpit’s depth of conviction. I can’t remember the last time I read such an impassioned call to action by someone who feels they are at a critical juncture in their life. It’s really hard for me to do it justice by describing it here, but at the end of it he cites the 1993 movie Rudy as an inspiration – another guy who has all the heart in the world.

We started with Arpit on a year-long internship plan, and his work has been consistently great. You may recognize Arpit’s hand in …

  • Big improvements in our invite system
  • The ability for any Discourse user to export their posts
  • Managing our Transifex translations
  • Many Discourse howto guides
  • Our internal site monitoring script for hosted clients

… among many, many other things that he did to improve Discourse in the last twelve months.

So when his internship year was up, of course Arpit “Rudy” Jalan made the Discourse team.

You might say he’s been ready for this his whole life.

Welcome to Discourse, Arpit. The work is hard, the hours are long, and our mission will take a decade – but I can’t think of anyone else I’d rather have on our team.

8 comments