Spam fight: Editors on boarding

Hello everybody,

Regarding the SPAM issue on MDN we are currently looking at some technical
change that could help mitigating the SPAM up to the point it becomes
manageable.

Technic is one thing, but at some point it also force us to rethink the way
editors start contributing and what they are allowed to do. And ultimately
how they gain trust and privilege over time. The question is: What is the
on boarding process for new editors.

My understanding of the current situation is that we are granting too many
privileges, without safeguards, to our new users up to the point a spammer
can arm us the way it is now.

One of the thing I'm considering to fight SPAM on MDN is to lower down the
right of new editors and create an on boarding pathway for editors to let
them improve their edit rights. Will made a suggestion that also go that
way. As it is something that will have a direct impact on how we will
interact with our contributors in the future, I wish to have you're
feedback and suggestion about this.

Here is a first draft I made to try setting such an on barding pathway,
this describe the possible rights users could have and the condition they
must reach to get more, cumulative, rights (Note: it miss what we can do to
encourage users to move from one level to the other, but that can be
discute aside to that):


1. MDN Reader
*Any one accessing MDN without being logged.*
- Rights:
- Can read contents
- Extra constrains:
- None
- Condition to reach next level
- Must sign up for an account on MDN
2. New editor
*Someone who sign up to become an MDN editor*
- New rights:
- Can edit existing contents
- Can revert its own edits
- Can access beta display features
- Extra constrains:
- Can edit only once every 5min
- Cannot edit more than 20 times a day
- Condition to reach next level
- Must introduce themselves on IRC or mailing list
- Must be manually upgraded by a Core contributor or an Admin
3. Regular editor
*Someone who sign up to become an MDN editor and that we know about.*
- New rights:
- Can create new contents
- Can create tags
- Can validate review requests
- Can access beta edit features
- Extra constrains:
- Can edit only once every 2min
- Can create content only every 5min
- No more limit to the number of edit or creation
- Condition to reach next level
- Must be manually upgraded by an Admin
4. Trusted editor
*Someone who has proven being a good MDN citizen.*
- New rights:
- Can revert all users edits
- Can move pages
- Can upload attachement
- Extra constrains:
- Not anymore
- Condition to reach next level
- Must be manually upgraded by an Admin
5. Core editor
*Someone who has been extremely dedicated to MDN*
- New rights:
- Can edit kuma macro
- Can upgrade new editors
- Can ban users
- Extra constrains:
- None
- Condition to reach next level
- Must be vouch by two Admin
- Must be approved by legal as Admin can access confidential data.
6. Admin
*Us!*
- Everything without constrain, which mean too much


What do you think? Should we have more steps? Did I miss some right?
Suggestion on how to handle some condition?

Thanks in advance for all you're feedback
Best,
Jeremie
.............................
Web : http://jeremie.patonnier.net
Twitter : @JeremiePat <http://twitter.com/JeremiePat>

This looks promising. I'm not really sure how the human intervention is
going to scale since the plan is to manually validate every editor who
needs more than basic editing privileges.

Also one suggestion regarding reverting. I've seen quite a few times new
users signing up just to revert spam edits. So if we are indeed going to
restrict global reverts to validated users only, I suggest providing an
easier mechanism for users to "report abuse" or "notify admin" when they
see something off on a page.

- jsx
> _______________________________________________
> mdn mailing list
> [email protected]
> https://lists.mozilla.org/listinfo/mdn
>

This is really complicated. I think we should start much simpler see if it
works and build on that. Something along the lines of....


1. New Account: Users first edit must be manually approved.
2. Contributor: User who has had a first edit approved. All subsequent
edits go to akismet. After *N *edits added to queue of potential trusted
writers for vetting by content team.
3. Trusted Writer: Vetted by content team, edits no longer submitted to
Akismet.




----
Benjamin Sternthal
[email protected]

Web Development Manager
Mozilla

irc: bensternthal
(m) 503.702.0963

Hi Ben,

In the short term I totally agree with you.

That said, having a larger discussion to what would be a good way to on
board our users is necessary as it's not just a technical question. The
technical side of it is: How to restrict right (Yes that's not simple). The
functional side of it is how to get our editor moving from a basic set of
action to a whole qualitative full commitment to MDN.

I agree that simpler is better to start with, and complexity can be
added as needed.

We need to balance the need for restricting access by spammers with our
value of openness to encourage participation by well-intentioned humans.
(This touches fundamental issues like "Why is MDN a wiki and not a
closed publishing platform?")

We also need to ensure that we use automation where we need scale, and
keep manual intervention for levels where scale is less of an issue.

Therefore, I think requiring manual approval of first edits is an
non-starter both for scalability and for user experience. (Unless we
want to go to a completely different contribution model like on SUMO.)

Here are some behaviors that are abused by spammers, which we should
restrict new users from doing:
* Create an article
* Change an article's title
* Replace large amounts of content in an existing article (something
confused translators also do)
* Add or change links
* Add attachments
* Make many changes rapidly
* Add irrelevant content

Until recent changes, we have had a post-hoc, manual review of
first-time edits; if a change was not rejected (i.e., reverted or
deleted, and the user banned) within a few days, that edit and that
editor were implicitly accepted.

This process could be automated so that users who are not rejected after
some criterion (e.g., time and/or number of edits) automatically gain
some expanded privileges (a subset of those listed above). I think
manual rejection is more scalable than manual approval.

Higher levels of trust and privilege (such as Jeremie describes) could
be manually managed, although automated tools to present candidates for
review might be helpful.

Ideally, we want a system where users who do constructive, helpful
behaviors are rewarded with more ability to be constructive and helpful.
Many people point to StackOverflow as an exemplary model (which it is),
but it has a very different purpose than MDN. I recommend the book
"Building Web Reputation Systems"[0] for a framework for thinking about
approaching this issue.


[0] http://www.amazon.com/gp/product/059615979X The examples may seem
dated (2010? yeesh!), but I think the principles are not.
> subsequent edits go to akismet. After /N /edits added to queue of
> [email protected] <mailto:[email protected]>
> <[email protected] <mailto:[email protected]>>
> [email protected] <mailto:[email protected]>
> https://lists.mozilla.org/listinfo/mdn
>
>
>

--

Janet Swisher <mailto:[email protected]>
Mozilla Developer Network <https://developer.mozilla.org>
MDN Community Manager

Jeremie: I think the general idea of giving new editors lower rights, and
having admins grant them extra rights, seems like a good idea.

Your proposal doesn't mention Akismet, that seems weird to me. Your
proposal mostly seems to be about rate-limiting. The really nice thing
about Akismet is that the spam never gets onto the public pages, so the
spammers get nothing. With rate limiting, they still get some juice, so
there's still some incentive to attack us.

I didn't suggest rate limiting in my proposal because I worry about there
being situations in which is it quite annoying. For instance, at the Write
the Docs writing day, we might hope to get, say, 30 people signed up in 10
minutes. It would be annoying if we had to deal with this not being allowed.

Having said that, I agree that there might be a place for rate limiting.
Your proposal only limits the # of new edits. I'd expect that in this
scenario, spammers would just create hundreds of accounts and make 20 edits
from each account every day - easily enough to be a massive pain for the
team.


On Mon, Mar 21, 2016 at 8:16 AM, Benjamin Sternthal <[email protected]>
So the suggestion here is that the edit is kept in moderation? I think this
would be hard to implement - for example, what happens when a edit is in
moderation, then someone else edits the same page? But I'm not on the dev
team so this is not my call. Also, what if the first edit is at a time when
the admins are all afk, the editor just has to wait for hours? Also, quite
often, first edits are "nothing edits" - just someone trying out the
editor. That is, the edits don't contain enough information in them for us
to determine whether they are spammers or not. This is why, in my proposal,
the admins keep getting sent edits from new starters until the admins have
enough information to decide whether the new editor is (1) a spammer or (2)
a genuine editor.
> edits go to akismet. After *N *edits added to queue of potential trusted
This makes sense, although I'm not sure about having a hardcoded N for
this. Is the idea that, say, after N edits, admins get an email inviting us
to check the record of the editor for a possible upgrade? If so then we
need an option for "not sure yet", maybe that resets N to 1?

I'm actually not sure that Jeremie's proposal is much more complicated than
yours, it's just that he has included everything, including categories that
are implicit in your proposal. For example:

* readers who are not editors
* people who are allowed to edit macros
* admins

These things all need to be addressed somehow, anyway.

But I do think we could consider collapsing Jeremie's 3 and 4.


>
>
>
> ----
> Benjamin Sternthal
> >> https://lists.mozilla.org/listinfo/mdn
> https://lists.mozilla.org/listinfo/mdn
>

That's certainly true, and could be a useful data point when determining
how to handle posts by new contributors.


> isn't the main problem external links in new and existing pages? How
> much spam content is create /without/ external links?

--

Eric Shepherd
Senior Technical Writer
Mozilla Developer Network <https://developer.mozilla.org/>
Blog: https://www.bitstampede.com/
Twitter: http://twitter.com/sheppy
Doodle: http://doodle.com/the.sheppy

spam content is create *without* external links?

On Mon, Mar 21, 2016 at 10:03 AM, Eric Shepherd <[email protected]>
wrote:

> I feel pretty strongly that we need to avoid human interaction while
> having users scale upward in their permissions. In particular, I very much
> want to see new editors have tight limits on how soon and how often they
> can edit or create new pages, then get promoted to lower restrictions
> automatically once they've successfully contributed some amount of content.
> The less we have to do by hand the better, especially with privilege
> management itself.
--
ali spivak
Head of Developer Marketing
[email protected]

I didn't get the email Sheppy replies to here, but: unfortunately not.
Until recently, yes (and my original proposal here was to ban external
links that weren't on a special "allowed list"). But the recent stuff is
phone numbers.

On Mon, Mar 21, 2016 at 10:39 AM, Eric Shepherd <[email protected]>
wrote:
> > much spam content is create /without/ external links?

Hi Saurabh
Yes, scaling is not a small thing and I look forward to get feed back from
the MDN team about that :)
Good point, noted :)

Hi Will
My bad, just forget about it. Actually, Akismet is more something that we
could use to allow some automation about Banning or removing constrain. I'm
gonna add it to the proposal (I'm in the process to have a living document
to keep track of all the various point raised into that discussion). An
aside question, regardless of the Akismet implementation we are doing, does
that new on boarding workflow could be manageable in the current situation
if we hadn't Akismet?
Well in there is the general case and the specific case. In general I would
like to having things being a little annoying for new user in order to
encourage them to introduce themselves to the community in order to be
granted new rights and getting right of the most annoying restriction. The
idea is to have a better interaction between MDN users: You get more
privilege if you are know from the community.

For special cases like Write the Doc, it is something that is manageable by
hand: During the event you meet people so it's okay to grant them with
extra right immediately (but it remain a manual process as we have to know
them). I don't see those restrictions as a blocker but as a way to promote
the way we are careful bout our content and users.
Indeed, rate limit is not a blocker for spammer, but it is an annoyance
that will cost them more ressources to become a PITA for us. That said rate
limit are a good prevention against bot, but not human. However, Akismet is
still an option ;) as well as other technical solution (like IP banning or
SEO blocking for new content, etc.). The idea behind rate limit is to
increase the cost of spamming MDN up to the point it doesn't worth the
effort to spammer. But let's be clear: There is no perfect solution to
block a massive attack on any web site, only mitigation of the attack. And
as all security related topic, the ultimate question is: Up to which point
do we want to bully legitimate users.
Mmh… I would like a little bit more reasoning for that. 3 is basically
anyone who is not an obvious spammer but I would not be super confortable
to give those users "page move" and "attachement upload" rights as long as
they haven't clearly expressed a need for it. If we want to get read of
rate limit and so in 3, well, why not. But again a little level of
annoyance is a good incentive to do more and get more privileges based on
you're commitment. That said, maybe 3 is a good candidate to get ride of
limitation based on some automated measurement (for example, X month of
contribution without being flagged by Akismet and you automatically get
ride of limitation but do not get any extra rights).

Janet, any though on this?

Best,

Hi Eric :)
I tend to disagree with that but not for Spam reason. For the spam part, I
would be okay that any users in 2 or 3 could see the restriction soften or
removed based on some automation (A number of contribution, a time in which
they are contributing, a clearance from Akismet, etc...). But to grant them
new privileges, I would have the users asking for them and making the
effort to reach to us, to become community members rather than just
anonymous editors. The idea is to increase the quality of the commitment of
our users and the bounds between them. Yes, that's annoying for admin (and
core editors) but that's how we are able to build stronger relationship
between us and our contributors.

As a reminder, having a stronger community of contributors on MDN is one of
this year goal for the team. That Spam issue is an opportunity to rethink
the way we are on boarding editors to become more than that, to become
community members. FWIW my opinion is that we can only build a stronger
community by making an effort to engage with them. If we force them to
reach out to us by annoying them we have to be committed to them by
answering their call for more privilege. It's a give-give situation: I'm
annoying you, so that's okay you annoy me in order to be both relieved of
annoyance.

The following needs to be double check but I'll do some research/math at
some point: If I remember well we have 10 new editors a day. If a tenth of
them reach out to us to get more privilege, it cannot be a burden for Admin
to spend some time discussing with them and grant them new privilege. Even
10 time that is not a burden. If, as I suggest, some core editors (5) have
the opportunity to help moving new users (2) to regular users (3) it's both
not problematic for the Admin and strengthen the relation between community
members. So even if I'm mistaken with the numbers I hardly see how it could
become out of hand for the Admin team.

In any case, it worth starting managing things by hand to see how it goes,
and if really it gets out of hand it will be still possible to automate
things.

Of course there is maybe better way to handle our community building and I
would love to hear about you about that.

Hi Janet :)
Yep, build iteratively based on emerging needs along the way :)
I would also like having us investing more effort into community scaling
for various manual intervention. Currently only Admin are allowed to grant
privilege which is painful, but if we are able to share a part of that
management with community members, than we release the burden for Admin and
increase the community strength.
+1
Great summary, I'm taking notes of them (I'm in the process of setting up a
place to record the outcome of that discussion, more about this in the week)
Good point, however, as I explained in some other e-mails in this thread,
manual approval is not just about granting privileges but also about
creating bounds between editors and the MDN Admin and core editors to
strengthen our community.
Mmh… I like that idea :)

Here's an idea for a first-step which would require only a small amount
of work in the short term, I think (says the guy not writing the code).
It's based very much on ideas from Janet and others here, but sort of
compiled into what I think is a shorter and simpler form.

Let's start with a vouching system. You want to edit, pop into IRC or
onto the mailing list and ask for someone to vouch for you. People who
have been around a while (that is, members of some specific privileged
group) would have vouching permissions and be able to vouch for someone
so that they can sign up. This could be done with a workflow like:

1. User clicks the register button.
2. They go through the same signup process they do now, but at the end
of it they're told that due to the current situation, we have an
added step they need to go through. They need to get vouched for by
one of our longer-time contributors, as well as some information on
how to contact one.
3. The account is created, but is marked as unvouched and as not having
edit permissions (or as inactive, or whatever is simplest). By being
inactive, we need to make no additional changes whatsoever to the
wiki contribution code, which all already knows that these users
can't edit.
4. To get vouched for, they can ask on IRC/mailing list/private mail if
they have it/mdn-admins/whatever makes sense.
5. When the vouching contributor decides to vouch for the newcomer,
they go into their admin panel and click the appropriate button next
to the newcomer's name in the list of pending new accounts to enable
the account and set the "vouched" field appropriately.
6. The account is activated and the user receives their welcome aboard
email.

This lets us keep mostly the same startup flow we have now; the only
addition is a "vouched" field on each user; this could be a boolean but
might be better as a string to contain the voucher's ID, plus the UI to
announce the need to get vouched and how to do it, as well as any
changes needed on the admin side to support the new "can-vouch"
permission and its use.

We could even add a field to the sign-up process so that they can enter
some sort of code that would auto-vouch the user; these codes would be
some kind of pre-generated value that we would be able to create in
advance of events, or which contributors could create and send to
newcomers they want to sign up for an account, so that vouching can
happen instantly instead of having a delay while waiting on a voucher to
go through the admin panel to activate them.
> Mmh... I like that idea :)
> Janet Swisher <mailto:[email protected]>
> March 21, 2016 at 12:47 PM
> Benjamin Sternthal <mailto:[email protected]>
> March 21, 2016 at 11:16 AM
> Saurabh Nair <mailto:[email protected]>
> March 21, 2016 at 11:01 AM
> Jeremie Patonnier <mailto:[email protected]>
> March 21, 2016 at 9:33 AM

So you are asking for 400 vouches/month at current rate, assuming all
new contributors follow these extra steps. Vouching includes answering a
person on IRC or per e-mail, so about 10 minutes work each time: 66h of
extra work per month. [Assuming the current rates]

Asking people to go through complex steps to fix a typo means they
either won't fix the typo and go, or pop in and ask us to fix the typo.



Another way could be to rate-limit overall account creation, in addition
to editions for newbie editors. That way we go back to the situation we
were earlier where spammers couldn't be quicker than we can kill them
and fix the pages affected (and editions should anyway go through
Akismet and 70% at least blocked).

--
Jean-Yves Perrier
Senior St. Project Manager, Developer Documentation / MDN

Hello everybody!

Thanks you all for that first round of feedback :)

I started putting all the results of that discussion on WikiMo:

- https://wiki.mozilla.org/MDN/Get_involved/Becoming_editor

Before moving forward I would emphasis the fact that this specific
discussion is not about short term solution but long term vision and plan.

The current discussion suggest we have two possible strategies to onboard
new editors:

- A "gatekeeper" strategy which requires new editor to be vouch be
someone (another editor, an admin, etc.) and grant them with a large set of
right
- A "progressive enhancement" strategy which gives new editor a limited
set of right and progressively enhance their right based on their behaviors

My initial suggestion is that we should go through a mix of the two, but I
would love to here more about the pro and con of each strategy.

In any case feel free to get into the details of any part of that document.
For example, as you'll see on the wikimo page, there is a set of open
questions for which I would love to get you're though and feedback :)

Like I said, it's a stopgap solution. Jérémie and others have said we're
looking for something we can do quickly that will get us good results,
so I'm making suggestions.

I think whatever idea we come up with, we are going to lose potential
new contributors for a time, until the problem is proparly resolved. The
goal is to get the ability to create new accounts turned back on as soon
as possible, albeit imperfectly, and then we can start work on a less
terrible solution. My ideas have been imperfect but I'm trying, at
least, to make suggestions that are realistic in the timeframe involved.

------------------------------------------------------------------------
*From:* Jean-Yves Perrier
*Sent:* Tuesday, Mar 22, 2016 4:31:32 AM EDT
*To:* [email protected]
*Subject:* Spam fight: Editors on boarding